Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.6 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 1516 2426631 2422031 2026-06-13T22:49:36Z Stashbot 7414 AntiComposite: CVNBot4 drop it.wikinews (T428622) 2426631 wikitext text/x-wiki === 2026-06-13 === * 22:49 AntiComposite: CVNBot4 drop it.wikinews ([[phab:T428622|T428622]]) === 2026-06-02 === * 01:03 Krinkle: /cs flags #cvn-sw Divinations voiced === 2026-05-26 === * 18:07 AntiComposite: restart all bots -- disconnected === 2026-05-03 === * 13:39 Krinkle: Disable "Admin immed notify" for cvn-private https://lists.wikimedia.org/postorius/lists/cvn-private.lists.wikimedia.org/settings/automatic_responses. We previously removed the sub form but this is no longer supported in mailman3. We require confirm/moderate for new subs, there is no way to turn it off. But we can at least disable the noise. === 2026-04-27 === * 12:22 Krinkle: /cs flags #cvn-meta NathanVeritas voiced === 2026-04-01 === * 13:34 AntiComposite: restart all bots === 2026-02-04 === * 20:33 AntiComposite: Restart all bots === 2025-12-26 === * 15:54 Operator873: /cs flags #cvn-zh-scan nya_1F616EMO voiced === 2025-11-27 === * 13:48 AntiComposite: CVNBot10 load tok.wikipedia tok: ([[phab:T404567|T404567]]) * 13:47 AntiComposite: CVNBot9 load ms.wikiquote q:ms: ([[phab:T404700|T404700]]) * 13:45 AntiComposite: CVNBot8 load min.wikisource s:min: ([[phab:T408343|T408343]]) * 13:44 AntiComposite: CVNBot7 load pcm.wikiquote q:pcm: ([[phab:T408351|T408351]]) * 13:43 AntiComposite: CVNBot6 load tl.wikisource s:tl: ([[phab:T388654|T388654]]) * 13:42 AntiComposite: CVNBot10 load bew.wiktionary wikt:bew: ([[phab:T402134|T402134]]) * 13:41 AntiComposite: CVNBot9 load zgh.wiktionary wikt:zgh: ([[phab:T399785|T399785]]) * 13:40 AntiComposite: CVNBot8 load min.wikibooks b:min: ([[phab:T395499|T395499]]) * 13:38 AntiComposite: CVNBot7 load rki.wikipedia rki: ([[phab:T392499|T392499]]) * 13:37 AntiComposite: CVNBot6 load mad.wikisource s:mad: ([[phab:T391767|T391767]]) === 2025-10-28 === * 23:16 AntiComposite: /cs flags #cvn-commons revi local_op === 2025-08-20 === * 20:35 AntiComposite: CVNBot10 load nup.wikipedia nup: ([[phab:T390711|T390711]]) === 2025-07-11 === * 14:38 AntiComposite: cvn-app10 restart all bots * 11:10 AntiComposite: cvn-app12 restart all bots * 11:09 AntiComposite: cvn-app10 restart all bots === 2025-06-20 === * 20:49 AntiComposite: cvn-app12: restart all bots * 20:48 AntiComposite: cvn-app10: restart all bots === 2025-05-26 === * 17:59 Krinkle: Create cvn-app14 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:59 Krinkle: Create cvn-app13 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:57 Krinkle: Delete cvn-apache10 instance (replaced/shutdown 2 days ago), ref [[phab:T395164|T395164]] === 2025-05-23 === * 20:30 Krinkle: Shut off cvn-apache10, [[phab:T395164|T395164]] * 20:29 Krinkle: Change cvn.wmcloud.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 20:22 Krinkle: Change cvn.wmflabs.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 19:45 Krinkle: Create cvn-apache11 (debian-12.0-bookworm, g4.cores2.ram4.disk20), [[phab:T395164|T395164]]) === 2025-05-16 === * 18:22 Krinkle: Replace outreach.wikipedia with outreach.wikimedia in cvn-sw/CVNBot19 per https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/820245 since the source channel was renamed * 17:30 Krinkle: krinkle@cvn-apache10:/srv/cvn/git/infrastructure$ git pull -- Deploy https://gerrit.wikimedia.org/r/1146724 * 17:30 Krinkle: krinkle@cvn-apache10 Update git remote in /srv/cvn/git/infrastructure from github.com/countervandalism to https://gerrit.wikimedia.org/r/labs/countervandalism/cvn-infrastructure === 2025-04-21 === * 17:22 AntiComposite: Hard reboot cvn-app10, flapping and not responsive to ssh === 2025-03-30 === * 06:55 Krinkle: krinkle@cvn-apache10: Run `sudo chmod 644 /srv/cvn/git/infrastructure/crontab-config/*.cron`, per [[phab:T390415|T390415]] === 2025-03-12 === * 02:18 AntiComposite: CVNBot9 load id.wikivoyage voy:id: ([[phab:T381080|T381080]]) * 02:15 AntiComposite: CVNBot8 load tig.wikipedia tig: ([[phab:T381379|T381379]]) * 02:14 AntiComposite: CVNBot7 load knc.wikipedia knc: ([[phab:T385185|T385185]]) * 02:11 AntiComposite: CVNBot6 load syl.wikipedia syl: ([[phab:T386464|T386464]]) * 02:08 AntiComposite: CVNBot10 load sat.wiktionary wikt:sat: ([[phab:T386631|T386631]]) === 2025-02-03 === * 22:05 AntiComposite: Hard reboot cvn-apache10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ * 21:58 AntiComposite: Hard reboot cvn-app10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ === 2025-01-02 === * 12:46 Krinkle: /cs flags #cvn-wp-en Lordseriouspig voiced * 12:45 Krinkle: /cs flags #cvn-sw Lordseriouspig voiced === 2024-11-23 === * 00:41 AntiComposite: CVNBot9 load ka.wikisource s:ka: ([[phab:T363243|T363243]]) * 00:38 AntiComposite: CVNBot8 load tcy.wikisource s:tcy: ([[phab:T378471|T378471]]) * 00:37 AntiComposite: CVNBot7 load tcy.wiktionary wikt:tcy: ([[phab:T378463|T378463]]) * 00:25 AntiComposite: Upgrade CVNBot29 to v4.0.4 * 00:25 AntiComposite: Upgrade CVNBot28 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot27 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot26 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot25 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot24 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot23 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot22 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot19 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot17 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot16 to v4.0.4 * 00:20 AntiComposite: Upgrade CVNBot10 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot9 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot8 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot7 to v4.0.4 * 00:17 AntiComposite: Upgrade CVNBot6 to v4.0.4 === 2024-11-22 === * 23:52 AntiComposite: Upgrade CVNBot21 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot20 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot18 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot15 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot14 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot13 to v4.0.4 * 23:49 AntiComposite: Upgrade CVNBot12 to v4.0.4 * 23:48 AntiComposite: Upgrade CVNBot11 to v4.0.4 * 23:47 AntiComposite: Upgrade CVNBot5 to v4.0.4 * 23:45 AntiComposite: Upgrade CVNBot3 to v4.0.4 * 23:44 AntiComposite: Upgrade CVNBot2 to v4.0.4 * 23:41 AntiComposite: Upgrade CVNBot1 to v4.0.4 * 23:32 AntiComposite: Upgrade CVNBot4 to v4.0.4 * 17:08 AntiComposite: restart CVNBots on cvn-app12 due to simultaneous RCReader failure 91950.519949 seconds === 2024-11-08 === * 23:24 AntiComposite: Restarting all CVNBots due to simultaneous RCReader disconnect 54323.128318 seconds ago === 2024-10-29 === * 20:56 AntiComposite: add sh.wikipedia to CVNBot6 as #cvn-wp-sh didn't survive the libera migration * 14:22 AntiComposite: restart all CVNBots === 2024-10-28 === * 12:50 AntiComposite: restarting all CVNBots, not coming up cleanly === 2024-10-25 === * 02:23 AntiComposite: add cs.wikivoyage to CVNBot10 ([[phab:T370913|T370913]]) * 02:21 AntiComposite: add bdr.wikipedia to CVNBot9 ([[phab:T371760|T371760]]) * 02:18 AntiComposite: add mos.wikipedia to CVNBot8 ([[phab:T374644|T374644]]) * 02:14 AntiComposite: add kge.wikipedia to CVNBot7 ([[phab:T374815|T374815]]) * 02:11 AntiComposite: add rsk.wikipedia to CVNBot6 ([[phab:T375017|T375017]]) * 02:07 AntiComposite: add mad.wiktionary to CVNBot9 ([[phab:T375024|T375024]]) * 02:06 AntiComposite: add gor.wikiquote to CVNBot8 ([[phab:T375095|T375095]]) * 02:04 AntiComposite: add nr.wikipedia to CVNBot7 ([[phab:T375102|T375102]]) * 02:01 AntiComposite: add tdd.wikipedia to CVNBot6 ([[phab:T375424|T375424]]) * 01:54 AntiComposite: add shn.wikinews to CVNBot9 ([[phab:T375433|T375433]]) * 01:52 AntiComposite: add iba.wikipedia to CVNBot8 ([[phab:T376572|T376572]]) * 01:50 AntiComposite: add bcl.wikisource to CVNBot7 ([[phab:T377088|T377088]]) * 01:47 AntiComposite: add ann.wikipedia to CVNBot6 ([[phab:T377160|T377160]]) * 01:43 AntiComposite: add igl.wikipedia to CVNBot9 ( [[phab:T363263|T363263]] ) * 01:41 AntiComposite: add my.wikisource to CVNBot8 ([[phab:T363270|T363270]]) * 01:39 AntiComposite: add foundation.wikimedia to CVNBot19 * 01:38 AntiComposite: add wikitech.wikimedia to CVNBot19 === 2024-10-24 === * 11:36 AntiComposite: restart all CVNBots === 2024-10-23 === * 17:33 AntiComposite: restart all CVNBots === 2024-07-03 === * 02:00 AntiComposite: add kus.wikipedia to CVNBot7 ([[phab:T360303|T360303]]) * 01:57 AntiComposite: add bew.wikipedia to CVNBot6 ([[phab:T360310|T360310]]) * 01:54 AntiComposite: add ms.wikisource to CVNBot9 ([[phab:T363250|T363250]]) * 01:53 AntiComposite: add kaa.wiktionary to CVNBot8 ([[phab:T363256|T363256]]) * 01:50 AntiComposite: add dtp.wikipedia to CVNBot7 ([[phab:T365230|T365230]]) * 01:48 AntiComposite: add btm.wikipedia to CVNBot6 ([[phab:T368067|T368067]]) * 01:45 AntiComposite: add fon.wikipedia to CVNBot9 ([[phab:T347939|T347939]]) * 01:43 AntiComposite: add blk.wikisource to CVNBot8 ([[phab:T343542|T343542]]) * 01:41 AntiComposite: su.wikisource to CVNBot7 ([[phab:T343548|T343548]]) * 01:39 AntiComposite: add tly.wikipedia to CVNBot6 ([[phab:T345170|T345170]]) * 01:37 AntiComposite: add dga.wikipedia to CVNBot9 ([[phab:T350229|T350229]]) * 01:35 AntiComposite: add bjn.wikiquote to CVNBot8 ([[phab:T350235|T350235]]) * 01:32 AntiComposite: add zgh.wikipedia to CVNBot7 ([[phab:T350241|T350241]]) * 01:28 AntiComposite: add bbc.wikipedia to CVNBot6 ([[phab:T350373|T350373]]) === 2024-06-24 === * 16:40 Krinkle: cvn-clerkbot parts #cvn-unifications (not operated by CVN, renamed to #wikimedia-unifications) === 2024-06-18 === * 08:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs === 2024-03-22 === * 05:30 Operator873: /cs flags #cvn-simplewikis Drummingman +voice === 2024-02-28 === * 21:34 Krinkle: /cs flags #cvn-wp-da Sarrus local_op === 2024-01-11 === * 12:19 AntiComposite: /cs flags #cvn-meta Bsadowski1 local_op === 2023-12-01 === * 15:30 AntiComposite: restart everything after WMCS network outage === 2023-10-07 === * 14:50 AntiComposite: kill 2 CVNBot11 processes and restart, bot not joined to IRC === 2023-09-22 === * 00:06 Op873: /cs flags #cvn-wp-en Oshwah +AV === 2023-09-16 === * 10:33 JackSparrow: /cs flags #cvn-wp-fa Arian_Ar local_op === 2023-09-07 === * 01:35 AntiComposite: restart all cvn-app12 bots * 01:33 AntiComposite: restart all cvn-app10 bots === 2023-08-15 === * 14:44 AntiComposite: reboot cvn-app10 from Horizon, bots dead and not responding to SSH === 2023-08-09 === * 00:07 AntiComposite: add 9 wikis to #cvn-sw (ref [[phab:T332379|T332379]] [[phab:T336115|T336115]] [[phab:T332093|T332093]] [[phab:T332093|T332093]] [[phab:T335987|T335987]] [[phab:T334459|T334459]] [[phab:T333271|T333271]] [[phab:T334740|T334740]] [[phab:T342865|T342865]]) === 2023-08-08 === * 23:46 AntiComposite: drop wo.wikiquote from CVNBot10 (closed) [[phab:T334482|T334482]] === 2023-07-27 === * 18:15 AntiComposite: Kill and restart CVNBot29 on cvn-app12 === 2023-07-06 === * 16:21 AntiComposite: point git repos to gerrit on cvn-app10 * 16:19 AntiComposite: point git repos to gerrit on cvn-app12 * 16:03 AntiComposite: CVNBot v4.0.3 deployed to all bots ([[phab:T327126|T327126]], [[phab:T327127|T327127]]) * 16:01 AntiComposite: Upgrade CVNBot29 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot28 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot27 to v4.0.3 * 15:59 AntiComposite: Upgrade CVNBot26 to v4.0.3 * 15:58 AntiComposite: Upgrade CVNBot25 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot24 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot23 to v4.0.3 * 15:55 AntiComposite: Upgrade CVNBot22 to v4.0.3 * 15:54 AntiComposite: Upgrade CVNBot19 to v4.0.3 * 15:53 AntiComposite: Upgrade CVNBot17 to v4.0.3 * 15:46 AntiComposite: Upgrade CVNBot16 to v4.0.3 * 15:44 AntiComposite: Upgrade CVNBot10 to v4.0.3 * 15:41 AntiComposite: Upgrade CVNBot9 to v4.0.3 * 15:40 AntiComposite: Upgrade CVNBot8 to v4.0.3 * 15:39 AntiComposite: Upgrade CVNBot7 to v4.0.3 * 15:38 AntiComposite: Upgrade CVNBot6 to v4.0.3 * 04:37 AntiComposite: Upgrade CVNBot21 to v4.0.3 * 04:34 AntiComposite: Upgrade CVNBot20 to v4.0.3 * 04:33 AntiComposite: Upgrade CVNBot18 to v4.0.3 * 04:30 AntiComposite: Upgrade CVNBot15 to v4.0.3 * 04:23 AntiComposite: Upgrade CVNBot14 to v4.0.3 * 04:22 AntiComposite: Upgrade CVNBot13 to v4.0.3 * 04:14 AntiComposite: Upgrade CVNBot12 to v4.0.3 * 04:09 AntiComposite: Upgrade CVNBot11 to v4.0.3 * 04:03 AntiComposite: Upgrade CVNBot5 to v4.0.3 * 04:01 AntiComposite: Upgrade CVNBot4 to v4.0.3 * 04:00 AntiComposite: Upgrade CVNBot3 to v4.0.3 * 03:57 AntiComposite: Upgrade CVNBot2 to v4.0.3 * 03:51 AntiComposite: Upgrade CVNBot1 to v4.0.3 === 2023-06-28 === * 02:34 Operator873: /cs flags #cvn-sw Fehufanga voiced === 2023-06-16 === * 22:05 AntiComposite: manually restart cvn-clerkbot === 2023-05-15 === * 14:58 hauskater: Dropped akwiki and nawiki from CVNBot10 as closed wikis. On-wiki lists require an update. === 2023-04-26 === * 20:07 AntiComposite: /cs flags #cvn-mk-scan M4r51n voiced === 2023-04-21 === * 22:12 Operator873: granted voice to Fehufanga in #cvn-simplewikis === 2023-04-14 === * 18:28 AntiComposite: restart cvn-app10 from horizon, bots quit and ssh times out === 2023-03-22 === * 03:33 Operator873: Voiced Tulsi in #cvn-sw -meta -mediawiki -commons -simplewikis === 2023-03-13 === * 19:46 Operator873: CVNBot18 restarted === 2023-03-03 === * 14:45 AntiComposite: /cs flags #cvn-sw-spam COIBot bot === 2023-02-27 === * 22:33 herzog: Loaded gur.wikipedia to SWMT Group 4 (CVNBot9) - [[phab:T327842|T327842]] * 18:04 herzog: Loaded guc.wikipedia to CVNBot9 / Group 4 - [[phab:T326236|T326236]] === 2023-02-02 === * 00:21 ma: Added 12 new wikis to CVNBot<nowiki>{</nowiki>6,7,8<nowiki>}</nowiki>, 4 to each one. Refs.: [[phab:T321283|T321283]] [[phab:T321289|T321289]] [[phab:T321295|T321295]] [[phab:T326139|T326139]] [[phab:T305281|T305281]] [[phab:T310873|T310873]] [[phab:T312215|T312215]] [[phab:T314640|T314640]] [[phab:T314646|T314646]] [[phab:T316457|T316457]] [[phab:T317113|T317113]] [[phab:T319191|T319191]] === 2023-01-30 === * 22:50 Krinkle: Delete cvn-app8 and cvn-app9 instances, ref [[phab:T306066|T306066]] === 2023-01-28 === * 02:51 AntiComposite: /cs flags #cvn-sw Ajraddatz local_op === 2023-01-24 === * 08:54 Krinkle: Delete cvn-apache9, [[phab:T306066|T306066]] * 08:54 Krinkle: Suspend cvn-app8 and cvn-app9 (`pgrep -af cvn` is empty on both), [[phab:T306066|T306066]] === 2023-01-23 === * 16:53 AntiComposite: Deploy {{Gerrit|716e140}} to app12 ([[phab:T306066|T306066]]) * 16:50 AntiComposite: Deploy {{Gerrit|716e140}} to app9 ([[phab:T306066|T306066]]) * 16:29 AntiComposite: Deploy {{Gerrit|442f324}} to app12 ([[phab:T306066|T306066]]) * 16:25 AntiComposite: Deploy {{Gerrit|442f324}} to app9 ([[phab:T306066|T306066]]) * 16:01 AntiComposite: Deploy {{Gerrit|9024b8f}} to app12 ([[phab:T306066|T306066]]) * 15:59 AntiComposite: Deploy {{Gerrit|9024b8f}} to app9 ([[phab:T306066|T306066]]) === 2023-01-22 === * 21:40 AntiComposite: start cvndb-CVNBot14-publish on app10 * 21:07 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app10, starting bots ([[phab:T306066|T306066]]) * 20:56 AntiComposite: disable cvndb-CVNBot14-publish on app8 * 20:51 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app8, stopping bots ([[phab:T306066|T306066]]) * 19:53 AntiComposite: Deploy {{Gerrit|80ea1f5}} to cvn-app10 ([[phab:T306066|T306066]]) * 15:43 AntiComposite: restart all CVNBots on app9 * 15:42 AntiComposite: restart all CVNBots on app8 === 2023-01-17 === * 00:15 Krinkle: Suspend cvn-apache9, replaced by cvn-apache10, ref [[phab:T306066|T306066]] * 00:14 Krinkle: Switch cvn.wmflabs.org from cvn-apache9 to cvn-apache10 === 2023-01-16 === * 00:10 Krinkle: Move https://github.com/countervandalism/cvn-clerkbot to https://github.com/wikimedia/countervandalism-cvn-clerkbot (with HTTP and Git redirect preserved), and replace with Gerrit mirror === 2023-01-15 === * 23:12 Krinkle: Create 'labs-cvn' permission group in Gerrit with CVN staff members * 23:12 Krinkle: Move https://github.com/countervandalism/cvn-api to https://github.com/wikimedia/countervandalism-cvn-api (with HTTP and Git redirect preserved), and replace with Gerrit mirror * 22:02 Krinkle: Switch new cvn.wmcloud.org proxy from cvn-apache9 to cvn-apache10 (Leave main cvn.wmflabs.org as-is for now). === 2023-01-14 === * 21:45 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|4cee27a}}) * 21:22 AntiComposite: move cvn-clerbot back to cvn-app9 (deploy {{Gerrit|371ba2a}}) * 21:10 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|3f3f40f}}) === 2023-01-10 === * 23:22 Krinkle: krinkle@cvn-apache9$ update infrastructure.git, sudo apachectl graceful * 23:20 Krinkle: Create cvn.wmcloud.org web proxy (in addition to cvn.wmflabs.org) === 2023-01-07 === * 20:53 AntiComposite: apply role::labs::lvm::srv only to cvn-apache9, cvn-app8, and cvn-app9 to fix puppet failures on new instances === 2023-01-04 === * 20:47 Krinkle: Allocate new floating IPs to cvn-app10 and cvn-app11 * 20:46 Krinkle: Create new cvn-apache10, cvn-app10, cvn-app11 with Debian 11 Bullseye to replace the old Debian 9.1 Stretch instances * 20:04 taavi: bump floating ip quota from 2 to 4, [[phab:T326269|T326269]] === 2022-12-27 === * 20:11 Frosty873: /cs flags #cvn-meta xaosflux voiced * 20:11 Frosty873: /cs flags #cvn-wp-en xaosflux voiced === 2022-12-23 === * 03:25 AntiComposite: /cs flags #cvn-meta tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-mediawiki tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-sw tryvix1509 voiced === 2022-10-18 === * 23:13 Joan: CVNBot3 restarted (Last message was received on RCReader 62854.814658 seconds ag) === 2022-09-04 === * 22:21 Operator873: /cs flags #cvn-simplewikis Enfcer +AV * 02:20 Operator873: /cs flags #cvn-sw Bot873 +voiced === 2022-08-26 === * 14:09 hauskatze: Loaded pcm.wikipedia and guw.wiktionary to CVNBot8 & 9 respectively {{!}} [[phab:T310880|T310880]] [[phab:T309057|T309057]] === 2022-07-09 === * 16:42 AntiComposite: /cs flags #cvn-commons pandakekok9 voiced === 2022-07-08 === * 21:53 Krinkle: krinkle@horizon.wikimedia.org Add anticomposite as project member and project admin to cloudvps.cvn === 2022-07-01 === * 21:39 Krinkle: cvn-app8: kill CVNBot14.exe and two (!) procs for CVNBot18.exe === 2022-06-25 === * 03:25 AntiComposite: /cs flags #cvn-wp-en PhantomTech voiced === 2022-06-22 === * 21:04 op873: <+CVNBot3> Added: LuchoCR is on es.wikipedia bot list, added by Operator873{{!}}CVN until the end of time ("Mass blockiing P2P-proxies with script") * 20:34 op873: restart CVNBot3 (possibly caused by block flood) * 19:31 op873: restart CVNBot3 === 2022-06-15 === * 18:49 AntiComposite: /cs flags #cvn-wp-en Zppix voiced * 18:48 AntiComposite: /cs flags #cvn-simplewikis Zppix voiced === 2022-05-23 === * 00:24 Joan: Flags +AV were set on Sargento in cvn-wp-es * 00:23 Joan: Flags +AV were set on alhen in cvn-wp-es === 2022-05-19 === * 23:10 Joan: CVNBot3 restarted (Last message was received on RCReader 92593.747667 seconds ago) === 2022-05-11 === * 07:34 Operator873: /cs flags #cvn-wp-en Tamzin voiced === 2022-05-07 === * 17:40 Operator873: /cs flags #cvn-sw koi voiced * 17:39 Operator873: /cs flags #cvn-zh-scan koi voiced === 2022-04-28 === * 03:19 Joan: CVNBot3 restarted (Last message was received on RCReader 75273.332577 seconds ago) === 2022-04-22 === * 15:08 AntiComposite: /cs flags #cvn-meta Bsadowski1 voiced === 2022-04-18 === * 20:44 AntiComposite: /cs flags #cvn-sw Vermont voiced === 2022-04-13 === * 22:40 Operator873: /cs flags #cvn-meta Joan voiced * 22:40 Operator873: /cs flags #cvn-sw Joan voiced * 22:14 Joan: CVNBot3 restarted (Last message was received on RCReader 54942.175428 seconds ago) === 2022-04-07 === * 23:15 Operator873: /cs flags #cvn-wp-hr NovakWatchmen local_op * 23:13 Operator873: voiced Superpes (Superpes15) in #cvn-sw #cvn-sw-spam and #cvn-it-scan === 2022-04-04 === * 17:34 Operator873: Voiced Vermont in #cvn-meta and #cvn-simplewikis /cs flags #cvn-meta Vermont voiced === 2022-03-30 === * 14:33 Joan: CVNBot3 restarted (Last message was received on RCReader 26318.335196 seconds ago) === 2022-03-28 === * 02:38 AntiComposite: /cs flags #cvn-wp-en Bsoyka voiced === 2022-03-21 === * 20:22 Operator873: /cs flags #cvn-simplewikis Bsadowski1 +AfiotvV * 20:17 Operator873: Operator873{{!}}CVN (Operator873) set flags +AVfitv on Bsadowski1 * 20:03 Operator873: Operator873{{!}}CVN (Operator873) set flags +V on Bsadowski1 * 17:04 AntiComposite: /cs flags #cvn-sw Bsadowski1 local_op === 2022-03-15 === * 15:38 Joan: CVNBot3 restarted (Last message was received on RCReader 26424.279343 seconds ago) === 2022-03-14 === * 14:02 Joan: CVNBot3 restarted (Last message was received on RCReader 17096.72183 seconds ago) === 2022-03-12 === * 16:27 Joan: CVNBot3 restarted (Last message was received on RCReader 27236.775673 seconds ago) === 2022-03-11 === * 14:24 Joan: CVNBot3 restarted (Last message was received on RCReader 18853.006849 seconds ago) === 2022-03-10 === * 14:08 Joan: CVNBot3 restarted (Last message was received on RCReader 22518.614282 seconds ago) === 2022-03-08 === * 20:27 AntiComposite: /cs flags #cvn-wp-en Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-simplewikis Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-commons Sarrus voiced === 2022-03-07 === * 16:30 AntiComposite: /cs flags #cvn-meta zabe voiced * 16:25 AntiComposite: /cs flags #cvn-simplewikis DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-meta DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-sw TheresNoTime voiced * 16:07 Krinkle: /cs flags #cvn-staff Operator873 staff * 16:07 Krinkle: /cs flags #cvn-staff AntiComposite staff === 2022-03-05 === * 04:13 Joan: CVNBot3 restarted (Last message was received on RCReader 31573.894101 seconds ago) === 2022-03-03 === * 16:39 Joan: CVNBot3 restarted (Last message was received on RCReader 36578.236383 seconds ago) === 2022-03-01 === * 13:21 Joan: CVNBot3 restarted (Last message was received on RCReader 20646.781861 seconds ago) === 2022-02-15 === * 14:12 Joan: CVNBot3 restarted (Last message was received on RCReader 25001.391103 seconds ago) === 2022-02-13 === * 18:47 andrewbogott: switching to project-local nfs server cvn-nfs-1 * 17:54 andrewbogott: switching to project-local nfs server puppet-diffs-nfs-1 === 2022-02-10 === * 16:17 Joan: CVNBot3 restarted (Last message was received on RCReader 39817.871151 seconds ago) === 2022-02-08 === * 15:51 Joan: CVNBot3 restarted (Last message was received on RCReader 28868.916144 seconds ago) === 2022-02-04 === * 23:59 andrewbogott: accidentally restarted all VMs due to misreading the project purge page. sorry! === 2022-02-02 === * CVN: Several bots restarted after netsplit took nickserv and some bots with it. * 10:26 Krinkle: CVNBot1 bes del delete(?!d) — originally added by huh (reason: "widewuto") === 2022-02-01 === * 15:20 Joan: CVNBot3 restarted (Last message was received on RCReader 26990.323435 seconds ago) === 2022-01-31 === * 17:37 Joan: CVNBot3 restarted (Last message was received on RCReader 48827.882566 seconds ago) === 2022-01-27 === * 16:58 Joan: CVNBot3 restarted (Last message was received on RCReader 29206.852828 seconds ago) === 2022-01-21 === * 16:07 Joan: CVNBot3 restarted (Last message was received on RCReader 22091.557102 seconds ago) === 2022-01-20 === * 18:13 Cam11598: CVNBot15 restarted === 2022-01-19 === * 17:26 Joan: Restarted CVNBot3 (Last message was received on RCReader 28129.031916 seconds ago) === 2022-01-18 === * 16:55 Joan: Restarted CVNBot3 (Last message was received on RCReader 26283.381782 seconds ago) === 2022-01-17 === * 16:33 Joan: Restarted CVNBot3 (#cvn-wp-es) (Last message was received on RCReader 197065.877109 seconds ago) === 2022-01-15 === * 04:56 Cam11598: restarted CVNBOT18 8:55:47 PM <�25B100+ CVNBot18> Last message was received on RCReader 29723.456263 seconds ago === 2022-01-13 === * 01:29 Cam11598: restarted CVNBot2 nickserv issue * 01:29 Cam11598: restarted CVNBot18 - no response from RC feed === 2022-01-09 === * 18:18 Joan: Flags +AV were set on Hasley in cvn-wp-es (sysop at es.wikipedia) * 17:56 Krinkle: /cs flags #cvn-wp-es Joan local_op === 2022-01-07 === * 22:08 hauskatze: CVNBot9 load co.wiktionary wikt:co: * 22:04 hauskatze: CVNBot9 load ban.wikisource s:ban: * 22:04 hauskatze: CVNBot9 load ba.wikibooks b:ba: * 10:51 hauskatze: Loaded alt.wikipedia to Group 4 (CVNBot9) - small wiki not monitored === 2022-01-06 === * 19:42 hauskatze: Loaded ami.wikipedia to CVNBot8 - [[phab:T292421|T292421]] * 19:41 hauskatze: Loaded pwn.wikipedia to CVNBot7 - [[phab:T292419|T292419]] * 19:39 hauskatze: Loaded lmo.wiktionary to CVNBot6 - [[phab:T292076|T292076]] * 19:34 hauskatze: Loaded jv.wikisource to CVNBot6 refs. [[phab:T287319|T287319]] * 19:29 Krinkle: cs flags #cvn-sw hauskatze local_op * 13:57 Krinkle: Krinkle added $a:Cam11598 to the #cvn-staff I list (+I) {{SAL|Project Name=cvn}} <noinclude> ==Archives== * [[Nova Resource:Cvn/SAL/Archive 1|Archive 1]] (2006-2009) * [[Nova Resource:Cvn/SAL/Archive 2|Archive 2]] (2010-2011) * [[Nova Resource:Cvn/SAL/Archive 3|Archive 3]] (2012-2013) * [[Nova Resource:Cvn/SAL/Archive 4|Archive 4]] (2013-2021) (some parts in 2013 are not indexed) [[Category:SAL]]</noinclude> h65yqqrt9sr1ptsfs3jf7k3a0a53k6q 2426632 2426631 2026-06-13T22:56:43Z Stashbot 7414 AntiComposite: CVNBot6 drop & purge eo.wikinews, fr.wikinews, pl.wikinews, ro.wikinews, sv.wikinews, ta.wikinews (T428622) 2426632 wikitext text/x-wiki === 2026-06-13 === * 22:56 AntiComposite: CVNBot6 drop & purge eo.wikinews, fr.wikinews, pl.wikinews, ro.wikinews, sv.wikinews, ta.wikinews ([[phab:T428622|T428622]]) * 22:49 AntiComposite: CVNBot4 drop it.wikinews ([[phab:T428622|T428622]]) === 2026-06-02 === * 01:03 Krinkle: /cs flags #cvn-sw Divinations voiced === 2026-05-26 === * 18:07 AntiComposite: restart all bots -- disconnected === 2026-05-03 === * 13:39 Krinkle: Disable "Admin immed notify" for cvn-private https://lists.wikimedia.org/postorius/lists/cvn-private.lists.wikimedia.org/settings/automatic_responses. We previously removed the sub form but this is no longer supported in mailman3. We require confirm/moderate for new subs, there is no way to turn it off. But we can at least disable the noise. === 2026-04-27 === * 12:22 Krinkle: /cs flags #cvn-meta NathanVeritas voiced === 2026-04-01 === * 13:34 AntiComposite: restart all bots === 2026-02-04 === * 20:33 AntiComposite: Restart all bots === 2025-12-26 === * 15:54 Operator873: /cs flags #cvn-zh-scan nya_1F616EMO voiced === 2025-11-27 === * 13:48 AntiComposite: CVNBot10 load tok.wikipedia tok: ([[phab:T404567|T404567]]) * 13:47 AntiComposite: CVNBot9 load ms.wikiquote q:ms: ([[phab:T404700|T404700]]) * 13:45 AntiComposite: CVNBot8 load min.wikisource s:min: ([[phab:T408343|T408343]]) * 13:44 AntiComposite: CVNBot7 load pcm.wikiquote q:pcm: ([[phab:T408351|T408351]]) * 13:43 AntiComposite: CVNBot6 load tl.wikisource s:tl: ([[phab:T388654|T388654]]) * 13:42 AntiComposite: CVNBot10 load bew.wiktionary wikt:bew: ([[phab:T402134|T402134]]) * 13:41 AntiComposite: CVNBot9 load zgh.wiktionary wikt:zgh: ([[phab:T399785|T399785]]) * 13:40 AntiComposite: CVNBot8 load min.wikibooks b:min: ([[phab:T395499|T395499]]) * 13:38 AntiComposite: CVNBot7 load rki.wikipedia rki: ([[phab:T392499|T392499]]) * 13:37 AntiComposite: CVNBot6 load mad.wikisource s:mad: ([[phab:T391767|T391767]]) === 2025-10-28 === * 23:16 AntiComposite: /cs flags #cvn-commons revi local_op === 2025-08-20 === * 20:35 AntiComposite: CVNBot10 load nup.wikipedia nup: ([[phab:T390711|T390711]]) === 2025-07-11 === * 14:38 AntiComposite: cvn-app10 restart all bots * 11:10 AntiComposite: cvn-app12 restart all bots * 11:09 AntiComposite: cvn-app10 restart all bots === 2025-06-20 === * 20:49 AntiComposite: cvn-app12: restart all bots * 20:48 AntiComposite: cvn-app10: restart all bots === 2025-05-26 === * 17:59 Krinkle: Create cvn-app14 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:59 Krinkle: Create cvn-app13 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:57 Krinkle: Delete cvn-apache10 instance (replaced/shutdown 2 days ago), ref [[phab:T395164|T395164]] === 2025-05-23 === * 20:30 Krinkle: Shut off cvn-apache10, [[phab:T395164|T395164]] * 20:29 Krinkle: Change cvn.wmcloud.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 20:22 Krinkle: Change cvn.wmflabs.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 19:45 Krinkle: Create cvn-apache11 (debian-12.0-bookworm, g4.cores2.ram4.disk20), [[phab:T395164|T395164]]) === 2025-05-16 === * 18:22 Krinkle: Replace outreach.wikipedia with outreach.wikimedia in cvn-sw/CVNBot19 per https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/820245 since the source channel was renamed * 17:30 Krinkle: krinkle@cvn-apache10:/srv/cvn/git/infrastructure$ git pull -- Deploy https://gerrit.wikimedia.org/r/1146724 * 17:30 Krinkle: krinkle@cvn-apache10 Update git remote in /srv/cvn/git/infrastructure from github.com/countervandalism to https://gerrit.wikimedia.org/r/labs/countervandalism/cvn-infrastructure === 2025-04-21 === * 17:22 AntiComposite: Hard reboot cvn-app10, flapping and not responsive to ssh === 2025-03-30 === * 06:55 Krinkle: krinkle@cvn-apache10: Run `sudo chmod 644 /srv/cvn/git/infrastructure/crontab-config/*.cron`, per [[phab:T390415|T390415]] === 2025-03-12 === * 02:18 AntiComposite: CVNBot9 load id.wikivoyage voy:id: ([[phab:T381080|T381080]]) * 02:15 AntiComposite: CVNBot8 load tig.wikipedia tig: ([[phab:T381379|T381379]]) * 02:14 AntiComposite: CVNBot7 load knc.wikipedia knc: ([[phab:T385185|T385185]]) * 02:11 AntiComposite: CVNBot6 load syl.wikipedia syl: ([[phab:T386464|T386464]]) * 02:08 AntiComposite: CVNBot10 load sat.wiktionary wikt:sat: ([[phab:T386631|T386631]]) === 2025-02-03 === * 22:05 AntiComposite: Hard reboot cvn-apache10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ * 21:58 AntiComposite: Hard reboot cvn-app10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ === 2025-01-02 === * 12:46 Krinkle: /cs flags #cvn-wp-en Lordseriouspig voiced * 12:45 Krinkle: /cs flags #cvn-sw Lordseriouspig voiced === 2024-11-23 === * 00:41 AntiComposite: CVNBot9 load ka.wikisource s:ka: ([[phab:T363243|T363243]]) * 00:38 AntiComposite: CVNBot8 load tcy.wikisource s:tcy: ([[phab:T378471|T378471]]) * 00:37 AntiComposite: CVNBot7 load tcy.wiktionary wikt:tcy: ([[phab:T378463|T378463]]) * 00:25 AntiComposite: Upgrade CVNBot29 to v4.0.4 * 00:25 AntiComposite: Upgrade CVNBot28 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot27 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot26 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot25 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot24 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot23 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot22 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot19 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot17 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot16 to v4.0.4 * 00:20 AntiComposite: Upgrade CVNBot10 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot9 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot8 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot7 to v4.0.4 * 00:17 AntiComposite: Upgrade CVNBot6 to v4.0.4 === 2024-11-22 === * 23:52 AntiComposite: Upgrade CVNBot21 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot20 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot18 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot15 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot14 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot13 to v4.0.4 * 23:49 AntiComposite: Upgrade CVNBot12 to v4.0.4 * 23:48 AntiComposite: Upgrade CVNBot11 to v4.0.4 * 23:47 AntiComposite: Upgrade CVNBot5 to v4.0.4 * 23:45 AntiComposite: Upgrade CVNBot3 to v4.0.4 * 23:44 AntiComposite: Upgrade CVNBot2 to v4.0.4 * 23:41 AntiComposite: Upgrade CVNBot1 to v4.0.4 * 23:32 AntiComposite: Upgrade CVNBot4 to v4.0.4 * 17:08 AntiComposite: restart CVNBots on cvn-app12 due to simultaneous RCReader failure 91950.519949 seconds === 2024-11-08 === * 23:24 AntiComposite: Restarting all CVNBots due to simultaneous RCReader disconnect 54323.128318 seconds ago === 2024-10-29 === * 20:56 AntiComposite: add sh.wikipedia to CVNBot6 as #cvn-wp-sh didn't survive the libera migration * 14:22 AntiComposite: restart all CVNBots === 2024-10-28 === * 12:50 AntiComposite: restarting all CVNBots, not coming up cleanly === 2024-10-25 === * 02:23 AntiComposite: add cs.wikivoyage to CVNBot10 ([[phab:T370913|T370913]]) * 02:21 AntiComposite: add bdr.wikipedia to CVNBot9 ([[phab:T371760|T371760]]) * 02:18 AntiComposite: add mos.wikipedia to CVNBot8 ([[phab:T374644|T374644]]) * 02:14 AntiComposite: add kge.wikipedia to CVNBot7 ([[phab:T374815|T374815]]) * 02:11 AntiComposite: add rsk.wikipedia to CVNBot6 ([[phab:T375017|T375017]]) * 02:07 AntiComposite: add mad.wiktionary to CVNBot9 ([[phab:T375024|T375024]]) * 02:06 AntiComposite: add gor.wikiquote to CVNBot8 ([[phab:T375095|T375095]]) * 02:04 AntiComposite: add nr.wikipedia to CVNBot7 ([[phab:T375102|T375102]]) * 02:01 AntiComposite: add tdd.wikipedia to CVNBot6 ([[phab:T375424|T375424]]) * 01:54 AntiComposite: add shn.wikinews to CVNBot9 ([[phab:T375433|T375433]]) * 01:52 AntiComposite: add iba.wikipedia to CVNBot8 ([[phab:T376572|T376572]]) * 01:50 AntiComposite: add bcl.wikisource to CVNBot7 ([[phab:T377088|T377088]]) * 01:47 AntiComposite: add ann.wikipedia to CVNBot6 ([[phab:T377160|T377160]]) * 01:43 AntiComposite: add igl.wikipedia to CVNBot9 ( [[phab:T363263|T363263]] ) * 01:41 AntiComposite: add my.wikisource to CVNBot8 ([[phab:T363270|T363270]]) * 01:39 AntiComposite: add foundation.wikimedia to CVNBot19 * 01:38 AntiComposite: add wikitech.wikimedia to CVNBot19 === 2024-10-24 === * 11:36 AntiComposite: restart all CVNBots === 2024-10-23 === * 17:33 AntiComposite: restart all CVNBots === 2024-07-03 === * 02:00 AntiComposite: add kus.wikipedia to CVNBot7 ([[phab:T360303|T360303]]) * 01:57 AntiComposite: add bew.wikipedia to CVNBot6 ([[phab:T360310|T360310]]) * 01:54 AntiComposite: add ms.wikisource to CVNBot9 ([[phab:T363250|T363250]]) * 01:53 AntiComposite: add kaa.wiktionary to CVNBot8 ([[phab:T363256|T363256]]) * 01:50 AntiComposite: add dtp.wikipedia to CVNBot7 ([[phab:T365230|T365230]]) * 01:48 AntiComposite: add btm.wikipedia to CVNBot6 ([[phab:T368067|T368067]]) * 01:45 AntiComposite: add fon.wikipedia to CVNBot9 ([[phab:T347939|T347939]]) * 01:43 AntiComposite: add blk.wikisource to CVNBot8 ([[phab:T343542|T343542]]) * 01:41 AntiComposite: su.wikisource to CVNBot7 ([[phab:T343548|T343548]]) * 01:39 AntiComposite: add tly.wikipedia to CVNBot6 ([[phab:T345170|T345170]]) * 01:37 AntiComposite: add dga.wikipedia to CVNBot9 ([[phab:T350229|T350229]]) * 01:35 AntiComposite: add bjn.wikiquote to CVNBot8 ([[phab:T350235|T350235]]) * 01:32 AntiComposite: add zgh.wikipedia to CVNBot7 ([[phab:T350241|T350241]]) * 01:28 AntiComposite: add bbc.wikipedia to CVNBot6 ([[phab:T350373|T350373]]) === 2024-06-24 === * 16:40 Krinkle: cvn-clerkbot parts #cvn-unifications (not operated by CVN, renamed to #wikimedia-unifications) === 2024-06-18 === * 08:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs === 2024-03-22 === * 05:30 Operator873: /cs flags #cvn-simplewikis Drummingman +voice === 2024-02-28 === * 21:34 Krinkle: /cs flags #cvn-wp-da Sarrus local_op === 2024-01-11 === * 12:19 AntiComposite: /cs flags #cvn-meta Bsadowski1 local_op === 2023-12-01 === * 15:30 AntiComposite: restart everything after WMCS network outage === 2023-10-07 === * 14:50 AntiComposite: kill 2 CVNBot11 processes and restart, bot not joined to IRC === 2023-09-22 === * 00:06 Op873: /cs flags #cvn-wp-en Oshwah +AV === 2023-09-16 === * 10:33 JackSparrow: /cs flags #cvn-wp-fa Arian_Ar local_op === 2023-09-07 === * 01:35 AntiComposite: restart all cvn-app12 bots * 01:33 AntiComposite: restart all cvn-app10 bots === 2023-08-15 === * 14:44 AntiComposite: reboot cvn-app10 from Horizon, bots dead and not responding to SSH === 2023-08-09 === * 00:07 AntiComposite: add 9 wikis to #cvn-sw (ref [[phab:T332379|T332379]] [[phab:T336115|T336115]] [[phab:T332093|T332093]] [[phab:T332093|T332093]] [[phab:T335987|T335987]] [[phab:T334459|T334459]] [[phab:T333271|T333271]] [[phab:T334740|T334740]] [[phab:T342865|T342865]]) === 2023-08-08 === * 23:46 AntiComposite: drop wo.wikiquote from CVNBot10 (closed) [[phab:T334482|T334482]] === 2023-07-27 === * 18:15 AntiComposite: Kill and restart CVNBot29 on cvn-app12 === 2023-07-06 === * 16:21 AntiComposite: point git repos to gerrit on cvn-app10 * 16:19 AntiComposite: point git repos to gerrit on cvn-app12 * 16:03 AntiComposite: CVNBot v4.0.3 deployed to all bots ([[phab:T327126|T327126]], [[phab:T327127|T327127]]) * 16:01 AntiComposite: Upgrade CVNBot29 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot28 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot27 to v4.0.3 * 15:59 AntiComposite: Upgrade CVNBot26 to v4.0.3 * 15:58 AntiComposite: Upgrade CVNBot25 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot24 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot23 to v4.0.3 * 15:55 AntiComposite: Upgrade CVNBot22 to v4.0.3 * 15:54 AntiComposite: Upgrade CVNBot19 to v4.0.3 * 15:53 AntiComposite: Upgrade CVNBot17 to v4.0.3 * 15:46 AntiComposite: Upgrade CVNBot16 to v4.0.3 * 15:44 AntiComposite: Upgrade CVNBot10 to v4.0.3 * 15:41 AntiComposite: Upgrade CVNBot9 to v4.0.3 * 15:40 AntiComposite: Upgrade CVNBot8 to v4.0.3 * 15:39 AntiComposite: Upgrade CVNBot7 to v4.0.3 * 15:38 AntiComposite: Upgrade CVNBot6 to v4.0.3 * 04:37 AntiComposite: Upgrade CVNBot21 to v4.0.3 * 04:34 AntiComposite: Upgrade CVNBot20 to v4.0.3 * 04:33 AntiComposite: Upgrade CVNBot18 to v4.0.3 * 04:30 AntiComposite: Upgrade CVNBot15 to v4.0.3 * 04:23 AntiComposite: Upgrade CVNBot14 to v4.0.3 * 04:22 AntiComposite: Upgrade CVNBot13 to v4.0.3 * 04:14 AntiComposite: Upgrade CVNBot12 to v4.0.3 * 04:09 AntiComposite: Upgrade CVNBot11 to v4.0.3 * 04:03 AntiComposite: Upgrade CVNBot5 to v4.0.3 * 04:01 AntiComposite: Upgrade CVNBot4 to v4.0.3 * 04:00 AntiComposite: Upgrade CVNBot3 to v4.0.3 * 03:57 AntiComposite: Upgrade CVNBot2 to v4.0.3 * 03:51 AntiComposite: Upgrade CVNBot1 to v4.0.3 === 2023-06-28 === * 02:34 Operator873: /cs flags #cvn-sw Fehufanga voiced === 2023-06-16 === * 22:05 AntiComposite: manually restart cvn-clerkbot === 2023-05-15 === * 14:58 hauskater: Dropped akwiki and nawiki from CVNBot10 as closed wikis. On-wiki lists require an update. === 2023-04-26 === * 20:07 AntiComposite: /cs flags #cvn-mk-scan M4r51n voiced === 2023-04-21 === * 22:12 Operator873: granted voice to Fehufanga in #cvn-simplewikis === 2023-04-14 === * 18:28 AntiComposite: restart cvn-app10 from horizon, bots quit and ssh times out === 2023-03-22 === * 03:33 Operator873: Voiced Tulsi in #cvn-sw -meta -mediawiki -commons -simplewikis === 2023-03-13 === * 19:46 Operator873: CVNBot18 restarted === 2023-03-03 === * 14:45 AntiComposite: /cs flags #cvn-sw-spam COIBot bot === 2023-02-27 === * 22:33 herzog: Loaded gur.wikipedia to SWMT Group 4 (CVNBot9) - [[phab:T327842|T327842]] * 18:04 herzog: Loaded guc.wikipedia to CVNBot9 / Group 4 - [[phab:T326236|T326236]] === 2023-02-02 === * 00:21 ma: Added 12 new wikis to CVNBot<nowiki>{</nowiki>6,7,8<nowiki>}</nowiki>, 4 to each one. Refs.: [[phab:T321283|T321283]] [[phab:T321289|T321289]] [[phab:T321295|T321295]] [[phab:T326139|T326139]] [[phab:T305281|T305281]] [[phab:T310873|T310873]] [[phab:T312215|T312215]] [[phab:T314640|T314640]] [[phab:T314646|T314646]] [[phab:T316457|T316457]] [[phab:T317113|T317113]] [[phab:T319191|T319191]] === 2023-01-30 === * 22:50 Krinkle: Delete cvn-app8 and cvn-app9 instances, ref [[phab:T306066|T306066]] === 2023-01-28 === * 02:51 AntiComposite: /cs flags #cvn-sw Ajraddatz local_op === 2023-01-24 === * 08:54 Krinkle: Delete cvn-apache9, [[phab:T306066|T306066]] * 08:54 Krinkle: Suspend cvn-app8 and cvn-app9 (`pgrep -af cvn` is empty on both), [[phab:T306066|T306066]] === 2023-01-23 === * 16:53 AntiComposite: Deploy {{Gerrit|716e140}} to app12 ([[phab:T306066|T306066]]) * 16:50 AntiComposite: Deploy {{Gerrit|716e140}} to app9 ([[phab:T306066|T306066]]) * 16:29 AntiComposite: Deploy {{Gerrit|442f324}} to app12 ([[phab:T306066|T306066]]) * 16:25 AntiComposite: Deploy {{Gerrit|442f324}} to app9 ([[phab:T306066|T306066]]) * 16:01 AntiComposite: Deploy {{Gerrit|9024b8f}} to app12 ([[phab:T306066|T306066]]) * 15:59 AntiComposite: Deploy {{Gerrit|9024b8f}} to app9 ([[phab:T306066|T306066]]) === 2023-01-22 === * 21:40 AntiComposite: start cvndb-CVNBot14-publish on app10 * 21:07 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app10, starting bots ([[phab:T306066|T306066]]) * 20:56 AntiComposite: disable cvndb-CVNBot14-publish on app8 * 20:51 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app8, stopping bots ([[phab:T306066|T306066]]) * 19:53 AntiComposite: Deploy {{Gerrit|80ea1f5}} to cvn-app10 ([[phab:T306066|T306066]]) * 15:43 AntiComposite: restart all CVNBots on app9 * 15:42 AntiComposite: restart all CVNBots on app8 === 2023-01-17 === * 00:15 Krinkle: Suspend cvn-apache9, replaced by cvn-apache10, ref [[phab:T306066|T306066]] * 00:14 Krinkle: Switch cvn.wmflabs.org from cvn-apache9 to cvn-apache10 === 2023-01-16 === * 00:10 Krinkle: Move https://github.com/countervandalism/cvn-clerkbot to https://github.com/wikimedia/countervandalism-cvn-clerkbot (with HTTP and Git redirect preserved), and replace with Gerrit mirror === 2023-01-15 === * 23:12 Krinkle: Create 'labs-cvn' permission group in Gerrit with CVN staff members * 23:12 Krinkle: Move https://github.com/countervandalism/cvn-api to https://github.com/wikimedia/countervandalism-cvn-api (with HTTP and Git redirect preserved), and replace with Gerrit mirror * 22:02 Krinkle: Switch new cvn.wmcloud.org proxy from cvn-apache9 to cvn-apache10 (Leave main cvn.wmflabs.org as-is for now). === 2023-01-14 === * 21:45 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|4cee27a}}) * 21:22 AntiComposite: move cvn-clerbot back to cvn-app9 (deploy {{Gerrit|371ba2a}}) * 21:10 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|3f3f40f}}) === 2023-01-10 === * 23:22 Krinkle: krinkle@cvn-apache9$ update infrastructure.git, sudo apachectl graceful * 23:20 Krinkle: Create cvn.wmcloud.org web proxy (in addition to cvn.wmflabs.org) === 2023-01-07 === * 20:53 AntiComposite: apply role::labs::lvm::srv only to cvn-apache9, cvn-app8, and cvn-app9 to fix puppet failures on new instances === 2023-01-04 === * 20:47 Krinkle: Allocate new floating IPs to cvn-app10 and cvn-app11 * 20:46 Krinkle: Create new cvn-apache10, cvn-app10, cvn-app11 with Debian 11 Bullseye to replace the old Debian 9.1 Stretch instances * 20:04 taavi: bump floating ip quota from 2 to 4, [[phab:T326269|T326269]] === 2022-12-27 === * 20:11 Frosty873: /cs flags #cvn-meta xaosflux voiced * 20:11 Frosty873: /cs flags #cvn-wp-en xaosflux voiced === 2022-12-23 === * 03:25 AntiComposite: /cs flags #cvn-meta tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-mediawiki tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-sw tryvix1509 voiced === 2022-10-18 === * 23:13 Joan: CVNBot3 restarted (Last message was received on RCReader 62854.814658 seconds ag) === 2022-09-04 === * 22:21 Operator873: /cs flags #cvn-simplewikis Enfcer +AV * 02:20 Operator873: /cs flags #cvn-sw Bot873 +voiced === 2022-08-26 === * 14:09 hauskatze: Loaded pcm.wikipedia and guw.wiktionary to CVNBot8 & 9 respectively {{!}} [[phab:T310880|T310880]] [[phab:T309057|T309057]] === 2022-07-09 === * 16:42 AntiComposite: /cs flags #cvn-commons pandakekok9 voiced === 2022-07-08 === * 21:53 Krinkle: krinkle@horizon.wikimedia.org Add anticomposite as project member and project admin to cloudvps.cvn === 2022-07-01 === * 21:39 Krinkle: cvn-app8: kill CVNBot14.exe and two (!) procs for CVNBot18.exe === 2022-06-25 === * 03:25 AntiComposite: /cs flags #cvn-wp-en PhantomTech voiced === 2022-06-22 === * 21:04 op873: <+CVNBot3> Added: LuchoCR is on es.wikipedia bot list, added by Operator873{{!}}CVN until the end of time ("Mass blockiing P2P-proxies with script") * 20:34 op873: restart CVNBot3 (possibly caused by block flood) * 19:31 op873: restart CVNBot3 === 2022-06-15 === * 18:49 AntiComposite: /cs flags #cvn-wp-en Zppix voiced * 18:48 AntiComposite: /cs flags #cvn-simplewikis Zppix voiced === 2022-05-23 === * 00:24 Joan: Flags +AV were set on Sargento in cvn-wp-es * 00:23 Joan: Flags +AV were set on alhen in cvn-wp-es === 2022-05-19 === * 23:10 Joan: CVNBot3 restarted (Last message was received on RCReader 92593.747667 seconds ago) === 2022-05-11 === * 07:34 Operator873: /cs flags #cvn-wp-en Tamzin voiced === 2022-05-07 === * 17:40 Operator873: /cs flags #cvn-sw koi voiced * 17:39 Operator873: /cs flags #cvn-zh-scan koi voiced === 2022-04-28 === * 03:19 Joan: CVNBot3 restarted (Last message was received on RCReader 75273.332577 seconds ago) === 2022-04-22 === * 15:08 AntiComposite: /cs flags #cvn-meta Bsadowski1 voiced === 2022-04-18 === * 20:44 AntiComposite: /cs flags #cvn-sw Vermont voiced === 2022-04-13 === * 22:40 Operator873: /cs flags #cvn-meta Joan voiced * 22:40 Operator873: /cs flags #cvn-sw Joan voiced * 22:14 Joan: CVNBot3 restarted (Last message was received on RCReader 54942.175428 seconds ago) === 2022-04-07 === * 23:15 Operator873: /cs flags #cvn-wp-hr NovakWatchmen local_op * 23:13 Operator873: voiced Superpes (Superpes15) in #cvn-sw #cvn-sw-spam and #cvn-it-scan === 2022-04-04 === * 17:34 Operator873: Voiced Vermont in #cvn-meta and #cvn-simplewikis /cs flags #cvn-meta Vermont voiced === 2022-03-30 === * 14:33 Joan: CVNBot3 restarted (Last message was received on RCReader 26318.335196 seconds ago) === 2022-03-28 === * 02:38 AntiComposite: /cs flags #cvn-wp-en Bsoyka voiced === 2022-03-21 === * 20:22 Operator873: /cs flags #cvn-simplewikis Bsadowski1 +AfiotvV * 20:17 Operator873: Operator873{{!}}CVN (Operator873) set flags +AVfitv on Bsadowski1 * 20:03 Operator873: Operator873{{!}}CVN (Operator873) set flags +V on Bsadowski1 * 17:04 AntiComposite: /cs flags #cvn-sw Bsadowski1 local_op === 2022-03-15 === * 15:38 Joan: CVNBot3 restarted (Last message was received on RCReader 26424.279343 seconds ago) === 2022-03-14 === * 14:02 Joan: CVNBot3 restarted (Last message was received on RCReader 17096.72183 seconds ago) === 2022-03-12 === * 16:27 Joan: CVNBot3 restarted (Last message was received on RCReader 27236.775673 seconds ago) === 2022-03-11 === * 14:24 Joan: CVNBot3 restarted (Last message was received on RCReader 18853.006849 seconds ago) === 2022-03-10 === * 14:08 Joan: CVNBot3 restarted (Last message was received on RCReader 22518.614282 seconds ago) === 2022-03-08 === * 20:27 AntiComposite: /cs flags #cvn-wp-en Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-simplewikis Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-commons Sarrus voiced === 2022-03-07 === * 16:30 AntiComposite: /cs flags #cvn-meta zabe voiced * 16:25 AntiComposite: /cs flags #cvn-simplewikis DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-meta DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-sw TheresNoTime voiced * 16:07 Krinkle: /cs flags #cvn-staff Operator873 staff * 16:07 Krinkle: /cs flags #cvn-staff AntiComposite staff === 2022-03-05 === * 04:13 Joan: CVNBot3 restarted (Last message was received on RCReader 31573.894101 seconds ago) === 2022-03-03 === * 16:39 Joan: CVNBot3 restarted (Last message was received on RCReader 36578.236383 seconds ago) === 2022-03-01 === * 13:21 Joan: CVNBot3 restarted (Last message was received on RCReader 20646.781861 seconds ago) === 2022-02-15 === * 14:12 Joan: CVNBot3 restarted (Last message was received on RCReader 25001.391103 seconds ago) === 2022-02-13 === * 18:47 andrewbogott: switching to project-local nfs server cvn-nfs-1 * 17:54 andrewbogott: switching to project-local nfs server puppet-diffs-nfs-1 === 2022-02-10 === * 16:17 Joan: CVNBot3 restarted (Last message was received on RCReader 39817.871151 seconds ago) === 2022-02-08 === * 15:51 Joan: CVNBot3 restarted (Last message was received on RCReader 28868.916144 seconds ago) === 2022-02-04 === * 23:59 andrewbogott: accidentally restarted all VMs due to misreading the project purge page. sorry! === 2022-02-02 === * CVN: Several bots restarted after netsplit took nickserv and some bots with it. * 10:26 Krinkle: CVNBot1 bes del delete(?!d) — originally added by huh (reason: "widewuto") === 2022-02-01 === * 15:20 Joan: CVNBot3 restarted (Last message was received on RCReader 26990.323435 seconds ago) === 2022-01-31 === * 17:37 Joan: CVNBot3 restarted (Last message was received on RCReader 48827.882566 seconds ago) === 2022-01-27 === * 16:58 Joan: CVNBot3 restarted (Last message was received on RCReader 29206.852828 seconds ago) === 2022-01-21 === * 16:07 Joan: CVNBot3 restarted (Last message was received on RCReader 22091.557102 seconds ago) === 2022-01-20 === * 18:13 Cam11598: CVNBot15 restarted === 2022-01-19 === * 17:26 Joan: Restarted CVNBot3 (Last message was received on RCReader 28129.031916 seconds ago) === 2022-01-18 === * 16:55 Joan: Restarted CVNBot3 (Last message was received on RCReader 26283.381782 seconds ago) === 2022-01-17 === * 16:33 Joan: Restarted CVNBot3 (#cvn-wp-es) (Last message was received on RCReader 197065.877109 seconds ago) === 2022-01-15 === * 04:56 Cam11598: restarted CVNBOT18 8:55:47 PM <�25B100+ CVNBot18> Last message was received on RCReader 29723.456263 seconds ago === 2022-01-13 === * 01:29 Cam11598: restarted CVNBot2 nickserv issue * 01:29 Cam11598: restarted CVNBot18 - no response from RC feed === 2022-01-09 === * 18:18 Joan: Flags +AV were set on Hasley in cvn-wp-es (sysop at es.wikipedia) * 17:56 Krinkle: /cs flags #cvn-wp-es Joan local_op === 2022-01-07 === * 22:08 hauskatze: CVNBot9 load co.wiktionary wikt:co: * 22:04 hauskatze: CVNBot9 load ban.wikisource s:ban: * 22:04 hauskatze: CVNBot9 load ba.wikibooks b:ba: * 10:51 hauskatze: Loaded alt.wikipedia to Group 4 (CVNBot9) - small wiki not monitored === 2022-01-06 === * 19:42 hauskatze: Loaded ami.wikipedia to CVNBot8 - [[phab:T292421|T292421]] * 19:41 hauskatze: Loaded pwn.wikipedia to CVNBot7 - [[phab:T292419|T292419]] * 19:39 hauskatze: Loaded lmo.wiktionary to CVNBot6 - [[phab:T292076|T292076]] * 19:34 hauskatze: Loaded jv.wikisource to CVNBot6 refs. [[phab:T287319|T287319]] * 19:29 Krinkle: cs flags #cvn-sw hauskatze local_op * 13:57 Krinkle: Krinkle added $a:Cam11598 to the #cvn-staff I list (+I) {{SAL|Project Name=cvn}} <noinclude> ==Archives== * [[Nova Resource:Cvn/SAL/Archive 1|Archive 1]] (2006-2009) * [[Nova Resource:Cvn/SAL/Archive 2|Archive 2]] (2010-2011) * [[Nova Resource:Cvn/SAL/Archive 3|Archive 3]] (2012-2013) * [[Nova Resource:Cvn/SAL/Archive 4|Archive 4]] (2013-2021) (some parts in 2013 are not indexed) [[Category:SAL]]</noinclude> fbvjkx3ys5e7pgi3nhu8415nd556q68 2426633 2426632 2026-06-13T22:58:39Z Stashbot 7414 AntiComposite: CVNBot7 drop & purge es.wikinews, guw.wikinews, pt.wikinews (T428622) 2426633 wikitext text/x-wiki === 2026-06-13 === * 22:58 AntiComposite: CVNBot7 drop & purge es.wikinews, guw.wikinews, pt.wikinews ([[phab:T428622|T428622]]) * 22:56 AntiComposite: CVNBot6 drop & purge eo.wikinews, fr.wikinews, pl.wikinews, ro.wikinews, sv.wikinews, ta.wikinews ([[phab:T428622|T428622]]) * 22:49 AntiComposite: CVNBot4 drop it.wikinews ([[phab:T428622|T428622]]) === 2026-06-02 === * 01:03 Krinkle: /cs flags #cvn-sw Divinations voiced === 2026-05-26 === * 18:07 AntiComposite: restart all bots -- disconnected === 2026-05-03 === * 13:39 Krinkle: Disable "Admin immed notify" for cvn-private https://lists.wikimedia.org/postorius/lists/cvn-private.lists.wikimedia.org/settings/automatic_responses. We previously removed the sub form but this is no longer supported in mailman3. We require confirm/moderate for new subs, there is no way to turn it off. But we can at least disable the noise. === 2026-04-27 === * 12:22 Krinkle: /cs flags #cvn-meta NathanVeritas voiced === 2026-04-01 === * 13:34 AntiComposite: restart all bots === 2026-02-04 === * 20:33 AntiComposite: Restart all bots === 2025-12-26 === * 15:54 Operator873: /cs flags #cvn-zh-scan nya_1F616EMO voiced === 2025-11-27 === * 13:48 AntiComposite: CVNBot10 load tok.wikipedia tok: ([[phab:T404567|T404567]]) * 13:47 AntiComposite: CVNBot9 load ms.wikiquote q:ms: ([[phab:T404700|T404700]]) * 13:45 AntiComposite: CVNBot8 load min.wikisource s:min: ([[phab:T408343|T408343]]) * 13:44 AntiComposite: CVNBot7 load pcm.wikiquote q:pcm: ([[phab:T408351|T408351]]) * 13:43 AntiComposite: CVNBot6 load tl.wikisource s:tl: ([[phab:T388654|T388654]]) * 13:42 AntiComposite: CVNBot10 load bew.wiktionary wikt:bew: ([[phab:T402134|T402134]]) * 13:41 AntiComposite: CVNBot9 load zgh.wiktionary wikt:zgh: ([[phab:T399785|T399785]]) * 13:40 AntiComposite: CVNBot8 load min.wikibooks b:min: ([[phab:T395499|T395499]]) * 13:38 AntiComposite: CVNBot7 load rki.wikipedia rki: ([[phab:T392499|T392499]]) * 13:37 AntiComposite: CVNBot6 load mad.wikisource s:mad: ([[phab:T391767|T391767]]) === 2025-10-28 === * 23:16 AntiComposite: /cs flags #cvn-commons revi local_op === 2025-08-20 === * 20:35 AntiComposite: CVNBot10 load nup.wikipedia nup: ([[phab:T390711|T390711]]) === 2025-07-11 === * 14:38 AntiComposite: cvn-app10 restart all bots * 11:10 AntiComposite: cvn-app12 restart all bots * 11:09 AntiComposite: cvn-app10 restart all bots === 2025-06-20 === * 20:49 AntiComposite: cvn-app12: restart all bots * 20:48 AntiComposite: cvn-app10: restart all bots === 2025-05-26 === * 17:59 Krinkle: Create cvn-app14 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:59 Krinkle: Create cvn-app13 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:57 Krinkle: Delete cvn-apache10 instance (replaced/shutdown 2 days ago), ref [[phab:T395164|T395164]] === 2025-05-23 === * 20:30 Krinkle: Shut off cvn-apache10, [[phab:T395164|T395164]] * 20:29 Krinkle: Change cvn.wmcloud.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 20:22 Krinkle: Change cvn.wmflabs.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 19:45 Krinkle: Create cvn-apache11 (debian-12.0-bookworm, g4.cores2.ram4.disk20), [[phab:T395164|T395164]]) === 2025-05-16 === * 18:22 Krinkle: Replace outreach.wikipedia with outreach.wikimedia in cvn-sw/CVNBot19 per https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/820245 since the source channel was renamed * 17:30 Krinkle: krinkle@cvn-apache10:/srv/cvn/git/infrastructure$ git pull -- Deploy https://gerrit.wikimedia.org/r/1146724 * 17:30 Krinkle: krinkle@cvn-apache10 Update git remote in /srv/cvn/git/infrastructure from github.com/countervandalism to https://gerrit.wikimedia.org/r/labs/countervandalism/cvn-infrastructure === 2025-04-21 === * 17:22 AntiComposite: Hard reboot cvn-app10, flapping and not responsive to ssh === 2025-03-30 === * 06:55 Krinkle: krinkle@cvn-apache10: Run `sudo chmod 644 /srv/cvn/git/infrastructure/crontab-config/*.cron`, per [[phab:T390415|T390415]] === 2025-03-12 === * 02:18 AntiComposite: CVNBot9 load id.wikivoyage voy:id: ([[phab:T381080|T381080]]) * 02:15 AntiComposite: CVNBot8 load tig.wikipedia tig: ([[phab:T381379|T381379]]) * 02:14 AntiComposite: CVNBot7 load knc.wikipedia knc: ([[phab:T385185|T385185]]) * 02:11 AntiComposite: CVNBot6 load syl.wikipedia syl: ([[phab:T386464|T386464]]) * 02:08 AntiComposite: CVNBot10 load sat.wiktionary wikt:sat: ([[phab:T386631|T386631]]) === 2025-02-03 === * 22:05 AntiComposite: Hard reboot cvn-apache10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ * 21:58 AntiComposite: Hard reboot cvn-app10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ === 2025-01-02 === * 12:46 Krinkle: /cs flags #cvn-wp-en Lordseriouspig voiced * 12:45 Krinkle: /cs flags #cvn-sw Lordseriouspig voiced === 2024-11-23 === * 00:41 AntiComposite: CVNBot9 load ka.wikisource s:ka: ([[phab:T363243|T363243]]) * 00:38 AntiComposite: CVNBot8 load tcy.wikisource s:tcy: ([[phab:T378471|T378471]]) * 00:37 AntiComposite: CVNBot7 load tcy.wiktionary wikt:tcy: ([[phab:T378463|T378463]]) * 00:25 AntiComposite: Upgrade CVNBot29 to v4.0.4 * 00:25 AntiComposite: Upgrade CVNBot28 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot27 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot26 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot25 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot24 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot23 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot22 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot19 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot17 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot16 to v4.0.4 * 00:20 AntiComposite: Upgrade CVNBot10 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot9 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot8 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot7 to v4.0.4 * 00:17 AntiComposite: Upgrade CVNBot6 to v4.0.4 === 2024-11-22 === * 23:52 AntiComposite: Upgrade CVNBot21 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot20 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot18 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot15 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot14 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot13 to v4.0.4 * 23:49 AntiComposite: Upgrade CVNBot12 to v4.0.4 * 23:48 AntiComposite: Upgrade CVNBot11 to v4.0.4 * 23:47 AntiComposite: Upgrade CVNBot5 to v4.0.4 * 23:45 AntiComposite: Upgrade CVNBot3 to v4.0.4 * 23:44 AntiComposite: Upgrade CVNBot2 to v4.0.4 * 23:41 AntiComposite: Upgrade CVNBot1 to v4.0.4 * 23:32 AntiComposite: Upgrade CVNBot4 to v4.0.4 * 17:08 AntiComposite: restart CVNBots on cvn-app12 due to simultaneous RCReader failure 91950.519949 seconds === 2024-11-08 === * 23:24 AntiComposite: Restarting all CVNBots due to simultaneous RCReader disconnect 54323.128318 seconds ago === 2024-10-29 === * 20:56 AntiComposite: add sh.wikipedia to CVNBot6 as #cvn-wp-sh didn't survive the libera migration * 14:22 AntiComposite: restart all CVNBots === 2024-10-28 === * 12:50 AntiComposite: restarting all CVNBots, not coming up cleanly === 2024-10-25 === * 02:23 AntiComposite: add cs.wikivoyage to CVNBot10 ([[phab:T370913|T370913]]) * 02:21 AntiComposite: add bdr.wikipedia to CVNBot9 ([[phab:T371760|T371760]]) * 02:18 AntiComposite: add mos.wikipedia to CVNBot8 ([[phab:T374644|T374644]]) * 02:14 AntiComposite: add kge.wikipedia to CVNBot7 ([[phab:T374815|T374815]]) * 02:11 AntiComposite: add rsk.wikipedia to CVNBot6 ([[phab:T375017|T375017]]) * 02:07 AntiComposite: add mad.wiktionary to CVNBot9 ([[phab:T375024|T375024]]) * 02:06 AntiComposite: add gor.wikiquote to CVNBot8 ([[phab:T375095|T375095]]) * 02:04 AntiComposite: add nr.wikipedia to CVNBot7 ([[phab:T375102|T375102]]) * 02:01 AntiComposite: add tdd.wikipedia to CVNBot6 ([[phab:T375424|T375424]]) * 01:54 AntiComposite: add shn.wikinews to CVNBot9 ([[phab:T375433|T375433]]) * 01:52 AntiComposite: add iba.wikipedia to CVNBot8 ([[phab:T376572|T376572]]) * 01:50 AntiComposite: add bcl.wikisource to CVNBot7 ([[phab:T377088|T377088]]) * 01:47 AntiComposite: add ann.wikipedia to CVNBot6 ([[phab:T377160|T377160]]) * 01:43 AntiComposite: add igl.wikipedia to CVNBot9 ( [[phab:T363263|T363263]] ) * 01:41 AntiComposite: add my.wikisource to CVNBot8 ([[phab:T363270|T363270]]) * 01:39 AntiComposite: add foundation.wikimedia to CVNBot19 * 01:38 AntiComposite: add wikitech.wikimedia to CVNBot19 === 2024-10-24 === * 11:36 AntiComposite: restart all CVNBots === 2024-10-23 === * 17:33 AntiComposite: restart all CVNBots === 2024-07-03 === * 02:00 AntiComposite: add kus.wikipedia to CVNBot7 ([[phab:T360303|T360303]]) * 01:57 AntiComposite: add bew.wikipedia to CVNBot6 ([[phab:T360310|T360310]]) * 01:54 AntiComposite: add ms.wikisource to CVNBot9 ([[phab:T363250|T363250]]) * 01:53 AntiComposite: add kaa.wiktionary to CVNBot8 ([[phab:T363256|T363256]]) * 01:50 AntiComposite: add dtp.wikipedia to CVNBot7 ([[phab:T365230|T365230]]) * 01:48 AntiComposite: add btm.wikipedia to CVNBot6 ([[phab:T368067|T368067]]) * 01:45 AntiComposite: add fon.wikipedia to CVNBot9 ([[phab:T347939|T347939]]) * 01:43 AntiComposite: add blk.wikisource to CVNBot8 ([[phab:T343542|T343542]]) * 01:41 AntiComposite: su.wikisource to CVNBot7 ([[phab:T343548|T343548]]) * 01:39 AntiComposite: add tly.wikipedia to CVNBot6 ([[phab:T345170|T345170]]) * 01:37 AntiComposite: add dga.wikipedia to CVNBot9 ([[phab:T350229|T350229]]) * 01:35 AntiComposite: add bjn.wikiquote to CVNBot8 ([[phab:T350235|T350235]]) * 01:32 AntiComposite: add zgh.wikipedia to CVNBot7 ([[phab:T350241|T350241]]) * 01:28 AntiComposite: add bbc.wikipedia to CVNBot6 ([[phab:T350373|T350373]]) === 2024-06-24 === * 16:40 Krinkle: cvn-clerkbot parts #cvn-unifications (not operated by CVN, renamed to #wikimedia-unifications) === 2024-06-18 === * 08:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs === 2024-03-22 === * 05:30 Operator873: /cs flags #cvn-simplewikis Drummingman +voice === 2024-02-28 === * 21:34 Krinkle: /cs flags #cvn-wp-da Sarrus local_op === 2024-01-11 === * 12:19 AntiComposite: /cs flags #cvn-meta Bsadowski1 local_op === 2023-12-01 === * 15:30 AntiComposite: restart everything after WMCS network outage === 2023-10-07 === * 14:50 AntiComposite: kill 2 CVNBot11 processes and restart, bot not joined to IRC === 2023-09-22 === * 00:06 Op873: /cs flags #cvn-wp-en Oshwah +AV === 2023-09-16 === * 10:33 JackSparrow: /cs flags #cvn-wp-fa Arian_Ar local_op === 2023-09-07 === * 01:35 AntiComposite: restart all cvn-app12 bots * 01:33 AntiComposite: restart all cvn-app10 bots === 2023-08-15 === * 14:44 AntiComposite: reboot cvn-app10 from Horizon, bots dead and not responding to SSH === 2023-08-09 === * 00:07 AntiComposite: add 9 wikis to #cvn-sw (ref [[phab:T332379|T332379]] [[phab:T336115|T336115]] [[phab:T332093|T332093]] [[phab:T332093|T332093]] [[phab:T335987|T335987]] [[phab:T334459|T334459]] [[phab:T333271|T333271]] [[phab:T334740|T334740]] [[phab:T342865|T342865]]) === 2023-08-08 === * 23:46 AntiComposite: drop wo.wikiquote from CVNBot10 (closed) [[phab:T334482|T334482]] === 2023-07-27 === * 18:15 AntiComposite: Kill and restart CVNBot29 on cvn-app12 === 2023-07-06 === * 16:21 AntiComposite: point git repos to gerrit on cvn-app10 * 16:19 AntiComposite: point git repos to gerrit on cvn-app12 * 16:03 AntiComposite: CVNBot v4.0.3 deployed to all bots ([[phab:T327126|T327126]], [[phab:T327127|T327127]]) * 16:01 AntiComposite: Upgrade CVNBot29 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot28 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot27 to v4.0.3 * 15:59 AntiComposite: Upgrade CVNBot26 to v4.0.3 * 15:58 AntiComposite: Upgrade CVNBot25 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot24 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot23 to v4.0.3 * 15:55 AntiComposite: Upgrade CVNBot22 to v4.0.3 * 15:54 AntiComposite: Upgrade CVNBot19 to v4.0.3 * 15:53 AntiComposite: Upgrade CVNBot17 to v4.0.3 * 15:46 AntiComposite: Upgrade CVNBot16 to v4.0.3 * 15:44 AntiComposite: Upgrade CVNBot10 to v4.0.3 * 15:41 AntiComposite: Upgrade CVNBot9 to v4.0.3 * 15:40 AntiComposite: Upgrade CVNBot8 to v4.0.3 * 15:39 AntiComposite: Upgrade CVNBot7 to v4.0.3 * 15:38 AntiComposite: Upgrade CVNBot6 to v4.0.3 * 04:37 AntiComposite: Upgrade CVNBot21 to v4.0.3 * 04:34 AntiComposite: Upgrade CVNBot20 to v4.0.3 * 04:33 AntiComposite: Upgrade CVNBot18 to v4.0.3 * 04:30 AntiComposite: Upgrade CVNBot15 to v4.0.3 * 04:23 AntiComposite: Upgrade CVNBot14 to v4.0.3 * 04:22 AntiComposite: Upgrade CVNBot13 to v4.0.3 * 04:14 AntiComposite: Upgrade CVNBot12 to v4.0.3 * 04:09 AntiComposite: Upgrade CVNBot11 to v4.0.3 * 04:03 AntiComposite: Upgrade CVNBot5 to v4.0.3 * 04:01 AntiComposite: Upgrade CVNBot4 to v4.0.3 * 04:00 AntiComposite: Upgrade CVNBot3 to v4.0.3 * 03:57 AntiComposite: Upgrade CVNBot2 to v4.0.3 * 03:51 AntiComposite: Upgrade CVNBot1 to v4.0.3 === 2023-06-28 === * 02:34 Operator873: /cs flags #cvn-sw Fehufanga voiced === 2023-06-16 === * 22:05 AntiComposite: manually restart cvn-clerkbot === 2023-05-15 === * 14:58 hauskater: Dropped akwiki and nawiki from CVNBot10 as closed wikis. On-wiki lists require an update. === 2023-04-26 === * 20:07 AntiComposite: /cs flags #cvn-mk-scan M4r51n voiced === 2023-04-21 === * 22:12 Operator873: granted voice to Fehufanga in #cvn-simplewikis === 2023-04-14 === * 18:28 AntiComposite: restart cvn-app10 from horizon, bots quit and ssh times out === 2023-03-22 === * 03:33 Operator873: Voiced Tulsi in #cvn-sw -meta -mediawiki -commons -simplewikis === 2023-03-13 === * 19:46 Operator873: CVNBot18 restarted === 2023-03-03 === * 14:45 AntiComposite: /cs flags #cvn-sw-spam COIBot bot === 2023-02-27 === * 22:33 herzog: Loaded gur.wikipedia to SWMT Group 4 (CVNBot9) - [[phab:T327842|T327842]] * 18:04 herzog: Loaded guc.wikipedia to CVNBot9 / Group 4 - [[phab:T326236|T326236]] === 2023-02-02 === * 00:21 ma: Added 12 new wikis to CVNBot<nowiki>{</nowiki>6,7,8<nowiki>}</nowiki>, 4 to each one. Refs.: [[phab:T321283|T321283]] [[phab:T321289|T321289]] [[phab:T321295|T321295]] [[phab:T326139|T326139]] [[phab:T305281|T305281]] [[phab:T310873|T310873]] [[phab:T312215|T312215]] [[phab:T314640|T314640]] [[phab:T314646|T314646]] [[phab:T316457|T316457]] [[phab:T317113|T317113]] [[phab:T319191|T319191]] === 2023-01-30 === * 22:50 Krinkle: Delete cvn-app8 and cvn-app9 instances, ref [[phab:T306066|T306066]] === 2023-01-28 === * 02:51 AntiComposite: /cs flags #cvn-sw Ajraddatz local_op === 2023-01-24 === * 08:54 Krinkle: Delete cvn-apache9, [[phab:T306066|T306066]] * 08:54 Krinkle: Suspend cvn-app8 and cvn-app9 (`pgrep -af cvn` is empty on both), [[phab:T306066|T306066]] === 2023-01-23 === * 16:53 AntiComposite: Deploy {{Gerrit|716e140}} to app12 ([[phab:T306066|T306066]]) * 16:50 AntiComposite: Deploy {{Gerrit|716e140}} to app9 ([[phab:T306066|T306066]]) * 16:29 AntiComposite: Deploy {{Gerrit|442f324}} to app12 ([[phab:T306066|T306066]]) * 16:25 AntiComposite: Deploy {{Gerrit|442f324}} to app9 ([[phab:T306066|T306066]]) * 16:01 AntiComposite: Deploy {{Gerrit|9024b8f}} to app12 ([[phab:T306066|T306066]]) * 15:59 AntiComposite: Deploy {{Gerrit|9024b8f}} to app9 ([[phab:T306066|T306066]]) === 2023-01-22 === * 21:40 AntiComposite: start cvndb-CVNBot14-publish on app10 * 21:07 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app10, starting bots ([[phab:T306066|T306066]]) * 20:56 AntiComposite: disable cvndb-CVNBot14-publish on app8 * 20:51 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app8, stopping bots ([[phab:T306066|T306066]]) * 19:53 AntiComposite: Deploy {{Gerrit|80ea1f5}} to cvn-app10 ([[phab:T306066|T306066]]) * 15:43 AntiComposite: restart all CVNBots on app9 * 15:42 AntiComposite: restart all CVNBots on app8 === 2023-01-17 === * 00:15 Krinkle: Suspend cvn-apache9, replaced by cvn-apache10, ref [[phab:T306066|T306066]] * 00:14 Krinkle: Switch cvn.wmflabs.org from cvn-apache9 to cvn-apache10 === 2023-01-16 === * 00:10 Krinkle: Move https://github.com/countervandalism/cvn-clerkbot to https://github.com/wikimedia/countervandalism-cvn-clerkbot (with HTTP and Git redirect preserved), and replace with Gerrit mirror === 2023-01-15 === * 23:12 Krinkle: Create 'labs-cvn' permission group in Gerrit with CVN staff members * 23:12 Krinkle: Move https://github.com/countervandalism/cvn-api to https://github.com/wikimedia/countervandalism-cvn-api (with HTTP and Git redirect preserved), and replace with Gerrit mirror * 22:02 Krinkle: Switch new cvn.wmcloud.org proxy from cvn-apache9 to cvn-apache10 (Leave main cvn.wmflabs.org as-is for now). === 2023-01-14 === * 21:45 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|4cee27a}}) * 21:22 AntiComposite: move cvn-clerbot back to cvn-app9 (deploy {{Gerrit|371ba2a}}) * 21:10 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|3f3f40f}}) === 2023-01-10 === * 23:22 Krinkle: krinkle@cvn-apache9$ update infrastructure.git, sudo apachectl graceful * 23:20 Krinkle: Create cvn.wmcloud.org web proxy (in addition to cvn.wmflabs.org) === 2023-01-07 === * 20:53 AntiComposite: apply role::labs::lvm::srv only to cvn-apache9, cvn-app8, and cvn-app9 to fix puppet failures on new instances === 2023-01-04 === * 20:47 Krinkle: Allocate new floating IPs to cvn-app10 and cvn-app11 * 20:46 Krinkle: Create new cvn-apache10, cvn-app10, cvn-app11 with Debian 11 Bullseye to replace the old Debian 9.1 Stretch instances * 20:04 taavi: bump floating ip quota from 2 to 4, [[phab:T326269|T326269]] === 2022-12-27 === * 20:11 Frosty873: /cs flags #cvn-meta xaosflux voiced * 20:11 Frosty873: /cs flags #cvn-wp-en xaosflux voiced === 2022-12-23 === * 03:25 AntiComposite: /cs flags #cvn-meta tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-mediawiki tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-sw tryvix1509 voiced === 2022-10-18 === * 23:13 Joan: CVNBot3 restarted (Last message was received on RCReader 62854.814658 seconds ag) === 2022-09-04 === * 22:21 Operator873: /cs flags #cvn-simplewikis Enfcer +AV * 02:20 Operator873: /cs flags #cvn-sw Bot873 +voiced === 2022-08-26 === * 14:09 hauskatze: Loaded pcm.wikipedia and guw.wiktionary to CVNBot8 & 9 respectively {{!}} [[phab:T310880|T310880]] [[phab:T309057|T309057]] === 2022-07-09 === * 16:42 AntiComposite: /cs flags #cvn-commons pandakekok9 voiced === 2022-07-08 === * 21:53 Krinkle: krinkle@horizon.wikimedia.org Add anticomposite as project member and project admin to cloudvps.cvn === 2022-07-01 === * 21:39 Krinkle: cvn-app8: kill CVNBot14.exe and two (!) procs for CVNBot18.exe === 2022-06-25 === * 03:25 AntiComposite: /cs flags #cvn-wp-en PhantomTech voiced === 2022-06-22 === * 21:04 op873: <+CVNBot3> Added: LuchoCR is on es.wikipedia bot list, added by Operator873{{!}}CVN until the end of time ("Mass blockiing P2P-proxies with script") * 20:34 op873: restart CVNBot3 (possibly caused by block flood) * 19:31 op873: restart CVNBot3 === 2022-06-15 === * 18:49 AntiComposite: /cs flags #cvn-wp-en Zppix voiced * 18:48 AntiComposite: /cs flags #cvn-simplewikis Zppix voiced === 2022-05-23 === * 00:24 Joan: Flags +AV were set on Sargento in cvn-wp-es * 00:23 Joan: Flags +AV were set on alhen in cvn-wp-es === 2022-05-19 === * 23:10 Joan: CVNBot3 restarted (Last message was received on RCReader 92593.747667 seconds ago) === 2022-05-11 === * 07:34 Operator873: /cs flags #cvn-wp-en Tamzin voiced === 2022-05-07 === * 17:40 Operator873: /cs flags #cvn-sw koi voiced * 17:39 Operator873: /cs flags #cvn-zh-scan koi voiced === 2022-04-28 === * 03:19 Joan: CVNBot3 restarted (Last message was received on RCReader 75273.332577 seconds ago) === 2022-04-22 === * 15:08 AntiComposite: /cs flags #cvn-meta Bsadowski1 voiced === 2022-04-18 === * 20:44 AntiComposite: /cs flags #cvn-sw Vermont voiced === 2022-04-13 === * 22:40 Operator873: /cs flags #cvn-meta Joan voiced * 22:40 Operator873: /cs flags #cvn-sw Joan voiced * 22:14 Joan: CVNBot3 restarted (Last message was received on RCReader 54942.175428 seconds ago) === 2022-04-07 === * 23:15 Operator873: /cs flags #cvn-wp-hr NovakWatchmen local_op * 23:13 Operator873: voiced Superpes (Superpes15) in #cvn-sw #cvn-sw-spam and #cvn-it-scan === 2022-04-04 === * 17:34 Operator873: Voiced Vermont in #cvn-meta and #cvn-simplewikis /cs flags #cvn-meta Vermont voiced === 2022-03-30 === * 14:33 Joan: CVNBot3 restarted (Last message was received on RCReader 26318.335196 seconds ago) === 2022-03-28 === * 02:38 AntiComposite: /cs flags #cvn-wp-en Bsoyka voiced === 2022-03-21 === * 20:22 Operator873: /cs flags #cvn-simplewikis Bsadowski1 +AfiotvV * 20:17 Operator873: Operator873{{!}}CVN (Operator873) set flags +AVfitv on Bsadowski1 * 20:03 Operator873: Operator873{{!}}CVN (Operator873) set flags +V on Bsadowski1 * 17:04 AntiComposite: /cs flags #cvn-sw Bsadowski1 local_op === 2022-03-15 === * 15:38 Joan: CVNBot3 restarted (Last message was received on RCReader 26424.279343 seconds ago) === 2022-03-14 === * 14:02 Joan: CVNBot3 restarted (Last message was received on RCReader 17096.72183 seconds ago) === 2022-03-12 === * 16:27 Joan: CVNBot3 restarted (Last message was received on RCReader 27236.775673 seconds ago) === 2022-03-11 === * 14:24 Joan: CVNBot3 restarted (Last message was received on RCReader 18853.006849 seconds ago) === 2022-03-10 === * 14:08 Joan: CVNBot3 restarted (Last message was received on RCReader 22518.614282 seconds ago) === 2022-03-08 === * 20:27 AntiComposite: /cs flags #cvn-wp-en Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-simplewikis Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-commons Sarrus voiced === 2022-03-07 === * 16:30 AntiComposite: /cs flags #cvn-meta zabe voiced * 16:25 AntiComposite: /cs flags #cvn-simplewikis DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-meta DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-sw TheresNoTime voiced * 16:07 Krinkle: /cs flags #cvn-staff Operator873 staff * 16:07 Krinkle: /cs flags #cvn-staff AntiComposite staff === 2022-03-05 === * 04:13 Joan: CVNBot3 restarted (Last message was received on RCReader 31573.894101 seconds ago) === 2022-03-03 === * 16:39 Joan: CVNBot3 restarted (Last message was received on RCReader 36578.236383 seconds ago) === 2022-03-01 === * 13:21 Joan: CVNBot3 restarted (Last message was received on RCReader 20646.781861 seconds ago) === 2022-02-15 === * 14:12 Joan: CVNBot3 restarted (Last message was received on RCReader 25001.391103 seconds ago) === 2022-02-13 === * 18:47 andrewbogott: switching to project-local nfs server cvn-nfs-1 * 17:54 andrewbogott: switching to project-local nfs server puppet-diffs-nfs-1 === 2022-02-10 === * 16:17 Joan: CVNBot3 restarted (Last message was received on RCReader 39817.871151 seconds ago) === 2022-02-08 === * 15:51 Joan: CVNBot3 restarted (Last message was received on RCReader 28868.916144 seconds ago) === 2022-02-04 === * 23:59 andrewbogott: accidentally restarted all VMs due to misreading the project purge page. sorry! === 2022-02-02 === * CVN: Several bots restarted after netsplit took nickserv and some bots with it. * 10:26 Krinkle: CVNBot1 bes del delete(?!d) — originally added by huh (reason: "widewuto") === 2022-02-01 === * 15:20 Joan: CVNBot3 restarted (Last message was received on RCReader 26990.323435 seconds ago) === 2022-01-31 === * 17:37 Joan: CVNBot3 restarted (Last message was received on RCReader 48827.882566 seconds ago) === 2022-01-27 === * 16:58 Joan: CVNBot3 restarted (Last message was received on RCReader 29206.852828 seconds ago) === 2022-01-21 === * 16:07 Joan: CVNBot3 restarted (Last message was received on RCReader 22091.557102 seconds ago) === 2022-01-20 === * 18:13 Cam11598: CVNBot15 restarted === 2022-01-19 === * 17:26 Joan: Restarted CVNBot3 (Last message was received on RCReader 28129.031916 seconds ago) === 2022-01-18 === * 16:55 Joan: Restarted CVNBot3 (Last message was received on RCReader 26283.381782 seconds ago) === 2022-01-17 === * 16:33 Joan: Restarted CVNBot3 (#cvn-wp-es) (Last message was received on RCReader 197065.877109 seconds ago) === 2022-01-15 === * 04:56 Cam11598: restarted CVNBOT18 8:55:47 PM <�25B100+ CVNBot18> Last message was received on RCReader 29723.456263 seconds ago === 2022-01-13 === * 01:29 Cam11598: restarted CVNBot2 nickserv issue * 01:29 Cam11598: restarted CVNBot18 - no response from RC feed === 2022-01-09 === * 18:18 Joan: Flags +AV were set on Hasley in cvn-wp-es (sysop at es.wikipedia) * 17:56 Krinkle: /cs flags #cvn-wp-es Joan local_op === 2022-01-07 === * 22:08 hauskatze: CVNBot9 load co.wiktionary wikt:co: * 22:04 hauskatze: CVNBot9 load ban.wikisource s:ban: * 22:04 hauskatze: CVNBot9 load ba.wikibooks b:ba: * 10:51 hauskatze: Loaded alt.wikipedia to Group 4 (CVNBot9) - small wiki not monitored === 2022-01-06 === * 19:42 hauskatze: Loaded ami.wikipedia to CVNBot8 - [[phab:T292421|T292421]] * 19:41 hauskatze: Loaded pwn.wikipedia to CVNBot7 - [[phab:T292419|T292419]] * 19:39 hauskatze: Loaded lmo.wiktionary to CVNBot6 - [[phab:T292076|T292076]] * 19:34 hauskatze: Loaded jv.wikisource to CVNBot6 refs. [[phab:T287319|T287319]] * 19:29 Krinkle: cs flags #cvn-sw hauskatze local_op * 13:57 Krinkle: Krinkle added $a:Cam11598 to the #cvn-staff I list (+I) {{SAL|Project Name=cvn}} <noinclude> ==Archives== * [[Nova Resource:Cvn/SAL/Archive 1|Archive 1]] (2006-2009) * [[Nova Resource:Cvn/SAL/Archive 2|Archive 2]] (2010-2011) * [[Nova Resource:Cvn/SAL/Archive 3|Archive 3]] (2012-2013) * [[Nova Resource:Cvn/SAL/Archive 4|Archive 4]] (2013-2021) (some parts in 2013 are not indexed) [[Category:SAL]]</noinclude> 5f53kzv5nm15o2wwk4qq3hf3qb30dbx 2426634 2426633 2026-06-13T23:03:10Z Stashbot 7414 AntiComposite: CVNBot8 drop & purge ar.wikinews, cs.wikinews, de.wikinews, fi.wikinews, he.wikinews, ru.wikinews, sq.wikinews, sr.wikinews, uk.wikinews (T428622) 2426634 wikitext text/x-wiki === 2026-06-13 === * 23:03 AntiComposite: CVNBot8 drop & purge ar.wikinews, cs.wikinews, de.wikinews, fi.wikinews, he.wikinews, ru.wikinews, sq.wikinews, sr.wikinews, uk.wikinews ([[phab:T428622|T428622]]) * 22:58 AntiComposite: CVNBot7 drop & purge es.wikinews, guw.wikinews, pt.wikinews ([[phab:T428622|T428622]]) * 22:56 AntiComposite: CVNBot6 drop & purge eo.wikinews, fr.wikinews, pl.wikinews, ro.wikinews, sv.wikinews, ta.wikinews ([[phab:T428622|T428622]]) * 22:49 AntiComposite: CVNBot4 drop it.wikinews ([[phab:T428622|T428622]]) === 2026-06-02 === * 01:03 Krinkle: /cs flags #cvn-sw Divinations voiced === 2026-05-26 === * 18:07 AntiComposite: restart all bots -- disconnected === 2026-05-03 === * 13:39 Krinkle: Disable "Admin immed notify" for cvn-private https://lists.wikimedia.org/postorius/lists/cvn-private.lists.wikimedia.org/settings/automatic_responses. We previously removed the sub form but this is no longer supported in mailman3. We require confirm/moderate for new subs, there is no way to turn it off. But we can at least disable the noise. === 2026-04-27 === * 12:22 Krinkle: /cs flags #cvn-meta NathanVeritas voiced === 2026-04-01 === * 13:34 AntiComposite: restart all bots === 2026-02-04 === * 20:33 AntiComposite: Restart all bots === 2025-12-26 === * 15:54 Operator873: /cs flags #cvn-zh-scan nya_1F616EMO voiced === 2025-11-27 === * 13:48 AntiComposite: CVNBot10 load tok.wikipedia tok: ([[phab:T404567|T404567]]) * 13:47 AntiComposite: CVNBot9 load ms.wikiquote q:ms: ([[phab:T404700|T404700]]) * 13:45 AntiComposite: CVNBot8 load min.wikisource s:min: ([[phab:T408343|T408343]]) * 13:44 AntiComposite: CVNBot7 load pcm.wikiquote q:pcm: ([[phab:T408351|T408351]]) * 13:43 AntiComposite: CVNBot6 load tl.wikisource s:tl: ([[phab:T388654|T388654]]) * 13:42 AntiComposite: CVNBot10 load bew.wiktionary wikt:bew: ([[phab:T402134|T402134]]) * 13:41 AntiComposite: CVNBot9 load zgh.wiktionary wikt:zgh: ([[phab:T399785|T399785]]) * 13:40 AntiComposite: CVNBot8 load min.wikibooks b:min: ([[phab:T395499|T395499]]) * 13:38 AntiComposite: CVNBot7 load rki.wikipedia rki: ([[phab:T392499|T392499]]) * 13:37 AntiComposite: CVNBot6 load mad.wikisource s:mad: ([[phab:T391767|T391767]]) === 2025-10-28 === * 23:16 AntiComposite: /cs flags #cvn-commons revi local_op === 2025-08-20 === * 20:35 AntiComposite: CVNBot10 load nup.wikipedia nup: ([[phab:T390711|T390711]]) === 2025-07-11 === * 14:38 AntiComposite: cvn-app10 restart all bots * 11:10 AntiComposite: cvn-app12 restart all bots * 11:09 AntiComposite: cvn-app10 restart all bots === 2025-06-20 === * 20:49 AntiComposite: cvn-app12: restart all bots * 20:48 AntiComposite: cvn-app10: restart all bots === 2025-05-26 === * 17:59 Krinkle: Create cvn-app14 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:59 Krinkle: Create cvn-app13 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:57 Krinkle: Delete cvn-apache10 instance (replaced/shutdown 2 days ago), ref [[phab:T395164|T395164]] === 2025-05-23 === * 20:30 Krinkle: Shut off cvn-apache10, [[phab:T395164|T395164]] * 20:29 Krinkle: Change cvn.wmcloud.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 20:22 Krinkle: Change cvn.wmflabs.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 19:45 Krinkle: Create cvn-apache11 (debian-12.0-bookworm, g4.cores2.ram4.disk20), [[phab:T395164|T395164]]) === 2025-05-16 === * 18:22 Krinkle: Replace outreach.wikipedia with outreach.wikimedia in cvn-sw/CVNBot19 per https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/820245 since the source channel was renamed * 17:30 Krinkle: krinkle@cvn-apache10:/srv/cvn/git/infrastructure$ git pull -- Deploy https://gerrit.wikimedia.org/r/1146724 * 17:30 Krinkle: krinkle@cvn-apache10 Update git remote in /srv/cvn/git/infrastructure from github.com/countervandalism to https://gerrit.wikimedia.org/r/labs/countervandalism/cvn-infrastructure === 2025-04-21 === * 17:22 AntiComposite: Hard reboot cvn-app10, flapping and not responsive to ssh === 2025-03-30 === * 06:55 Krinkle: krinkle@cvn-apache10: Run `sudo chmod 644 /srv/cvn/git/infrastructure/crontab-config/*.cron`, per [[phab:T390415|T390415]] === 2025-03-12 === * 02:18 AntiComposite: CVNBot9 load id.wikivoyage voy:id: ([[phab:T381080|T381080]]) * 02:15 AntiComposite: CVNBot8 load tig.wikipedia tig: ([[phab:T381379|T381379]]) * 02:14 AntiComposite: CVNBot7 load knc.wikipedia knc: ([[phab:T385185|T385185]]) * 02:11 AntiComposite: CVNBot6 load syl.wikipedia syl: ([[phab:T386464|T386464]]) * 02:08 AntiComposite: CVNBot10 load sat.wiktionary wikt:sat: ([[phab:T386631|T386631]]) === 2025-02-03 === * 22:05 AntiComposite: Hard reboot cvn-apache10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ * 21:58 AntiComposite: Hard reboot cvn-app10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ === 2025-01-02 === * 12:46 Krinkle: /cs flags #cvn-wp-en Lordseriouspig voiced * 12:45 Krinkle: /cs flags #cvn-sw Lordseriouspig voiced === 2024-11-23 === * 00:41 AntiComposite: CVNBot9 load ka.wikisource s:ka: ([[phab:T363243|T363243]]) * 00:38 AntiComposite: CVNBot8 load tcy.wikisource s:tcy: ([[phab:T378471|T378471]]) * 00:37 AntiComposite: CVNBot7 load tcy.wiktionary wikt:tcy: ([[phab:T378463|T378463]]) * 00:25 AntiComposite: Upgrade CVNBot29 to v4.0.4 * 00:25 AntiComposite: Upgrade CVNBot28 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot27 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot26 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot25 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot24 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot23 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot22 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot19 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot17 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot16 to v4.0.4 * 00:20 AntiComposite: Upgrade CVNBot10 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot9 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot8 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot7 to v4.0.4 * 00:17 AntiComposite: Upgrade CVNBot6 to v4.0.4 === 2024-11-22 === * 23:52 AntiComposite: Upgrade CVNBot21 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot20 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot18 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot15 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot14 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot13 to v4.0.4 * 23:49 AntiComposite: Upgrade CVNBot12 to v4.0.4 * 23:48 AntiComposite: Upgrade CVNBot11 to v4.0.4 * 23:47 AntiComposite: Upgrade CVNBot5 to v4.0.4 * 23:45 AntiComposite: Upgrade CVNBot3 to v4.0.4 * 23:44 AntiComposite: Upgrade CVNBot2 to v4.0.4 * 23:41 AntiComposite: Upgrade CVNBot1 to v4.0.4 * 23:32 AntiComposite: Upgrade CVNBot4 to v4.0.4 * 17:08 AntiComposite: restart CVNBots on cvn-app12 due to simultaneous RCReader failure 91950.519949 seconds === 2024-11-08 === * 23:24 AntiComposite: Restarting all CVNBots due to simultaneous RCReader disconnect 54323.128318 seconds ago === 2024-10-29 === * 20:56 AntiComposite: add sh.wikipedia to CVNBot6 as #cvn-wp-sh didn't survive the libera migration * 14:22 AntiComposite: restart all CVNBots === 2024-10-28 === * 12:50 AntiComposite: restarting all CVNBots, not coming up cleanly === 2024-10-25 === * 02:23 AntiComposite: add cs.wikivoyage to CVNBot10 ([[phab:T370913|T370913]]) * 02:21 AntiComposite: add bdr.wikipedia to CVNBot9 ([[phab:T371760|T371760]]) * 02:18 AntiComposite: add mos.wikipedia to CVNBot8 ([[phab:T374644|T374644]]) * 02:14 AntiComposite: add kge.wikipedia to CVNBot7 ([[phab:T374815|T374815]]) * 02:11 AntiComposite: add rsk.wikipedia to CVNBot6 ([[phab:T375017|T375017]]) * 02:07 AntiComposite: add mad.wiktionary to CVNBot9 ([[phab:T375024|T375024]]) * 02:06 AntiComposite: add gor.wikiquote to CVNBot8 ([[phab:T375095|T375095]]) * 02:04 AntiComposite: add nr.wikipedia to CVNBot7 ([[phab:T375102|T375102]]) * 02:01 AntiComposite: add tdd.wikipedia to CVNBot6 ([[phab:T375424|T375424]]) * 01:54 AntiComposite: add shn.wikinews to CVNBot9 ([[phab:T375433|T375433]]) * 01:52 AntiComposite: add iba.wikipedia to CVNBot8 ([[phab:T376572|T376572]]) * 01:50 AntiComposite: add bcl.wikisource to CVNBot7 ([[phab:T377088|T377088]]) * 01:47 AntiComposite: add ann.wikipedia to CVNBot6 ([[phab:T377160|T377160]]) * 01:43 AntiComposite: add igl.wikipedia to CVNBot9 ( [[phab:T363263|T363263]] ) * 01:41 AntiComposite: add my.wikisource to CVNBot8 ([[phab:T363270|T363270]]) * 01:39 AntiComposite: add foundation.wikimedia to CVNBot19 * 01:38 AntiComposite: add wikitech.wikimedia to CVNBot19 === 2024-10-24 === * 11:36 AntiComposite: restart all CVNBots === 2024-10-23 === * 17:33 AntiComposite: restart all CVNBots === 2024-07-03 === * 02:00 AntiComposite: add kus.wikipedia to CVNBot7 ([[phab:T360303|T360303]]) * 01:57 AntiComposite: add bew.wikipedia to CVNBot6 ([[phab:T360310|T360310]]) * 01:54 AntiComposite: add ms.wikisource to CVNBot9 ([[phab:T363250|T363250]]) * 01:53 AntiComposite: add kaa.wiktionary to CVNBot8 ([[phab:T363256|T363256]]) * 01:50 AntiComposite: add dtp.wikipedia to CVNBot7 ([[phab:T365230|T365230]]) * 01:48 AntiComposite: add btm.wikipedia to CVNBot6 ([[phab:T368067|T368067]]) * 01:45 AntiComposite: add fon.wikipedia to CVNBot9 ([[phab:T347939|T347939]]) * 01:43 AntiComposite: add blk.wikisource to CVNBot8 ([[phab:T343542|T343542]]) * 01:41 AntiComposite: su.wikisource to CVNBot7 ([[phab:T343548|T343548]]) * 01:39 AntiComposite: add tly.wikipedia to CVNBot6 ([[phab:T345170|T345170]]) * 01:37 AntiComposite: add dga.wikipedia to CVNBot9 ([[phab:T350229|T350229]]) * 01:35 AntiComposite: add bjn.wikiquote to CVNBot8 ([[phab:T350235|T350235]]) * 01:32 AntiComposite: add zgh.wikipedia to CVNBot7 ([[phab:T350241|T350241]]) * 01:28 AntiComposite: add bbc.wikipedia to CVNBot6 ([[phab:T350373|T350373]]) === 2024-06-24 === * 16:40 Krinkle: cvn-clerkbot parts #cvn-unifications (not operated by CVN, renamed to #wikimedia-unifications) === 2024-06-18 === * 08:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs === 2024-03-22 === * 05:30 Operator873: /cs flags #cvn-simplewikis Drummingman +voice === 2024-02-28 === * 21:34 Krinkle: /cs flags #cvn-wp-da Sarrus local_op === 2024-01-11 === * 12:19 AntiComposite: /cs flags #cvn-meta Bsadowski1 local_op === 2023-12-01 === * 15:30 AntiComposite: restart everything after WMCS network outage === 2023-10-07 === * 14:50 AntiComposite: kill 2 CVNBot11 processes and restart, bot not joined to IRC === 2023-09-22 === * 00:06 Op873: /cs flags #cvn-wp-en Oshwah +AV === 2023-09-16 === * 10:33 JackSparrow: /cs flags #cvn-wp-fa Arian_Ar local_op === 2023-09-07 === * 01:35 AntiComposite: restart all cvn-app12 bots * 01:33 AntiComposite: restart all cvn-app10 bots === 2023-08-15 === * 14:44 AntiComposite: reboot cvn-app10 from Horizon, bots dead and not responding to SSH === 2023-08-09 === * 00:07 AntiComposite: add 9 wikis to #cvn-sw (ref [[phab:T332379|T332379]] [[phab:T336115|T336115]] [[phab:T332093|T332093]] [[phab:T332093|T332093]] [[phab:T335987|T335987]] [[phab:T334459|T334459]] [[phab:T333271|T333271]] [[phab:T334740|T334740]] [[phab:T342865|T342865]]) === 2023-08-08 === * 23:46 AntiComposite: drop wo.wikiquote from CVNBot10 (closed) [[phab:T334482|T334482]] === 2023-07-27 === * 18:15 AntiComposite: Kill and restart CVNBot29 on cvn-app12 === 2023-07-06 === * 16:21 AntiComposite: point git repos to gerrit on cvn-app10 * 16:19 AntiComposite: point git repos to gerrit on cvn-app12 * 16:03 AntiComposite: CVNBot v4.0.3 deployed to all bots ([[phab:T327126|T327126]], [[phab:T327127|T327127]]) * 16:01 AntiComposite: Upgrade CVNBot29 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot28 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot27 to v4.0.3 * 15:59 AntiComposite: Upgrade CVNBot26 to v4.0.3 * 15:58 AntiComposite: Upgrade CVNBot25 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot24 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot23 to v4.0.3 * 15:55 AntiComposite: Upgrade CVNBot22 to v4.0.3 * 15:54 AntiComposite: Upgrade CVNBot19 to v4.0.3 * 15:53 AntiComposite: Upgrade CVNBot17 to v4.0.3 * 15:46 AntiComposite: Upgrade CVNBot16 to v4.0.3 * 15:44 AntiComposite: Upgrade CVNBot10 to v4.0.3 * 15:41 AntiComposite: Upgrade CVNBot9 to v4.0.3 * 15:40 AntiComposite: Upgrade CVNBot8 to v4.0.3 * 15:39 AntiComposite: Upgrade CVNBot7 to v4.0.3 * 15:38 AntiComposite: Upgrade CVNBot6 to v4.0.3 * 04:37 AntiComposite: Upgrade CVNBot21 to v4.0.3 * 04:34 AntiComposite: Upgrade CVNBot20 to v4.0.3 * 04:33 AntiComposite: Upgrade CVNBot18 to v4.0.3 * 04:30 AntiComposite: Upgrade CVNBot15 to v4.0.3 * 04:23 AntiComposite: Upgrade CVNBot14 to v4.0.3 * 04:22 AntiComposite: Upgrade CVNBot13 to v4.0.3 * 04:14 AntiComposite: Upgrade CVNBot12 to v4.0.3 * 04:09 AntiComposite: Upgrade CVNBot11 to v4.0.3 * 04:03 AntiComposite: Upgrade CVNBot5 to v4.0.3 * 04:01 AntiComposite: Upgrade CVNBot4 to v4.0.3 * 04:00 AntiComposite: Upgrade CVNBot3 to v4.0.3 * 03:57 AntiComposite: Upgrade CVNBot2 to v4.0.3 * 03:51 AntiComposite: Upgrade CVNBot1 to v4.0.3 === 2023-06-28 === * 02:34 Operator873: /cs flags #cvn-sw Fehufanga voiced === 2023-06-16 === * 22:05 AntiComposite: manually restart cvn-clerkbot === 2023-05-15 === * 14:58 hauskater: Dropped akwiki and nawiki from CVNBot10 as closed wikis. On-wiki lists require an update. === 2023-04-26 === * 20:07 AntiComposite: /cs flags #cvn-mk-scan M4r51n voiced === 2023-04-21 === * 22:12 Operator873: granted voice to Fehufanga in #cvn-simplewikis === 2023-04-14 === * 18:28 AntiComposite: restart cvn-app10 from horizon, bots quit and ssh times out === 2023-03-22 === * 03:33 Operator873: Voiced Tulsi in #cvn-sw -meta -mediawiki -commons -simplewikis === 2023-03-13 === * 19:46 Operator873: CVNBot18 restarted === 2023-03-03 === * 14:45 AntiComposite: /cs flags #cvn-sw-spam COIBot bot === 2023-02-27 === * 22:33 herzog: Loaded gur.wikipedia to SWMT Group 4 (CVNBot9) - [[phab:T327842|T327842]] * 18:04 herzog: Loaded guc.wikipedia to CVNBot9 / Group 4 - [[phab:T326236|T326236]] === 2023-02-02 === * 00:21 ma: Added 12 new wikis to CVNBot<nowiki>{</nowiki>6,7,8<nowiki>}</nowiki>, 4 to each one. Refs.: [[phab:T321283|T321283]] [[phab:T321289|T321289]] [[phab:T321295|T321295]] [[phab:T326139|T326139]] [[phab:T305281|T305281]] [[phab:T310873|T310873]] [[phab:T312215|T312215]] [[phab:T314640|T314640]] [[phab:T314646|T314646]] [[phab:T316457|T316457]] [[phab:T317113|T317113]] [[phab:T319191|T319191]] === 2023-01-30 === * 22:50 Krinkle: Delete cvn-app8 and cvn-app9 instances, ref [[phab:T306066|T306066]] === 2023-01-28 === * 02:51 AntiComposite: /cs flags #cvn-sw Ajraddatz local_op === 2023-01-24 === * 08:54 Krinkle: Delete cvn-apache9, [[phab:T306066|T306066]] * 08:54 Krinkle: Suspend cvn-app8 and cvn-app9 (`pgrep -af cvn` is empty on both), [[phab:T306066|T306066]] === 2023-01-23 === * 16:53 AntiComposite: Deploy {{Gerrit|716e140}} to app12 ([[phab:T306066|T306066]]) * 16:50 AntiComposite: Deploy {{Gerrit|716e140}} to app9 ([[phab:T306066|T306066]]) * 16:29 AntiComposite: Deploy {{Gerrit|442f324}} to app12 ([[phab:T306066|T306066]]) * 16:25 AntiComposite: Deploy {{Gerrit|442f324}} to app9 ([[phab:T306066|T306066]]) * 16:01 AntiComposite: Deploy {{Gerrit|9024b8f}} to app12 ([[phab:T306066|T306066]]) * 15:59 AntiComposite: Deploy {{Gerrit|9024b8f}} to app9 ([[phab:T306066|T306066]]) === 2023-01-22 === * 21:40 AntiComposite: start cvndb-CVNBot14-publish on app10 * 21:07 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app10, starting bots ([[phab:T306066|T306066]]) * 20:56 AntiComposite: disable cvndb-CVNBot14-publish on app8 * 20:51 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app8, stopping bots ([[phab:T306066|T306066]]) * 19:53 AntiComposite: Deploy {{Gerrit|80ea1f5}} to cvn-app10 ([[phab:T306066|T306066]]) * 15:43 AntiComposite: restart all CVNBots on app9 * 15:42 AntiComposite: restart all CVNBots on app8 === 2023-01-17 === * 00:15 Krinkle: Suspend cvn-apache9, replaced by cvn-apache10, ref [[phab:T306066|T306066]] * 00:14 Krinkle: Switch cvn.wmflabs.org from cvn-apache9 to cvn-apache10 === 2023-01-16 === * 00:10 Krinkle: Move https://github.com/countervandalism/cvn-clerkbot to https://github.com/wikimedia/countervandalism-cvn-clerkbot (with HTTP and Git redirect preserved), and replace with Gerrit mirror === 2023-01-15 === * 23:12 Krinkle: Create 'labs-cvn' permission group in Gerrit with CVN staff members * 23:12 Krinkle: Move https://github.com/countervandalism/cvn-api to https://github.com/wikimedia/countervandalism-cvn-api (with HTTP and Git redirect preserved), and replace with Gerrit mirror * 22:02 Krinkle: Switch new cvn.wmcloud.org proxy from cvn-apache9 to cvn-apache10 (Leave main cvn.wmflabs.org as-is for now). === 2023-01-14 === * 21:45 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|4cee27a}}) * 21:22 AntiComposite: move cvn-clerbot back to cvn-app9 (deploy {{Gerrit|371ba2a}}) * 21:10 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|3f3f40f}}) === 2023-01-10 === * 23:22 Krinkle: krinkle@cvn-apache9$ update infrastructure.git, sudo apachectl graceful * 23:20 Krinkle: Create cvn.wmcloud.org web proxy (in addition to cvn.wmflabs.org) === 2023-01-07 === * 20:53 AntiComposite: apply role::labs::lvm::srv only to cvn-apache9, cvn-app8, and cvn-app9 to fix puppet failures on new instances === 2023-01-04 === * 20:47 Krinkle: Allocate new floating IPs to cvn-app10 and cvn-app11 * 20:46 Krinkle: Create new cvn-apache10, cvn-app10, cvn-app11 with Debian 11 Bullseye to replace the old Debian 9.1 Stretch instances * 20:04 taavi: bump floating ip quota from 2 to 4, [[phab:T326269|T326269]] === 2022-12-27 === * 20:11 Frosty873: /cs flags #cvn-meta xaosflux voiced * 20:11 Frosty873: /cs flags #cvn-wp-en xaosflux voiced === 2022-12-23 === * 03:25 AntiComposite: /cs flags #cvn-meta tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-mediawiki tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-sw tryvix1509 voiced === 2022-10-18 === * 23:13 Joan: CVNBot3 restarted (Last message was received on RCReader 62854.814658 seconds ag) === 2022-09-04 === * 22:21 Operator873: /cs flags #cvn-simplewikis Enfcer +AV * 02:20 Operator873: /cs flags #cvn-sw Bot873 +voiced === 2022-08-26 === * 14:09 hauskatze: Loaded pcm.wikipedia and guw.wiktionary to CVNBot8 & 9 respectively {{!}} [[phab:T310880|T310880]] [[phab:T309057|T309057]] === 2022-07-09 === * 16:42 AntiComposite: /cs flags #cvn-commons pandakekok9 voiced === 2022-07-08 === * 21:53 Krinkle: krinkle@horizon.wikimedia.org Add anticomposite as project member and project admin to cloudvps.cvn === 2022-07-01 === * 21:39 Krinkle: cvn-app8: kill CVNBot14.exe and two (!) procs for CVNBot18.exe === 2022-06-25 === * 03:25 AntiComposite: /cs flags #cvn-wp-en PhantomTech voiced === 2022-06-22 === * 21:04 op873: <+CVNBot3> Added: LuchoCR is on es.wikipedia bot list, added by Operator873{{!}}CVN until the end of time ("Mass blockiing P2P-proxies with script") * 20:34 op873: restart CVNBot3 (possibly caused by block flood) * 19:31 op873: restart CVNBot3 === 2022-06-15 === * 18:49 AntiComposite: /cs flags #cvn-wp-en Zppix voiced * 18:48 AntiComposite: /cs flags #cvn-simplewikis Zppix voiced === 2022-05-23 === * 00:24 Joan: Flags +AV were set on Sargento in cvn-wp-es * 00:23 Joan: Flags +AV were set on alhen in cvn-wp-es === 2022-05-19 === * 23:10 Joan: CVNBot3 restarted (Last message was received on RCReader 92593.747667 seconds ago) === 2022-05-11 === * 07:34 Operator873: /cs flags #cvn-wp-en Tamzin voiced === 2022-05-07 === * 17:40 Operator873: /cs flags #cvn-sw koi voiced * 17:39 Operator873: /cs flags #cvn-zh-scan koi voiced === 2022-04-28 === * 03:19 Joan: CVNBot3 restarted (Last message was received on RCReader 75273.332577 seconds ago) === 2022-04-22 === * 15:08 AntiComposite: /cs flags #cvn-meta Bsadowski1 voiced === 2022-04-18 === * 20:44 AntiComposite: /cs flags #cvn-sw Vermont voiced === 2022-04-13 === * 22:40 Operator873: /cs flags #cvn-meta Joan voiced * 22:40 Operator873: /cs flags #cvn-sw Joan voiced * 22:14 Joan: CVNBot3 restarted (Last message was received on RCReader 54942.175428 seconds ago) === 2022-04-07 === * 23:15 Operator873: /cs flags #cvn-wp-hr NovakWatchmen local_op * 23:13 Operator873: voiced Superpes (Superpes15) in #cvn-sw #cvn-sw-spam and #cvn-it-scan === 2022-04-04 === * 17:34 Operator873: Voiced Vermont in #cvn-meta and #cvn-simplewikis /cs flags #cvn-meta Vermont voiced === 2022-03-30 === * 14:33 Joan: CVNBot3 restarted (Last message was received on RCReader 26318.335196 seconds ago) === 2022-03-28 === * 02:38 AntiComposite: /cs flags #cvn-wp-en Bsoyka voiced === 2022-03-21 === * 20:22 Operator873: /cs flags #cvn-simplewikis Bsadowski1 +AfiotvV * 20:17 Operator873: Operator873{{!}}CVN (Operator873) set flags +AVfitv on Bsadowski1 * 20:03 Operator873: Operator873{{!}}CVN (Operator873) set flags +V on Bsadowski1 * 17:04 AntiComposite: /cs flags #cvn-sw Bsadowski1 local_op === 2022-03-15 === * 15:38 Joan: CVNBot3 restarted (Last message was received on RCReader 26424.279343 seconds ago) === 2022-03-14 === * 14:02 Joan: CVNBot3 restarted (Last message was received on RCReader 17096.72183 seconds ago) === 2022-03-12 === * 16:27 Joan: CVNBot3 restarted (Last message was received on RCReader 27236.775673 seconds ago) === 2022-03-11 === * 14:24 Joan: CVNBot3 restarted (Last message was received on RCReader 18853.006849 seconds ago) === 2022-03-10 === * 14:08 Joan: CVNBot3 restarted (Last message was received on RCReader 22518.614282 seconds ago) === 2022-03-08 === * 20:27 AntiComposite: /cs flags #cvn-wp-en Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-simplewikis Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-commons Sarrus voiced === 2022-03-07 === * 16:30 AntiComposite: /cs flags #cvn-meta zabe voiced * 16:25 AntiComposite: /cs flags #cvn-simplewikis DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-meta DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-sw TheresNoTime voiced * 16:07 Krinkle: /cs flags #cvn-staff Operator873 staff * 16:07 Krinkle: /cs flags #cvn-staff AntiComposite staff === 2022-03-05 === * 04:13 Joan: CVNBot3 restarted (Last message was received on RCReader 31573.894101 seconds ago) === 2022-03-03 === * 16:39 Joan: CVNBot3 restarted (Last message was received on RCReader 36578.236383 seconds ago) === 2022-03-01 === * 13:21 Joan: CVNBot3 restarted (Last message was received on RCReader 20646.781861 seconds ago) === 2022-02-15 === * 14:12 Joan: CVNBot3 restarted (Last message was received on RCReader 25001.391103 seconds ago) === 2022-02-13 === * 18:47 andrewbogott: switching to project-local nfs server cvn-nfs-1 * 17:54 andrewbogott: switching to project-local nfs server puppet-diffs-nfs-1 === 2022-02-10 === * 16:17 Joan: CVNBot3 restarted (Last message was received on RCReader 39817.871151 seconds ago) === 2022-02-08 === * 15:51 Joan: CVNBot3 restarted (Last message was received on RCReader 28868.916144 seconds ago) === 2022-02-04 === * 23:59 andrewbogott: accidentally restarted all VMs due to misreading the project purge page. sorry! === 2022-02-02 === * CVN: Several bots restarted after netsplit took nickserv and some bots with it. * 10:26 Krinkle: CVNBot1 bes del delete(?!d) — originally added by huh (reason: "widewuto") === 2022-02-01 === * 15:20 Joan: CVNBot3 restarted (Last message was received on RCReader 26990.323435 seconds ago) === 2022-01-31 === * 17:37 Joan: CVNBot3 restarted (Last message was received on RCReader 48827.882566 seconds ago) === 2022-01-27 === * 16:58 Joan: CVNBot3 restarted (Last message was received on RCReader 29206.852828 seconds ago) === 2022-01-21 === * 16:07 Joan: CVNBot3 restarted (Last message was received on RCReader 22091.557102 seconds ago) === 2022-01-20 === * 18:13 Cam11598: CVNBot15 restarted === 2022-01-19 === * 17:26 Joan: Restarted CVNBot3 (Last message was received on RCReader 28129.031916 seconds ago) === 2022-01-18 === * 16:55 Joan: Restarted CVNBot3 (Last message was received on RCReader 26283.381782 seconds ago) === 2022-01-17 === * 16:33 Joan: Restarted CVNBot3 (#cvn-wp-es) (Last message was received on RCReader 197065.877109 seconds ago) === 2022-01-15 === * 04:56 Cam11598: restarted CVNBOT18 8:55:47 PM <�25B100+ CVNBot18> Last message was received on RCReader 29723.456263 seconds ago === 2022-01-13 === * 01:29 Cam11598: restarted CVNBot2 nickserv issue * 01:29 Cam11598: restarted CVNBot18 - no response from RC feed === 2022-01-09 === * 18:18 Joan: Flags +AV were set on Hasley in cvn-wp-es (sysop at es.wikipedia) * 17:56 Krinkle: /cs flags #cvn-wp-es Joan local_op === 2022-01-07 === * 22:08 hauskatze: CVNBot9 load co.wiktionary wikt:co: * 22:04 hauskatze: CVNBot9 load ban.wikisource s:ban: * 22:04 hauskatze: CVNBot9 load ba.wikibooks b:ba: * 10:51 hauskatze: Loaded alt.wikipedia to Group 4 (CVNBot9) - small wiki not monitored === 2022-01-06 === * 19:42 hauskatze: Loaded ami.wikipedia to CVNBot8 - [[phab:T292421|T292421]] * 19:41 hauskatze: Loaded pwn.wikipedia to CVNBot7 - [[phab:T292419|T292419]] * 19:39 hauskatze: Loaded lmo.wiktionary to CVNBot6 - [[phab:T292076|T292076]] * 19:34 hauskatze: Loaded jv.wikisource to CVNBot6 refs. [[phab:T287319|T287319]] * 19:29 Krinkle: cs flags #cvn-sw hauskatze local_op * 13:57 Krinkle: Krinkle added $a:Cam11598 to the #cvn-staff I list (+I) {{SAL|Project Name=cvn}} <noinclude> ==Archives== * [[Nova Resource:Cvn/SAL/Archive 1|Archive 1]] (2006-2009) * [[Nova Resource:Cvn/SAL/Archive 2|Archive 2]] (2010-2011) * [[Nova Resource:Cvn/SAL/Archive 3|Archive 3]] (2012-2013) * [[Nova Resource:Cvn/SAL/Archive 4|Archive 4]] (2013-2021) (some parts in 2013 are not indexed) [[Category:SAL]]</noinclude> t2nal2kiulrsumoog9gc58wc0jldvpr 2426635 2426634 2026-06-13T23:07:04Z Stashbot 7414 AntiComposite: CVNBot9 drop & purge bs.wikinews, el.wikinews, fa.wikinews, shn.wikinews, zh.wikinews (T428622) 2426635 wikitext text/x-wiki === 2026-06-13 === * 23:07 AntiComposite: CVNBot9 drop & purge bs.wikinews, el.wikinews, fa.wikinews, shn.wikinews, zh.wikinews ([[phab:T428622|T428622]]) * 23:03 AntiComposite: CVNBot8 drop & purge ar.wikinews, cs.wikinews, de.wikinews, fi.wikinews, he.wikinews, ru.wikinews, sq.wikinews, sr.wikinews, uk.wikinews ([[phab:T428622|T428622]]) * 22:58 AntiComposite: CVNBot7 drop & purge es.wikinews, guw.wikinews, pt.wikinews ([[phab:T428622|T428622]]) * 22:56 AntiComposite: CVNBot6 drop & purge eo.wikinews, fr.wikinews, pl.wikinews, ro.wikinews, sv.wikinews, ta.wikinews ([[phab:T428622|T428622]]) * 22:49 AntiComposite: CVNBot4 drop it.wikinews ([[phab:T428622|T428622]]) === 2026-06-02 === * 01:03 Krinkle: /cs flags #cvn-sw Divinations voiced === 2026-05-26 === * 18:07 AntiComposite: restart all bots -- disconnected === 2026-05-03 === * 13:39 Krinkle: Disable "Admin immed notify" for cvn-private https://lists.wikimedia.org/postorius/lists/cvn-private.lists.wikimedia.org/settings/automatic_responses. We previously removed the sub form but this is no longer supported in mailman3. We require confirm/moderate for new subs, there is no way to turn it off. But we can at least disable the noise. === 2026-04-27 === * 12:22 Krinkle: /cs flags #cvn-meta NathanVeritas voiced === 2026-04-01 === * 13:34 AntiComposite: restart all bots === 2026-02-04 === * 20:33 AntiComposite: Restart all bots === 2025-12-26 === * 15:54 Operator873: /cs flags #cvn-zh-scan nya_1F616EMO voiced === 2025-11-27 === * 13:48 AntiComposite: CVNBot10 load tok.wikipedia tok: ([[phab:T404567|T404567]]) * 13:47 AntiComposite: CVNBot9 load ms.wikiquote q:ms: ([[phab:T404700|T404700]]) * 13:45 AntiComposite: CVNBot8 load min.wikisource s:min: ([[phab:T408343|T408343]]) * 13:44 AntiComposite: CVNBot7 load pcm.wikiquote q:pcm: ([[phab:T408351|T408351]]) * 13:43 AntiComposite: CVNBot6 load tl.wikisource s:tl: ([[phab:T388654|T388654]]) * 13:42 AntiComposite: CVNBot10 load bew.wiktionary wikt:bew: ([[phab:T402134|T402134]]) * 13:41 AntiComposite: CVNBot9 load zgh.wiktionary wikt:zgh: ([[phab:T399785|T399785]]) * 13:40 AntiComposite: CVNBot8 load min.wikibooks b:min: ([[phab:T395499|T395499]]) * 13:38 AntiComposite: CVNBot7 load rki.wikipedia rki: ([[phab:T392499|T392499]]) * 13:37 AntiComposite: CVNBot6 load mad.wikisource s:mad: ([[phab:T391767|T391767]]) === 2025-10-28 === * 23:16 AntiComposite: /cs flags #cvn-commons revi local_op === 2025-08-20 === * 20:35 AntiComposite: CVNBot10 load nup.wikipedia nup: ([[phab:T390711|T390711]]) === 2025-07-11 === * 14:38 AntiComposite: cvn-app10 restart all bots * 11:10 AntiComposite: cvn-app12 restart all bots * 11:09 AntiComposite: cvn-app10 restart all bots === 2025-06-20 === * 20:49 AntiComposite: cvn-app12: restart all bots * 20:48 AntiComposite: cvn-app10: restart all bots === 2025-05-26 === * 17:59 Krinkle: Create cvn-app14 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:59 Krinkle: Create cvn-app13 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:57 Krinkle: Delete cvn-apache10 instance (replaced/shutdown 2 days ago), ref [[phab:T395164|T395164]] === 2025-05-23 === * 20:30 Krinkle: Shut off cvn-apache10, [[phab:T395164|T395164]] * 20:29 Krinkle: Change cvn.wmcloud.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 20:22 Krinkle: Change cvn.wmflabs.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 19:45 Krinkle: Create cvn-apache11 (debian-12.0-bookworm, g4.cores2.ram4.disk20), [[phab:T395164|T395164]]) === 2025-05-16 === * 18:22 Krinkle: Replace outreach.wikipedia with outreach.wikimedia in cvn-sw/CVNBot19 per https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/820245 since the source channel was renamed * 17:30 Krinkle: krinkle@cvn-apache10:/srv/cvn/git/infrastructure$ git pull -- Deploy https://gerrit.wikimedia.org/r/1146724 * 17:30 Krinkle: krinkle@cvn-apache10 Update git remote in /srv/cvn/git/infrastructure from github.com/countervandalism to https://gerrit.wikimedia.org/r/labs/countervandalism/cvn-infrastructure === 2025-04-21 === * 17:22 AntiComposite: Hard reboot cvn-app10, flapping and not responsive to ssh === 2025-03-30 === * 06:55 Krinkle: krinkle@cvn-apache10: Run `sudo chmod 644 /srv/cvn/git/infrastructure/crontab-config/*.cron`, per [[phab:T390415|T390415]] === 2025-03-12 === * 02:18 AntiComposite: CVNBot9 load id.wikivoyage voy:id: ([[phab:T381080|T381080]]) * 02:15 AntiComposite: CVNBot8 load tig.wikipedia tig: ([[phab:T381379|T381379]]) * 02:14 AntiComposite: CVNBot7 load knc.wikipedia knc: ([[phab:T385185|T385185]]) * 02:11 AntiComposite: CVNBot6 load syl.wikipedia syl: ([[phab:T386464|T386464]]) * 02:08 AntiComposite: CVNBot10 load sat.wiktionary wikt:sat: ([[phab:T386631|T386631]]) === 2025-02-03 === * 22:05 AntiComposite: Hard reboot cvn-apache10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ * 21:58 AntiComposite: Hard reboot cvn-app10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ === 2025-01-02 === * 12:46 Krinkle: /cs flags #cvn-wp-en Lordseriouspig voiced * 12:45 Krinkle: /cs flags #cvn-sw Lordseriouspig voiced === 2024-11-23 === * 00:41 AntiComposite: CVNBot9 load ka.wikisource s:ka: ([[phab:T363243|T363243]]) * 00:38 AntiComposite: CVNBot8 load tcy.wikisource s:tcy: ([[phab:T378471|T378471]]) * 00:37 AntiComposite: CVNBot7 load tcy.wiktionary wikt:tcy: ([[phab:T378463|T378463]]) * 00:25 AntiComposite: Upgrade CVNBot29 to v4.0.4 * 00:25 AntiComposite: Upgrade CVNBot28 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot27 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot26 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot25 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot24 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot23 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot22 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot19 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot17 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot16 to v4.0.4 * 00:20 AntiComposite: Upgrade CVNBot10 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot9 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot8 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot7 to v4.0.4 * 00:17 AntiComposite: Upgrade CVNBot6 to v4.0.4 === 2024-11-22 === * 23:52 AntiComposite: Upgrade CVNBot21 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot20 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot18 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot15 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot14 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot13 to v4.0.4 * 23:49 AntiComposite: Upgrade CVNBot12 to v4.0.4 * 23:48 AntiComposite: Upgrade CVNBot11 to v4.0.4 * 23:47 AntiComposite: Upgrade CVNBot5 to v4.0.4 * 23:45 AntiComposite: Upgrade CVNBot3 to v4.0.4 * 23:44 AntiComposite: Upgrade CVNBot2 to v4.0.4 * 23:41 AntiComposite: Upgrade CVNBot1 to v4.0.4 * 23:32 AntiComposite: Upgrade CVNBot4 to v4.0.4 * 17:08 AntiComposite: restart CVNBots on cvn-app12 due to simultaneous RCReader failure 91950.519949 seconds === 2024-11-08 === * 23:24 AntiComposite: Restarting all CVNBots due to simultaneous RCReader disconnect 54323.128318 seconds ago === 2024-10-29 === * 20:56 AntiComposite: add sh.wikipedia to CVNBot6 as #cvn-wp-sh didn't survive the libera migration * 14:22 AntiComposite: restart all CVNBots === 2024-10-28 === * 12:50 AntiComposite: restarting all CVNBots, not coming up cleanly === 2024-10-25 === * 02:23 AntiComposite: add cs.wikivoyage to CVNBot10 ([[phab:T370913|T370913]]) * 02:21 AntiComposite: add bdr.wikipedia to CVNBot9 ([[phab:T371760|T371760]]) * 02:18 AntiComposite: add mos.wikipedia to CVNBot8 ([[phab:T374644|T374644]]) * 02:14 AntiComposite: add kge.wikipedia to CVNBot7 ([[phab:T374815|T374815]]) * 02:11 AntiComposite: add rsk.wikipedia to CVNBot6 ([[phab:T375017|T375017]]) * 02:07 AntiComposite: add mad.wiktionary to CVNBot9 ([[phab:T375024|T375024]]) * 02:06 AntiComposite: add gor.wikiquote to CVNBot8 ([[phab:T375095|T375095]]) * 02:04 AntiComposite: add nr.wikipedia to CVNBot7 ([[phab:T375102|T375102]]) * 02:01 AntiComposite: add tdd.wikipedia to CVNBot6 ([[phab:T375424|T375424]]) * 01:54 AntiComposite: add shn.wikinews to CVNBot9 ([[phab:T375433|T375433]]) * 01:52 AntiComposite: add iba.wikipedia to CVNBot8 ([[phab:T376572|T376572]]) * 01:50 AntiComposite: add bcl.wikisource to CVNBot7 ([[phab:T377088|T377088]]) * 01:47 AntiComposite: add ann.wikipedia to CVNBot6 ([[phab:T377160|T377160]]) * 01:43 AntiComposite: add igl.wikipedia to CVNBot9 ( [[phab:T363263|T363263]] ) * 01:41 AntiComposite: add my.wikisource to CVNBot8 ([[phab:T363270|T363270]]) * 01:39 AntiComposite: add foundation.wikimedia to CVNBot19 * 01:38 AntiComposite: add wikitech.wikimedia to CVNBot19 === 2024-10-24 === * 11:36 AntiComposite: restart all CVNBots === 2024-10-23 === * 17:33 AntiComposite: restart all CVNBots === 2024-07-03 === * 02:00 AntiComposite: add kus.wikipedia to CVNBot7 ([[phab:T360303|T360303]]) * 01:57 AntiComposite: add bew.wikipedia to CVNBot6 ([[phab:T360310|T360310]]) * 01:54 AntiComposite: add ms.wikisource to CVNBot9 ([[phab:T363250|T363250]]) * 01:53 AntiComposite: add kaa.wiktionary to CVNBot8 ([[phab:T363256|T363256]]) * 01:50 AntiComposite: add dtp.wikipedia to CVNBot7 ([[phab:T365230|T365230]]) * 01:48 AntiComposite: add btm.wikipedia to CVNBot6 ([[phab:T368067|T368067]]) * 01:45 AntiComposite: add fon.wikipedia to CVNBot9 ([[phab:T347939|T347939]]) * 01:43 AntiComposite: add blk.wikisource to CVNBot8 ([[phab:T343542|T343542]]) * 01:41 AntiComposite: su.wikisource to CVNBot7 ([[phab:T343548|T343548]]) * 01:39 AntiComposite: add tly.wikipedia to CVNBot6 ([[phab:T345170|T345170]]) * 01:37 AntiComposite: add dga.wikipedia to CVNBot9 ([[phab:T350229|T350229]]) * 01:35 AntiComposite: add bjn.wikiquote to CVNBot8 ([[phab:T350235|T350235]]) * 01:32 AntiComposite: add zgh.wikipedia to CVNBot7 ([[phab:T350241|T350241]]) * 01:28 AntiComposite: add bbc.wikipedia to CVNBot6 ([[phab:T350373|T350373]]) === 2024-06-24 === * 16:40 Krinkle: cvn-clerkbot parts #cvn-unifications (not operated by CVN, renamed to #wikimedia-unifications) === 2024-06-18 === * 08:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs === 2024-03-22 === * 05:30 Operator873: /cs flags #cvn-simplewikis Drummingman +voice === 2024-02-28 === * 21:34 Krinkle: /cs flags #cvn-wp-da Sarrus local_op === 2024-01-11 === * 12:19 AntiComposite: /cs flags #cvn-meta Bsadowski1 local_op === 2023-12-01 === * 15:30 AntiComposite: restart everything after WMCS network outage === 2023-10-07 === * 14:50 AntiComposite: kill 2 CVNBot11 processes and restart, bot not joined to IRC === 2023-09-22 === * 00:06 Op873: /cs flags #cvn-wp-en Oshwah +AV === 2023-09-16 === * 10:33 JackSparrow: /cs flags #cvn-wp-fa Arian_Ar local_op === 2023-09-07 === * 01:35 AntiComposite: restart all cvn-app12 bots * 01:33 AntiComposite: restart all cvn-app10 bots === 2023-08-15 === * 14:44 AntiComposite: reboot cvn-app10 from Horizon, bots dead and not responding to SSH === 2023-08-09 === * 00:07 AntiComposite: add 9 wikis to #cvn-sw (ref [[phab:T332379|T332379]] [[phab:T336115|T336115]] [[phab:T332093|T332093]] [[phab:T332093|T332093]] [[phab:T335987|T335987]] [[phab:T334459|T334459]] [[phab:T333271|T333271]] [[phab:T334740|T334740]] [[phab:T342865|T342865]]) === 2023-08-08 === * 23:46 AntiComposite: drop wo.wikiquote from CVNBot10 (closed) [[phab:T334482|T334482]] === 2023-07-27 === * 18:15 AntiComposite: Kill and restart CVNBot29 on cvn-app12 === 2023-07-06 === * 16:21 AntiComposite: point git repos to gerrit on cvn-app10 * 16:19 AntiComposite: point git repos to gerrit on cvn-app12 * 16:03 AntiComposite: CVNBot v4.0.3 deployed to all bots ([[phab:T327126|T327126]], [[phab:T327127|T327127]]) * 16:01 AntiComposite: Upgrade CVNBot29 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot28 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot27 to v4.0.3 * 15:59 AntiComposite: Upgrade CVNBot26 to v4.0.3 * 15:58 AntiComposite: Upgrade CVNBot25 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot24 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot23 to v4.0.3 * 15:55 AntiComposite: Upgrade CVNBot22 to v4.0.3 * 15:54 AntiComposite: Upgrade CVNBot19 to v4.0.3 * 15:53 AntiComposite: Upgrade CVNBot17 to v4.0.3 * 15:46 AntiComposite: Upgrade CVNBot16 to v4.0.3 * 15:44 AntiComposite: Upgrade CVNBot10 to v4.0.3 * 15:41 AntiComposite: Upgrade CVNBot9 to v4.0.3 * 15:40 AntiComposite: Upgrade CVNBot8 to v4.0.3 * 15:39 AntiComposite: Upgrade CVNBot7 to v4.0.3 * 15:38 AntiComposite: Upgrade CVNBot6 to v4.0.3 * 04:37 AntiComposite: Upgrade CVNBot21 to v4.0.3 * 04:34 AntiComposite: Upgrade CVNBot20 to v4.0.3 * 04:33 AntiComposite: Upgrade CVNBot18 to v4.0.3 * 04:30 AntiComposite: Upgrade CVNBot15 to v4.0.3 * 04:23 AntiComposite: Upgrade CVNBot14 to v4.0.3 * 04:22 AntiComposite: Upgrade CVNBot13 to v4.0.3 * 04:14 AntiComposite: Upgrade CVNBot12 to v4.0.3 * 04:09 AntiComposite: Upgrade CVNBot11 to v4.0.3 * 04:03 AntiComposite: Upgrade CVNBot5 to v4.0.3 * 04:01 AntiComposite: Upgrade CVNBot4 to v4.0.3 * 04:00 AntiComposite: Upgrade CVNBot3 to v4.0.3 * 03:57 AntiComposite: Upgrade CVNBot2 to v4.0.3 * 03:51 AntiComposite: Upgrade CVNBot1 to v4.0.3 === 2023-06-28 === * 02:34 Operator873: /cs flags #cvn-sw Fehufanga voiced === 2023-06-16 === * 22:05 AntiComposite: manually restart cvn-clerkbot === 2023-05-15 === * 14:58 hauskater: Dropped akwiki and nawiki from CVNBot10 as closed wikis. On-wiki lists require an update. === 2023-04-26 === * 20:07 AntiComposite: /cs flags #cvn-mk-scan M4r51n voiced === 2023-04-21 === * 22:12 Operator873: granted voice to Fehufanga in #cvn-simplewikis === 2023-04-14 === * 18:28 AntiComposite: restart cvn-app10 from horizon, bots quit and ssh times out === 2023-03-22 === * 03:33 Operator873: Voiced Tulsi in #cvn-sw -meta -mediawiki -commons -simplewikis === 2023-03-13 === * 19:46 Operator873: CVNBot18 restarted === 2023-03-03 === * 14:45 AntiComposite: /cs flags #cvn-sw-spam COIBot bot === 2023-02-27 === * 22:33 herzog: Loaded gur.wikipedia to SWMT Group 4 (CVNBot9) - [[phab:T327842|T327842]] * 18:04 herzog: Loaded guc.wikipedia to CVNBot9 / Group 4 - [[phab:T326236|T326236]] === 2023-02-02 === * 00:21 ma: Added 12 new wikis to CVNBot<nowiki>{</nowiki>6,7,8<nowiki>}</nowiki>, 4 to each one. Refs.: [[phab:T321283|T321283]] [[phab:T321289|T321289]] [[phab:T321295|T321295]] [[phab:T326139|T326139]] [[phab:T305281|T305281]] [[phab:T310873|T310873]] [[phab:T312215|T312215]] [[phab:T314640|T314640]] [[phab:T314646|T314646]] [[phab:T316457|T316457]] [[phab:T317113|T317113]] [[phab:T319191|T319191]] === 2023-01-30 === * 22:50 Krinkle: Delete cvn-app8 and cvn-app9 instances, ref [[phab:T306066|T306066]] === 2023-01-28 === * 02:51 AntiComposite: /cs flags #cvn-sw Ajraddatz local_op === 2023-01-24 === * 08:54 Krinkle: Delete cvn-apache9, [[phab:T306066|T306066]] * 08:54 Krinkle: Suspend cvn-app8 and cvn-app9 (`pgrep -af cvn` is empty on both), [[phab:T306066|T306066]] === 2023-01-23 === * 16:53 AntiComposite: Deploy {{Gerrit|716e140}} to app12 ([[phab:T306066|T306066]]) * 16:50 AntiComposite: Deploy {{Gerrit|716e140}} to app9 ([[phab:T306066|T306066]]) * 16:29 AntiComposite: Deploy {{Gerrit|442f324}} to app12 ([[phab:T306066|T306066]]) * 16:25 AntiComposite: Deploy {{Gerrit|442f324}} to app9 ([[phab:T306066|T306066]]) * 16:01 AntiComposite: Deploy {{Gerrit|9024b8f}} to app12 ([[phab:T306066|T306066]]) * 15:59 AntiComposite: Deploy {{Gerrit|9024b8f}} to app9 ([[phab:T306066|T306066]]) === 2023-01-22 === * 21:40 AntiComposite: start cvndb-CVNBot14-publish on app10 * 21:07 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app10, starting bots ([[phab:T306066|T306066]]) * 20:56 AntiComposite: disable cvndb-CVNBot14-publish on app8 * 20:51 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app8, stopping bots ([[phab:T306066|T306066]]) * 19:53 AntiComposite: Deploy {{Gerrit|80ea1f5}} to cvn-app10 ([[phab:T306066|T306066]]) * 15:43 AntiComposite: restart all CVNBots on app9 * 15:42 AntiComposite: restart all CVNBots on app8 === 2023-01-17 === * 00:15 Krinkle: Suspend cvn-apache9, replaced by cvn-apache10, ref [[phab:T306066|T306066]] * 00:14 Krinkle: Switch cvn.wmflabs.org from cvn-apache9 to cvn-apache10 === 2023-01-16 === * 00:10 Krinkle: Move https://github.com/countervandalism/cvn-clerkbot to https://github.com/wikimedia/countervandalism-cvn-clerkbot (with HTTP and Git redirect preserved), and replace with Gerrit mirror === 2023-01-15 === * 23:12 Krinkle: Create 'labs-cvn' permission group in Gerrit with CVN staff members * 23:12 Krinkle: Move https://github.com/countervandalism/cvn-api to https://github.com/wikimedia/countervandalism-cvn-api (with HTTP and Git redirect preserved), and replace with Gerrit mirror * 22:02 Krinkle: Switch new cvn.wmcloud.org proxy from cvn-apache9 to cvn-apache10 (Leave main cvn.wmflabs.org as-is for now). === 2023-01-14 === * 21:45 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|4cee27a}}) * 21:22 AntiComposite: move cvn-clerbot back to cvn-app9 (deploy {{Gerrit|371ba2a}}) * 21:10 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|3f3f40f}}) === 2023-01-10 === * 23:22 Krinkle: krinkle@cvn-apache9$ update infrastructure.git, sudo apachectl graceful * 23:20 Krinkle: Create cvn.wmcloud.org web proxy (in addition to cvn.wmflabs.org) === 2023-01-07 === * 20:53 AntiComposite: apply role::labs::lvm::srv only to cvn-apache9, cvn-app8, and cvn-app9 to fix puppet failures on new instances === 2023-01-04 === * 20:47 Krinkle: Allocate new floating IPs to cvn-app10 and cvn-app11 * 20:46 Krinkle: Create new cvn-apache10, cvn-app10, cvn-app11 with Debian 11 Bullseye to replace the old Debian 9.1 Stretch instances * 20:04 taavi: bump floating ip quota from 2 to 4, [[phab:T326269|T326269]] === 2022-12-27 === * 20:11 Frosty873: /cs flags #cvn-meta xaosflux voiced * 20:11 Frosty873: /cs flags #cvn-wp-en xaosflux voiced === 2022-12-23 === * 03:25 AntiComposite: /cs flags #cvn-meta tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-mediawiki tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-sw tryvix1509 voiced === 2022-10-18 === * 23:13 Joan: CVNBot3 restarted (Last message was received on RCReader 62854.814658 seconds ag) === 2022-09-04 === * 22:21 Operator873: /cs flags #cvn-simplewikis Enfcer +AV * 02:20 Operator873: /cs flags #cvn-sw Bot873 +voiced === 2022-08-26 === * 14:09 hauskatze: Loaded pcm.wikipedia and guw.wiktionary to CVNBot8 & 9 respectively {{!}} [[phab:T310880|T310880]] [[phab:T309057|T309057]] === 2022-07-09 === * 16:42 AntiComposite: /cs flags #cvn-commons pandakekok9 voiced === 2022-07-08 === * 21:53 Krinkle: krinkle@horizon.wikimedia.org Add anticomposite as project member and project admin to cloudvps.cvn === 2022-07-01 === * 21:39 Krinkle: cvn-app8: kill CVNBot14.exe and two (!) procs for CVNBot18.exe === 2022-06-25 === * 03:25 AntiComposite: /cs flags #cvn-wp-en PhantomTech voiced === 2022-06-22 === * 21:04 op873: <+CVNBot3> Added: LuchoCR is on es.wikipedia bot list, added by Operator873{{!}}CVN until the end of time ("Mass blockiing P2P-proxies with script") * 20:34 op873: restart CVNBot3 (possibly caused by block flood) * 19:31 op873: restart CVNBot3 === 2022-06-15 === * 18:49 AntiComposite: /cs flags #cvn-wp-en Zppix voiced * 18:48 AntiComposite: /cs flags #cvn-simplewikis Zppix voiced === 2022-05-23 === * 00:24 Joan: Flags +AV were set on Sargento in cvn-wp-es * 00:23 Joan: Flags +AV were set on alhen in cvn-wp-es === 2022-05-19 === * 23:10 Joan: CVNBot3 restarted (Last message was received on RCReader 92593.747667 seconds ago) === 2022-05-11 === * 07:34 Operator873: /cs flags #cvn-wp-en Tamzin voiced === 2022-05-07 === * 17:40 Operator873: /cs flags #cvn-sw koi voiced * 17:39 Operator873: /cs flags #cvn-zh-scan koi voiced === 2022-04-28 === * 03:19 Joan: CVNBot3 restarted (Last message was received on RCReader 75273.332577 seconds ago) === 2022-04-22 === * 15:08 AntiComposite: /cs flags #cvn-meta Bsadowski1 voiced === 2022-04-18 === * 20:44 AntiComposite: /cs flags #cvn-sw Vermont voiced === 2022-04-13 === * 22:40 Operator873: /cs flags #cvn-meta Joan voiced * 22:40 Operator873: /cs flags #cvn-sw Joan voiced * 22:14 Joan: CVNBot3 restarted (Last message was received on RCReader 54942.175428 seconds ago) === 2022-04-07 === * 23:15 Operator873: /cs flags #cvn-wp-hr NovakWatchmen local_op * 23:13 Operator873: voiced Superpes (Superpes15) in #cvn-sw #cvn-sw-spam and #cvn-it-scan === 2022-04-04 === * 17:34 Operator873: Voiced Vermont in #cvn-meta and #cvn-simplewikis /cs flags #cvn-meta Vermont voiced === 2022-03-30 === * 14:33 Joan: CVNBot3 restarted (Last message was received on RCReader 26318.335196 seconds ago) === 2022-03-28 === * 02:38 AntiComposite: /cs flags #cvn-wp-en Bsoyka voiced === 2022-03-21 === * 20:22 Operator873: /cs flags #cvn-simplewikis Bsadowski1 +AfiotvV * 20:17 Operator873: Operator873{{!}}CVN (Operator873) set flags +AVfitv on Bsadowski1 * 20:03 Operator873: Operator873{{!}}CVN (Operator873) set flags +V on Bsadowski1 * 17:04 AntiComposite: /cs flags #cvn-sw Bsadowski1 local_op === 2022-03-15 === * 15:38 Joan: CVNBot3 restarted (Last message was received on RCReader 26424.279343 seconds ago) === 2022-03-14 === * 14:02 Joan: CVNBot3 restarted (Last message was received on RCReader 17096.72183 seconds ago) === 2022-03-12 === * 16:27 Joan: CVNBot3 restarted (Last message was received on RCReader 27236.775673 seconds ago) === 2022-03-11 === * 14:24 Joan: CVNBot3 restarted (Last message was received on RCReader 18853.006849 seconds ago) === 2022-03-10 === * 14:08 Joan: CVNBot3 restarted (Last message was received on RCReader 22518.614282 seconds ago) === 2022-03-08 === * 20:27 AntiComposite: /cs flags #cvn-wp-en Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-simplewikis Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-commons Sarrus voiced === 2022-03-07 === * 16:30 AntiComposite: /cs flags #cvn-meta zabe voiced * 16:25 AntiComposite: /cs flags #cvn-simplewikis DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-meta DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-sw TheresNoTime voiced * 16:07 Krinkle: /cs flags #cvn-staff Operator873 staff * 16:07 Krinkle: /cs flags #cvn-staff AntiComposite staff === 2022-03-05 === * 04:13 Joan: CVNBot3 restarted (Last message was received on RCReader 31573.894101 seconds ago) === 2022-03-03 === * 16:39 Joan: CVNBot3 restarted (Last message was received on RCReader 36578.236383 seconds ago) === 2022-03-01 === * 13:21 Joan: CVNBot3 restarted (Last message was received on RCReader 20646.781861 seconds ago) === 2022-02-15 === * 14:12 Joan: CVNBot3 restarted (Last message was received on RCReader 25001.391103 seconds ago) === 2022-02-13 === * 18:47 andrewbogott: switching to project-local nfs server cvn-nfs-1 * 17:54 andrewbogott: switching to project-local nfs server puppet-diffs-nfs-1 === 2022-02-10 === * 16:17 Joan: CVNBot3 restarted (Last message was received on RCReader 39817.871151 seconds ago) === 2022-02-08 === * 15:51 Joan: CVNBot3 restarted (Last message was received on RCReader 28868.916144 seconds ago) === 2022-02-04 === * 23:59 andrewbogott: accidentally restarted all VMs due to misreading the project purge page. sorry! === 2022-02-02 === * CVN: Several bots restarted after netsplit took nickserv and some bots with it. * 10:26 Krinkle: CVNBot1 bes del delete(?!d) — originally added by huh (reason: "widewuto") === 2022-02-01 === * 15:20 Joan: CVNBot3 restarted (Last message was received on RCReader 26990.323435 seconds ago) === 2022-01-31 === * 17:37 Joan: CVNBot3 restarted (Last message was received on RCReader 48827.882566 seconds ago) === 2022-01-27 === * 16:58 Joan: CVNBot3 restarted (Last message was received on RCReader 29206.852828 seconds ago) === 2022-01-21 === * 16:07 Joan: CVNBot3 restarted (Last message was received on RCReader 22091.557102 seconds ago) === 2022-01-20 === * 18:13 Cam11598: CVNBot15 restarted === 2022-01-19 === * 17:26 Joan: Restarted CVNBot3 (Last message was received on RCReader 28129.031916 seconds ago) === 2022-01-18 === * 16:55 Joan: Restarted CVNBot3 (Last message was received on RCReader 26283.381782 seconds ago) === 2022-01-17 === * 16:33 Joan: Restarted CVNBot3 (#cvn-wp-es) (Last message was received on RCReader 197065.877109 seconds ago) === 2022-01-15 === * 04:56 Cam11598: restarted CVNBOT18 8:55:47 PM <�25B100+ CVNBot18> Last message was received on RCReader 29723.456263 seconds ago === 2022-01-13 === * 01:29 Cam11598: restarted CVNBot2 nickserv issue * 01:29 Cam11598: restarted CVNBot18 - no response from RC feed === 2022-01-09 === * 18:18 Joan: Flags +AV were set on Hasley in cvn-wp-es (sysop at es.wikipedia) * 17:56 Krinkle: /cs flags #cvn-wp-es Joan local_op === 2022-01-07 === * 22:08 hauskatze: CVNBot9 load co.wiktionary wikt:co: * 22:04 hauskatze: CVNBot9 load ban.wikisource s:ban: * 22:04 hauskatze: CVNBot9 load ba.wikibooks b:ba: * 10:51 hauskatze: Loaded alt.wikipedia to Group 4 (CVNBot9) - small wiki not monitored === 2022-01-06 === * 19:42 hauskatze: Loaded ami.wikipedia to CVNBot8 - [[phab:T292421|T292421]] * 19:41 hauskatze: Loaded pwn.wikipedia to CVNBot7 - [[phab:T292419|T292419]] * 19:39 hauskatze: Loaded lmo.wiktionary to CVNBot6 - [[phab:T292076|T292076]] * 19:34 hauskatze: Loaded jv.wikisource to CVNBot6 refs. [[phab:T287319|T287319]] * 19:29 Krinkle: cs flags #cvn-sw hauskatze local_op * 13:57 Krinkle: Krinkle added $a:Cam11598 to the #cvn-staff I list (+I) {{SAL|Project Name=cvn}} <noinclude> ==Archives== * [[Nova Resource:Cvn/SAL/Archive 1|Archive 1]] (2006-2009) * [[Nova Resource:Cvn/SAL/Archive 2|Archive 2]] (2010-2011) * [[Nova Resource:Cvn/SAL/Archive 3|Archive 3]] (2012-2013) * [[Nova Resource:Cvn/SAL/Archive 4|Archive 4]] (2013-2021) (some parts in 2013 are not indexed) [[Category:SAL]]</noinclude> fp9l0iv52qc830hs25dul550yon91dz 2426636 2426635 2026-06-13T23:11:44Z Stashbot 7414 AntiComposite: CVNBot10 drop & purge ca.wikinews, ko.wikinews, no.wikinews (T428622) 2426636 wikitext text/x-wiki === 2026-06-13 === * 23:11 AntiComposite: CVNBot10 drop & purge ca.wikinews, ko.wikinews, no.wikinews ([[phab:T428622|T428622]]) * 23:07 AntiComposite: CVNBot9 drop & purge bs.wikinews, el.wikinews, fa.wikinews, shn.wikinews, zh.wikinews ([[phab:T428622|T428622]]) * 23:03 AntiComposite: CVNBot8 drop & purge ar.wikinews, cs.wikinews, de.wikinews, fi.wikinews, he.wikinews, ru.wikinews, sq.wikinews, sr.wikinews, uk.wikinews ([[phab:T428622|T428622]]) * 22:58 AntiComposite: CVNBot7 drop & purge es.wikinews, guw.wikinews, pt.wikinews ([[phab:T428622|T428622]]) * 22:56 AntiComposite: CVNBot6 drop & purge eo.wikinews, fr.wikinews, pl.wikinews, ro.wikinews, sv.wikinews, ta.wikinews ([[phab:T428622|T428622]]) * 22:49 AntiComposite: CVNBot4 drop it.wikinews ([[phab:T428622|T428622]]) === 2026-06-02 === * 01:03 Krinkle: /cs flags #cvn-sw Divinations voiced === 2026-05-26 === * 18:07 AntiComposite: restart all bots -- disconnected === 2026-05-03 === * 13:39 Krinkle: Disable "Admin immed notify" for cvn-private https://lists.wikimedia.org/postorius/lists/cvn-private.lists.wikimedia.org/settings/automatic_responses. We previously removed the sub form but this is no longer supported in mailman3. We require confirm/moderate for new subs, there is no way to turn it off. But we can at least disable the noise. === 2026-04-27 === * 12:22 Krinkle: /cs flags #cvn-meta NathanVeritas voiced === 2026-04-01 === * 13:34 AntiComposite: restart all bots === 2026-02-04 === * 20:33 AntiComposite: Restart all bots === 2025-12-26 === * 15:54 Operator873: /cs flags #cvn-zh-scan nya_1F616EMO voiced === 2025-11-27 === * 13:48 AntiComposite: CVNBot10 load tok.wikipedia tok: ([[phab:T404567|T404567]]) * 13:47 AntiComposite: CVNBot9 load ms.wikiquote q:ms: ([[phab:T404700|T404700]]) * 13:45 AntiComposite: CVNBot8 load min.wikisource s:min: ([[phab:T408343|T408343]]) * 13:44 AntiComposite: CVNBot7 load pcm.wikiquote q:pcm: ([[phab:T408351|T408351]]) * 13:43 AntiComposite: CVNBot6 load tl.wikisource s:tl: ([[phab:T388654|T388654]]) * 13:42 AntiComposite: CVNBot10 load bew.wiktionary wikt:bew: ([[phab:T402134|T402134]]) * 13:41 AntiComposite: CVNBot9 load zgh.wiktionary wikt:zgh: ([[phab:T399785|T399785]]) * 13:40 AntiComposite: CVNBot8 load min.wikibooks b:min: ([[phab:T395499|T395499]]) * 13:38 AntiComposite: CVNBot7 load rki.wikipedia rki: ([[phab:T392499|T392499]]) * 13:37 AntiComposite: CVNBot6 load mad.wikisource s:mad: ([[phab:T391767|T391767]]) === 2025-10-28 === * 23:16 AntiComposite: /cs flags #cvn-commons revi local_op === 2025-08-20 === * 20:35 AntiComposite: CVNBot10 load nup.wikipedia nup: ([[phab:T390711|T390711]]) === 2025-07-11 === * 14:38 AntiComposite: cvn-app10 restart all bots * 11:10 AntiComposite: cvn-app12 restart all bots * 11:09 AntiComposite: cvn-app10 restart all bots === 2025-06-20 === * 20:49 AntiComposite: cvn-app12: restart all bots * 20:48 AntiComposite: cvn-app10: restart all bots === 2025-05-26 === * 17:59 Krinkle: Create cvn-app14 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:59 Krinkle: Create cvn-app13 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:57 Krinkle: Delete cvn-apache10 instance (replaced/shutdown 2 days ago), ref [[phab:T395164|T395164]] === 2025-05-23 === * 20:30 Krinkle: Shut off cvn-apache10, [[phab:T395164|T395164]] * 20:29 Krinkle: Change cvn.wmcloud.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 20:22 Krinkle: Change cvn.wmflabs.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 19:45 Krinkle: Create cvn-apache11 (debian-12.0-bookworm, g4.cores2.ram4.disk20), [[phab:T395164|T395164]]) === 2025-05-16 === * 18:22 Krinkle: Replace outreach.wikipedia with outreach.wikimedia in cvn-sw/CVNBot19 per https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/820245 since the source channel was renamed * 17:30 Krinkle: krinkle@cvn-apache10:/srv/cvn/git/infrastructure$ git pull -- Deploy https://gerrit.wikimedia.org/r/1146724 * 17:30 Krinkle: krinkle@cvn-apache10 Update git remote in /srv/cvn/git/infrastructure from github.com/countervandalism to https://gerrit.wikimedia.org/r/labs/countervandalism/cvn-infrastructure === 2025-04-21 === * 17:22 AntiComposite: Hard reboot cvn-app10, flapping and not responsive to ssh === 2025-03-30 === * 06:55 Krinkle: krinkle@cvn-apache10: Run `sudo chmod 644 /srv/cvn/git/infrastructure/crontab-config/*.cron`, per [[phab:T390415|T390415]] === 2025-03-12 === * 02:18 AntiComposite: CVNBot9 load id.wikivoyage voy:id: ([[phab:T381080|T381080]]) * 02:15 AntiComposite: CVNBot8 load tig.wikipedia tig: ([[phab:T381379|T381379]]) * 02:14 AntiComposite: CVNBot7 load knc.wikipedia knc: ([[phab:T385185|T385185]]) * 02:11 AntiComposite: CVNBot6 load syl.wikipedia syl: ([[phab:T386464|T386464]]) * 02:08 AntiComposite: CVNBot10 load sat.wiktionary wikt:sat: ([[phab:T386631|T386631]]) === 2025-02-03 === * 22:05 AntiComposite: Hard reboot cvn-apache10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ * 21:58 AntiComposite: Hard reboot cvn-app10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ === 2025-01-02 === * 12:46 Krinkle: /cs flags #cvn-wp-en Lordseriouspig voiced * 12:45 Krinkle: /cs flags #cvn-sw Lordseriouspig voiced === 2024-11-23 === * 00:41 AntiComposite: CVNBot9 load ka.wikisource s:ka: ([[phab:T363243|T363243]]) * 00:38 AntiComposite: CVNBot8 load tcy.wikisource s:tcy: ([[phab:T378471|T378471]]) * 00:37 AntiComposite: CVNBot7 load tcy.wiktionary wikt:tcy: ([[phab:T378463|T378463]]) * 00:25 AntiComposite: Upgrade CVNBot29 to v4.0.4 * 00:25 AntiComposite: Upgrade CVNBot28 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot27 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot26 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot25 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot24 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot23 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot22 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot19 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot17 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot16 to v4.0.4 * 00:20 AntiComposite: Upgrade CVNBot10 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot9 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot8 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot7 to v4.0.4 * 00:17 AntiComposite: Upgrade CVNBot6 to v4.0.4 === 2024-11-22 === * 23:52 AntiComposite: Upgrade CVNBot21 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot20 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot18 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot15 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot14 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot13 to v4.0.4 * 23:49 AntiComposite: Upgrade CVNBot12 to v4.0.4 * 23:48 AntiComposite: Upgrade CVNBot11 to v4.0.4 * 23:47 AntiComposite: Upgrade CVNBot5 to v4.0.4 * 23:45 AntiComposite: Upgrade CVNBot3 to v4.0.4 * 23:44 AntiComposite: Upgrade CVNBot2 to v4.0.4 * 23:41 AntiComposite: Upgrade CVNBot1 to v4.0.4 * 23:32 AntiComposite: Upgrade CVNBot4 to v4.0.4 * 17:08 AntiComposite: restart CVNBots on cvn-app12 due to simultaneous RCReader failure 91950.519949 seconds === 2024-11-08 === * 23:24 AntiComposite: Restarting all CVNBots due to simultaneous RCReader disconnect 54323.128318 seconds ago === 2024-10-29 === * 20:56 AntiComposite: add sh.wikipedia to CVNBot6 as #cvn-wp-sh didn't survive the libera migration * 14:22 AntiComposite: restart all CVNBots === 2024-10-28 === * 12:50 AntiComposite: restarting all CVNBots, not coming up cleanly === 2024-10-25 === * 02:23 AntiComposite: add cs.wikivoyage to CVNBot10 ([[phab:T370913|T370913]]) * 02:21 AntiComposite: add bdr.wikipedia to CVNBot9 ([[phab:T371760|T371760]]) * 02:18 AntiComposite: add mos.wikipedia to CVNBot8 ([[phab:T374644|T374644]]) * 02:14 AntiComposite: add kge.wikipedia to CVNBot7 ([[phab:T374815|T374815]]) * 02:11 AntiComposite: add rsk.wikipedia to CVNBot6 ([[phab:T375017|T375017]]) * 02:07 AntiComposite: add mad.wiktionary to CVNBot9 ([[phab:T375024|T375024]]) * 02:06 AntiComposite: add gor.wikiquote to CVNBot8 ([[phab:T375095|T375095]]) * 02:04 AntiComposite: add nr.wikipedia to CVNBot7 ([[phab:T375102|T375102]]) * 02:01 AntiComposite: add tdd.wikipedia to CVNBot6 ([[phab:T375424|T375424]]) * 01:54 AntiComposite: add shn.wikinews to CVNBot9 ([[phab:T375433|T375433]]) * 01:52 AntiComposite: add iba.wikipedia to CVNBot8 ([[phab:T376572|T376572]]) * 01:50 AntiComposite: add bcl.wikisource to CVNBot7 ([[phab:T377088|T377088]]) * 01:47 AntiComposite: add ann.wikipedia to CVNBot6 ([[phab:T377160|T377160]]) * 01:43 AntiComposite: add igl.wikipedia to CVNBot9 ( [[phab:T363263|T363263]] ) * 01:41 AntiComposite: add my.wikisource to CVNBot8 ([[phab:T363270|T363270]]) * 01:39 AntiComposite: add foundation.wikimedia to CVNBot19 * 01:38 AntiComposite: add wikitech.wikimedia to CVNBot19 === 2024-10-24 === * 11:36 AntiComposite: restart all CVNBots === 2024-10-23 === * 17:33 AntiComposite: restart all CVNBots === 2024-07-03 === * 02:00 AntiComposite: add kus.wikipedia to CVNBot7 ([[phab:T360303|T360303]]) * 01:57 AntiComposite: add bew.wikipedia to CVNBot6 ([[phab:T360310|T360310]]) * 01:54 AntiComposite: add ms.wikisource to CVNBot9 ([[phab:T363250|T363250]]) * 01:53 AntiComposite: add kaa.wiktionary to CVNBot8 ([[phab:T363256|T363256]]) * 01:50 AntiComposite: add dtp.wikipedia to CVNBot7 ([[phab:T365230|T365230]]) * 01:48 AntiComposite: add btm.wikipedia to CVNBot6 ([[phab:T368067|T368067]]) * 01:45 AntiComposite: add fon.wikipedia to CVNBot9 ([[phab:T347939|T347939]]) * 01:43 AntiComposite: add blk.wikisource to CVNBot8 ([[phab:T343542|T343542]]) * 01:41 AntiComposite: su.wikisource to CVNBot7 ([[phab:T343548|T343548]]) * 01:39 AntiComposite: add tly.wikipedia to CVNBot6 ([[phab:T345170|T345170]]) * 01:37 AntiComposite: add dga.wikipedia to CVNBot9 ([[phab:T350229|T350229]]) * 01:35 AntiComposite: add bjn.wikiquote to CVNBot8 ([[phab:T350235|T350235]]) * 01:32 AntiComposite: add zgh.wikipedia to CVNBot7 ([[phab:T350241|T350241]]) * 01:28 AntiComposite: add bbc.wikipedia to CVNBot6 ([[phab:T350373|T350373]]) === 2024-06-24 === * 16:40 Krinkle: cvn-clerkbot parts #cvn-unifications (not operated by CVN, renamed to #wikimedia-unifications) === 2024-06-18 === * 08:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs === 2024-03-22 === * 05:30 Operator873: /cs flags #cvn-simplewikis Drummingman +voice === 2024-02-28 === * 21:34 Krinkle: /cs flags #cvn-wp-da Sarrus local_op === 2024-01-11 === * 12:19 AntiComposite: /cs flags #cvn-meta Bsadowski1 local_op === 2023-12-01 === * 15:30 AntiComposite: restart everything after WMCS network outage === 2023-10-07 === * 14:50 AntiComposite: kill 2 CVNBot11 processes and restart, bot not joined to IRC === 2023-09-22 === * 00:06 Op873: /cs flags #cvn-wp-en Oshwah +AV === 2023-09-16 === * 10:33 JackSparrow: /cs flags #cvn-wp-fa Arian_Ar local_op === 2023-09-07 === * 01:35 AntiComposite: restart all cvn-app12 bots * 01:33 AntiComposite: restart all cvn-app10 bots === 2023-08-15 === * 14:44 AntiComposite: reboot cvn-app10 from Horizon, bots dead and not responding to SSH === 2023-08-09 === * 00:07 AntiComposite: add 9 wikis to #cvn-sw (ref [[phab:T332379|T332379]] [[phab:T336115|T336115]] [[phab:T332093|T332093]] [[phab:T332093|T332093]] [[phab:T335987|T335987]] [[phab:T334459|T334459]] [[phab:T333271|T333271]] [[phab:T334740|T334740]] [[phab:T342865|T342865]]) === 2023-08-08 === * 23:46 AntiComposite: drop wo.wikiquote from CVNBot10 (closed) [[phab:T334482|T334482]] === 2023-07-27 === * 18:15 AntiComposite: Kill and restart CVNBot29 on cvn-app12 === 2023-07-06 === * 16:21 AntiComposite: point git repos to gerrit on cvn-app10 * 16:19 AntiComposite: point git repos to gerrit on cvn-app12 * 16:03 AntiComposite: CVNBot v4.0.3 deployed to all bots ([[phab:T327126|T327126]], [[phab:T327127|T327127]]) * 16:01 AntiComposite: Upgrade CVNBot29 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot28 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot27 to v4.0.3 * 15:59 AntiComposite: Upgrade CVNBot26 to v4.0.3 * 15:58 AntiComposite: Upgrade CVNBot25 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot24 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot23 to v4.0.3 * 15:55 AntiComposite: Upgrade CVNBot22 to v4.0.3 * 15:54 AntiComposite: Upgrade CVNBot19 to v4.0.3 * 15:53 AntiComposite: Upgrade CVNBot17 to v4.0.3 * 15:46 AntiComposite: Upgrade CVNBot16 to v4.0.3 * 15:44 AntiComposite: Upgrade CVNBot10 to v4.0.3 * 15:41 AntiComposite: Upgrade CVNBot9 to v4.0.3 * 15:40 AntiComposite: Upgrade CVNBot8 to v4.0.3 * 15:39 AntiComposite: Upgrade CVNBot7 to v4.0.3 * 15:38 AntiComposite: Upgrade CVNBot6 to v4.0.3 * 04:37 AntiComposite: Upgrade CVNBot21 to v4.0.3 * 04:34 AntiComposite: Upgrade CVNBot20 to v4.0.3 * 04:33 AntiComposite: Upgrade CVNBot18 to v4.0.3 * 04:30 AntiComposite: Upgrade CVNBot15 to v4.0.3 * 04:23 AntiComposite: Upgrade CVNBot14 to v4.0.3 * 04:22 AntiComposite: Upgrade CVNBot13 to v4.0.3 * 04:14 AntiComposite: Upgrade CVNBot12 to v4.0.3 * 04:09 AntiComposite: Upgrade CVNBot11 to v4.0.3 * 04:03 AntiComposite: Upgrade CVNBot5 to v4.0.3 * 04:01 AntiComposite: Upgrade CVNBot4 to v4.0.3 * 04:00 AntiComposite: Upgrade CVNBot3 to v4.0.3 * 03:57 AntiComposite: Upgrade CVNBot2 to v4.0.3 * 03:51 AntiComposite: Upgrade CVNBot1 to v4.0.3 === 2023-06-28 === * 02:34 Operator873: /cs flags #cvn-sw Fehufanga voiced === 2023-06-16 === * 22:05 AntiComposite: manually restart cvn-clerkbot === 2023-05-15 === * 14:58 hauskater: Dropped akwiki and nawiki from CVNBot10 as closed wikis. On-wiki lists require an update. === 2023-04-26 === * 20:07 AntiComposite: /cs flags #cvn-mk-scan M4r51n voiced === 2023-04-21 === * 22:12 Operator873: granted voice to Fehufanga in #cvn-simplewikis === 2023-04-14 === * 18:28 AntiComposite: restart cvn-app10 from horizon, bots quit and ssh times out === 2023-03-22 === * 03:33 Operator873: Voiced Tulsi in #cvn-sw -meta -mediawiki -commons -simplewikis === 2023-03-13 === * 19:46 Operator873: CVNBot18 restarted === 2023-03-03 === * 14:45 AntiComposite: /cs flags #cvn-sw-spam COIBot bot === 2023-02-27 === * 22:33 herzog: Loaded gur.wikipedia to SWMT Group 4 (CVNBot9) - [[phab:T327842|T327842]] * 18:04 herzog: Loaded guc.wikipedia to CVNBot9 / Group 4 - [[phab:T326236|T326236]] === 2023-02-02 === * 00:21 ma: Added 12 new wikis to CVNBot<nowiki>{</nowiki>6,7,8<nowiki>}</nowiki>, 4 to each one. Refs.: [[phab:T321283|T321283]] [[phab:T321289|T321289]] [[phab:T321295|T321295]] [[phab:T326139|T326139]] [[phab:T305281|T305281]] [[phab:T310873|T310873]] [[phab:T312215|T312215]] [[phab:T314640|T314640]] [[phab:T314646|T314646]] [[phab:T316457|T316457]] [[phab:T317113|T317113]] [[phab:T319191|T319191]] === 2023-01-30 === * 22:50 Krinkle: Delete cvn-app8 and cvn-app9 instances, ref [[phab:T306066|T306066]] === 2023-01-28 === * 02:51 AntiComposite: /cs flags #cvn-sw Ajraddatz local_op === 2023-01-24 === * 08:54 Krinkle: Delete cvn-apache9, [[phab:T306066|T306066]] * 08:54 Krinkle: Suspend cvn-app8 and cvn-app9 (`pgrep -af cvn` is empty on both), [[phab:T306066|T306066]] === 2023-01-23 === * 16:53 AntiComposite: Deploy {{Gerrit|716e140}} to app12 ([[phab:T306066|T306066]]) * 16:50 AntiComposite: Deploy {{Gerrit|716e140}} to app9 ([[phab:T306066|T306066]]) * 16:29 AntiComposite: Deploy {{Gerrit|442f324}} to app12 ([[phab:T306066|T306066]]) * 16:25 AntiComposite: Deploy {{Gerrit|442f324}} to app9 ([[phab:T306066|T306066]]) * 16:01 AntiComposite: Deploy {{Gerrit|9024b8f}} to app12 ([[phab:T306066|T306066]]) * 15:59 AntiComposite: Deploy {{Gerrit|9024b8f}} to app9 ([[phab:T306066|T306066]]) === 2023-01-22 === * 21:40 AntiComposite: start cvndb-CVNBot14-publish on app10 * 21:07 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app10, starting bots ([[phab:T306066|T306066]]) * 20:56 AntiComposite: disable cvndb-CVNBot14-publish on app8 * 20:51 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app8, stopping bots ([[phab:T306066|T306066]]) * 19:53 AntiComposite: Deploy {{Gerrit|80ea1f5}} to cvn-app10 ([[phab:T306066|T306066]]) * 15:43 AntiComposite: restart all CVNBots on app9 * 15:42 AntiComposite: restart all CVNBots on app8 === 2023-01-17 === * 00:15 Krinkle: Suspend cvn-apache9, replaced by cvn-apache10, ref [[phab:T306066|T306066]] * 00:14 Krinkle: Switch cvn.wmflabs.org from cvn-apache9 to cvn-apache10 === 2023-01-16 === * 00:10 Krinkle: Move https://github.com/countervandalism/cvn-clerkbot to https://github.com/wikimedia/countervandalism-cvn-clerkbot (with HTTP and Git redirect preserved), and replace with Gerrit mirror === 2023-01-15 === * 23:12 Krinkle: Create 'labs-cvn' permission group in Gerrit with CVN staff members * 23:12 Krinkle: Move https://github.com/countervandalism/cvn-api to https://github.com/wikimedia/countervandalism-cvn-api (with HTTP and Git redirect preserved), and replace with Gerrit mirror * 22:02 Krinkle: Switch new cvn.wmcloud.org proxy from cvn-apache9 to cvn-apache10 (Leave main cvn.wmflabs.org as-is for now). === 2023-01-14 === * 21:45 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|4cee27a}}) * 21:22 AntiComposite: move cvn-clerbot back to cvn-app9 (deploy {{Gerrit|371ba2a}}) * 21:10 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|3f3f40f}}) === 2023-01-10 === * 23:22 Krinkle: krinkle@cvn-apache9$ update infrastructure.git, sudo apachectl graceful * 23:20 Krinkle: Create cvn.wmcloud.org web proxy (in addition to cvn.wmflabs.org) === 2023-01-07 === * 20:53 AntiComposite: apply role::labs::lvm::srv only to cvn-apache9, cvn-app8, and cvn-app9 to fix puppet failures on new instances === 2023-01-04 === * 20:47 Krinkle: Allocate new floating IPs to cvn-app10 and cvn-app11 * 20:46 Krinkle: Create new cvn-apache10, cvn-app10, cvn-app11 with Debian 11 Bullseye to replace the old Debian 9.1 Stretch instances * 20:04 taavi: bump floating ip quota from 2 to 4, [[phab:T326269|T326269]] === 2022-12-27 === * 20:11 Frosty873: /cs flags #cvn-meta xaosflux voiced * 20:11 Frosty873: /cs flags #cvn-wp-en xaosflux voiced === 2022-12-23 === * 03:25 AntiComposite: /cs flags #cvn-meta tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-mediawiki tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-sw tryvix1509 voiced === 2022-10-18 === * 23:13 Joan: CVNBot3 restarted (Last message was received on RCReader 62854.814658 seconds ag) === 2022-09-04 === * 22:21 Operator873: /cs flags #cvn-simplewikis Enfcer +AV * 02:20 Operator873: /cs flags #cvn-sw Bot873 +voiced === 2022-08-26 === * 14:09 hauskatze: Loaded pcm.wikipedia and guw.wiktionary to CVNBot8 & 9 respectively {{!}} [[phab:T310880|T310880]] [[phab:T309057|T309057]] === 2022-07-09 === * 16:42 AntiComposite: /cs flags #cvn-commons pandakekok9 voiced === 2022-07-08 === * 21:53 Krinkle: krinkle@horizon.wikimedia.org Add anticomposite as project member and project admin to cloudvps.cvn === 2022-07-01 === * 21:39 Krinkle: cvn-app8: kill CVNBot14.exe and two (!) procs for CVNBot18.exe === 2022-06-25 === * 03:25 AntiComposite: /cs flags #cvn-wp-en PhantomTech voiced === 2022-06-22 === * 21:04 op873: <+CVNBot3> Added: LuchoCR is on es.wikipedia bot list, added by Operator873{{!}}CVN until the end of time ("Mass blockiing P2P-proxies with script") * 20:34 op873: restart CVNBot3 (possibly caused by block flood) * 19:31 op873: restart CVNBot3 === 2022-06-15 === * 18:49 AntiComposite: /cs flags #cvn-wp-en Zppix voiced * 18:48 AntiComposite: /cs flags #cvn-simplewikis Zppix voiced === 2022-05-23 === * 00:24 Joan: Flags +AV were set on Sargento in cvn-wp-es * 00:23 Joan: Flags +AV were set on alhen in cvn-wp-es === 2022-05-19 === * 23:10 Joan: CVNBot3 restarted (Last message was received on RCReader 92593.747667 seconds ago) === 2022-05-11 === * 07:34 Operator873: /cs flags #cvn-wp-en Tamzin voiced === 2022-05-07 === * 17:40 Operator873: /cs flags #cvn-sw koi voiced * 17:39 Operator873: /cs flags #cvn-zh-scan koi voiced === 2022-04-28 === * 03:19 Joan: CVNBot3 restarted (Last message was received on RCReader 75273.332577 seconds ago) === 2022-04-22 === * 15:08 AntiComposite: /cs flags #cvn-meta Bsadowski1 voiced === 2022-04-18 === * 20:44 AntiComposite: /cs flags #cvn-sw Vermont voiced === 2022-04-13 === * 22:40 Operator873: /cs flags #cvn-meta Joan voiced * 22:40 Operator873: /cs flags #cvn-sw Joan voiced * 22:14 Joan: CVNBot3 restarted (Last message was received on RCReader 54942.175428 seconds ago) === 2022-04-07 === * 23:15 Operator873: /cs flags #cvn-wp-hr NovakWatchmen local_op * 23:13 Operator873: voiced Superpes (Superpes15) in #cvn-sw #cvn-sw-spam and #cvn-it-scan === 2022-04-04 === * 17:34 Operator873: Voiced Vermont in #cvn-meta and #cvn-simplewikis /cs flags #cvn-meta Vermont voiced === 2022-03-30 === * 14:33 Joan: CVNBot3 restarted (Last message was received on RCReader 26318.335196 seconds ago) === 2022-03-28 === * 02:38 AntiComposite: /cs flags #cvn-wp-en Bsoyka voiced === 2022-03-21 === * 20:22 Operator873: /cs flags #cvn-simplewikis Bsadowski1 +AfiotvV * 20:17 Operator873: Operator873{{!}}CVN (Operator873) set flags +AVfitv on Bsadowski1 * 20:03 Operator873: Operator873{{!}}CVN (Operator873) set flags +V on Bsadowski1 * 17:04 AntiComposite: /cs flags #cvn-sw Bsadowski1 local_op === 2022-03-15 === * 15:38 Joan: CVNBot3 restarted (Last message was received on RCReader 26424.279343 seconds ago) === 2022-03-14 === * 14:02 Joan: CVNBot3 restarted (Last message was received on RCReader 17096.72183 seconds ago) === 2022-03-12 === * 16:27 Joan: CVNBot3 restarted (Last message was received on RCReader 27236.775673 seconds ago) === 2022-03-11 === * 14:24 Joan: CVNBot3 restarted (Last message was received on RCReader 18853.006849 seconds ago) === 2022-03-10 === * 14:08 Joan: CVNBot3 restarted (Last message was received on RCReader 22518.614282 seconds ago) === 2022-03-08 === * 20:27 AntiComposite: /cs flags #cvn-wp-en Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-simplewikis Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-commons Sarrus voiced === 2022-03-07 === * 16:30 AntiComposite: /cs flags #cvn-meta zabe voiced * 16:25 AntiComposite: /cs flags #cvn-simplewikis DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-meta DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-sw TheresNoTime voiced * 16:07 Krinkle: /cs flags #cvn-staff Operator873 staff * 16:07 Krinkle: /cs flags #cvn-staff AntiComposite staff === 2022-03-05 === * 04:13 Joan: CVNBot3 restarted (Last message was received on RCReader 31573.894101 seconds ago) === 2022-03-03 === * 16:39 Joan: CVNBot3 restarted (Last message was received on RCReader 36578.236383 seconds ago) === 2022-03-01 === * 13:21 Joan: CVNBot3 restarted (Last message was received on RCReader 20646.781861 seconds ago) === 2022-02-15 === * 14:12 Joan: CVNBot3 restarted (Last message was received on RCReader 25001.391103 seconds ago) === 2022-02-13 === * 18:47 andrewbogott: switching to project-local nfs server cvn-nfs-1 * 17:54 andrewbogott: switching to project-local nfs server puppet-diffs-nfs-1 === 2022-02-10 === * 16:17 Joan: CVNBot3 restarted (Last message was received on RCReader 39817.871151 seconds ago) === 2022-02-08 === * 15:51 Joan: CVNBot3 restarted (Last message was received on RCReader 28868.916144 seconds ago) === 2022-02-04 === * 23:59 andrewbogott: accidentally restarted all VMs due to misreading the project purge page. sorry! === 2022-02-02 === * CVN: Several bots restarted after netsplit took nickserv and some bots with it. * 10:26 Krinkle: CVNBot1 bes del delete(?!d) — originally added by huh (reason: "widewuto") === 2022-02-01 === * 15:20 Joan: CVNBot3 restarted (Last message was received on RCReader 26990.323435 seconds ago) === 2022-01-31 === * 17:37 Joan: CVNBot3 restarted (Last message was received on RCReader 48827.882566 seconds ago) === 2022-01-27 === * 16:58 Joan: CVNBot3 restarted (Last message was received on RCReader 29206.852828 seconds ago) === 2022-01-21 === * 16:07 Joan: CVNBot3 restarted (Last message was received on RCReader 22091.557102 seconds ago) === 2022-01-20 === * 18:13 Cam11598: CVNBot15 restarted === 2022-01-19 === * 17:26 Joan: Restarted CVNBot3 (Last message was received on RCReader 28129.031916 seconds ago) === 2022-01-18 === * 16:55 Joan: Restarted CVNBot3 (Last message was received on RCReader 26283.381782 seconds ago) === 2022-01-17 === * 16:33 Joan: Restarted CVNBot3 (#cvn-wp-es) (Last message was received on RCReader 197065.877109 seconds ago) === 2022-01-15 === * 04:56 Cam11598: restarted CVNBOT18 8:55:47 PM <�25B100+ CVNBot18> Last message was received on RCReader 29723.456263 seconds ago === 2022-01-13 === * 01:29 Cam11598: restarted CVNBot2 nickserv issue * 01:29 Cam11598: restarted CVNBot18 - no response from RC feed === 2022-01-09 === * 18:18 Joan: Flags +AV were set on Hasley in cvn-wp-es (sysop at es.wikipedia) * 17:56 Krinkle: /cs flags #cvn-wp-es Joan local_op === 2022-01-07 === * 22:08 hauskatze: CVNBot9 load co.wiktionary wikt:co: * 22:04 hauskatze: CVNBot9 load ban.wikisource s:ban: * 22:04 hauskatze: CVNBot9 load ba.wikibooks b:ba: * 10:51 hauskatze: Loaded alt.wikipedia to Group 4 (CVNBot9) - small wiki not monitored === 2022-01-06 === * 19:42 hauskatze: Loaded ami.wikipedia to CVNBot8 - [[phab:T292421|T292421]] * 19:41 hauskatze: Loaded pwn.wikipedia to CVNBot7 - [[phab:T292419|T292419]] * 19:39 hauskatze: Loaded lmo.wiktionary to CVNBot6 - [[phab:T292076|T292076]] * 19:34 hauskatze: Loaded jv.wikisource to CVNBot6 refs. [[phab:T287319|T287319]] * 19:29 Krinkle: cs flags #cvn-sw hauskatze local_op * 13:57 Krinkle: Krinkle added $a:Cam11598 to the #cvn-staff I list (+I) {{SAL|Project Name=cvn}} <noinclude> ==Archives== * [[Nova Resource:Cvn/SAL/Archive 1|Archive 1]] (2006-2009) * [[Nova Resource:Cvn/SAL/Archive 2|Archive 2]] (2010-2011) * [[Nova Resource:Cvn/SAL/Archive 3|Archive 3]] (2012-2013) * [[Nova Resource:Cvn/SAL/Archive 4|Archive 4]] (2013-2021) (some parts in 2013 are not indexed) [[Category:SAL]]</noinclude> 694b5w008z0sd9erggabli2p0hjlucp 2426637 2426636 2026-06-13T23:12:07Z Stashbot 7414 AntiComposite: CVNBot23 drop & purge zh.wikinews (T428622) 2426637 wikitext text/x-wiki === 2026-06-13 === * 23:12 AntiComposite: CVNBot23 drop & purge zh.wikinews ([[phab:T428622|T428622]]) * 23:11 AntiComposite: CVNBot10 drop & purge ca.wikinews, ko.wikinews, no.wikinews ([[phab:T428622|T428622]]) * 23:07 AntiComposite: CVNBot9 drop & purge bs.wikinews, el.wikinews, fa.wikinews, shn.wikinews, zh.wikinews ([[phab:T428622|T428622]]) * 23:03 AntiComposite: CVNBot8 drop & purge ar.wikinews, cs.wikinews, de.wikinews, fi.wikinews, he.wikinews, ru.wikinews, sq.wikinews, sr.wikinews, uk.wikinews ([[phab:T428622|T428622]]) * 22:58 AntiComposite: CVNBot7 drop & purge es.wikinews, guw.wikinews, pt.wikinews ([[phab:T428622|T428622]]) * 22:56 AntiComposite: CVNBot6 drop & purge eo.wikinews, fr.wikinews, pl.wikinews, ro.wikinews, sv.wikinews, ta.wikinews ([[phab:T428622|T428622]]) * 22:49 AntiComposite: CVNBot4 drop it.wikinews ([[phab:T428622|T428622]]) === 2026-06-02 === * 01:03 Krinkle: /cs flags #cvn-sw Divinations voiced === 2026-05-26 === * 18:07 AntiComposite: restart all bots -- disconnected === 2026-05-03 === * 13:39 Krinkle: Disable "Admin immed notify" for cvn-private https://lists.wikimedia.org/postorius/lists/cvn-private.lists.wikimedia.org/settings/automatic_responses. We previously removed the sub form but this is no longer supported in mailman3. We require confirm/moderate for new subs, there is no way to turn it off. But we can at least disable the noise. === 2026-04-27 === * 12:22 Krinkle: /cs flags #cvn-meta NathanVeritas voiced === 2026-04-01 === * 13:34 AntiComposite: restart all bots === 2026-02-04 === * 20:33 AntiComposite: Restart all bots === 2025-12-26 === * 15:54 Operator873: /cs flags #cvn-zh-scan nya_1F616EMO voiced === 2025-11-27 === * 13:48 AntiComposite: CVNBot10 load tok.wikipedia tok: ([[phab:T404567|T404567]]) * 13:47 AntiComposite: CVNBot9 load ms.wikiquote q:ms: ([[phab:T404700|T404700]]) * 13:45 AntiComposite: CVNBot8 load min.wikisource s:min: ([[phab:T408343|T408343]]) * 13:44 AntiComposite: CVNBot7 load pcm.wikiquote q:pcm: ([[phab:T408351|T408351]]) * 13:43 AntiComposite: CVNBot6 load tl.wikisource s:tl: ([[phab:T388654|T388654]]) * 13:42 AntiComposite: CVNBot10 load bew.wiktionary wikt:bew: ([[phab:T402134|T402134]]) * 13:41 AntiComposite: CVNBot9 load zgh.wiktionary wikt:zgh: ([[phab:T399785|T399785]]) * 13:40 AntiComposite: CVNBot8 load min.wikibooks b:min: ([[phab:T395499|T395499]]) * 13:38 AntiComposite: CVNBot7 load rki.wikipedia rki: ([[phab:T392499|T392499]]) * 13:37 AntiComposite: CVNBot6 load mad.wikisource s:mad: ([[phab:T391767|T391767]]) === 2025-10-28 === * 23:16 AntiComposite: /cs flags #cvn-commons revi local_op === 2025-08-20 === * 20:35 AntiComposite: CVNBot10 load nup.wikipedia nup: ([[phab:T390711|T390711]]) === 2025-07-11 === * 14:38 AntiComposite: cvn-app10 restart all bots * 11:10 AntiComposite: cvn-app12 restart all bots * 11:09 AntiComposite: cvn-app10 restart all bots === 2025-06-20 === * 20:49 AntiComposite: cvn-app12: restart all bots * 20:48 AntiComposite: cvn-app10: restart all bots === 2025-05-26 === * 17:59 Krinkle: Create cvn-app14 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:59 Krinkle: Create cvn-app13 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:57 Krinkle: Delete cvn-apache10 instance (replaced/shutdown 2 days ago), ref [[phab:T395164|T395164]] === 2025-05-23 === * 20:30 Krinkle: Shut off cvn-apache10, [[phab:T395164|T395164]] * 20:29 Krinkle: Change cvn.wmcloud.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 20:22 Krinkle: Change cvn.wmflabs.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 19:45 Krinkle: Create cvn-apache11 (debian-12.0-bookworm, g4.cores2.ram4.disk20), [[phab:T395164|T395164]]) === 2025-05-16 === * 18:22 Krinkle: Replace outreach.wikipedia with outreach.wikimedia in cvn-sw/CVNBot19 per https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/820245 since the source channel was renamed * 17:30 Krinkle: krinkle@cvn-apache10:/srv/cvn/git/infrastructure$ git pull -- Deploy https://gerrit.wikimedia.org/r/1146724 * 17:30 Krinkle: krinkle@cvn-apache10 Update git remote in /srv/cvn/git/infrastructure from github.com/countervandalism to https://gerrit.wikimedia.org/r/labs/countervandalism/cvn-infrastructure === 2025-04-21 === * 17:22 AntiComposite: Hard reboot cvn-app10, flapping and not responsive to ssh === 2025-03-30 === * 06:55 Krinkle: krinkle@cvn-apache10: Run `sudo chmod 644 /srv/cvn/git/infrastructure/crontab-config/*.cron`, per [[phab:T390415|T390415]] === 2025-03-12 === * 02:18 AntiComposite: CVNBot9 load id.wikivoyage voy:id: ([[phab:T381080|T381080]]) * 02:15 AntiComposite: CVNBot8 load tig.wikipedia tig: ([[phab:T381379|T381379]]) * 02:14 AntiComposite: CVNBot7 load knc.wikipedia knc: ([[phab:T385185|T385185]]) * 02:11 AntiComposite: CVNBot6 load syl.wikipedia syl: ([[phab:T386464|T386464]]) * 02:08 AntiComposite: CVNBot10 load sat.wiktionary wikt:sat: ([[phab:T386631|T386631]]) === 2025-02-03 === * 22:05 AntiComposite: Hard reboot cvn-apache10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ * 21:58 AntiComposite: Hard reboot cvn-app10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ === 2025-01-02 === * 12:46 Krinkle: /cs flags #cvn-wp-en Lordseriouspig voiced * 12:45 Krinkle: /cs flags #cvn-sw Lordseriouspig voiced === 2024-11-23 === * 00:41 AntiComposite: CVNBot9 load ka.wikisource s:ka: ([[phab:T363243|T363243]]) * 00:38 AntiComposite: CVNBot8 load tcy.wikisource s:tcy: ([[phab:T378471|T378471]]) * 00:37 AntiComposite: CVNBot7 load tcy.wiktionary wikt:tcy: ([[phab:T378463|T378463]]) * 00:25 AntiComposite: Upgrade CVNBot29 to v4.0.4 * 00:25 AntiComposite: Upgrade CVNBot28 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot27 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot26 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot25 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot24 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot23 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot22 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot19 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot17 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot16 to v4.0.4 * 00:20 AntiComposite: Upgrade CVNBot10 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot9 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot8 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot7 to v4.0.4 * 00:17 AntiComposite: Upgrade CVNBot6 to v4.0.4 === 2024-11-22 === * 23:52 AntiComposite: Upgrade CVNBot21 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot20 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot18 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot15 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot14 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot13 to v4.0.4 * 23:49 AntiComposite: Upgrade CVNBot12 to v4.0.4 * 23:48 AntiComposite: Upgrade CVNBot11 to v4.0.4 * 23:47 AntiComposite: Upgrade CVNBot5 to v4.0.4 * 23:45 AntiComposite: Upgrade CVNBot3 to v4.0.4 * 23:44 AntiComposite: Upgrade CVNBot2 to v4.0.4 * 23:41 AntiComposite: Upgrade CVNBot1 to v4.0.4 * 23:32 AntiComposite: Upgrade CVNBot4 to v4.0.4 * 17:08 AntiComposite: restart CVNBots on cvn-app12 due to simultaneous RCReader failure 91950.519949 seconds === 2024-11-08 === * 23:24 AntiComposite: Restarting all CVNBots due to simultaneous RCReader disconnect 54323.128318 seconds ago === 2024-10-29 === * 20:56 AntiComposite: add sh.wikipedia to CVNBot6 as #cvn-wp-sh didn't survive the libera migration * 14:22 AntiComposite: restart all CVNBots === 2024-10-28 === * 12:50 AntiComposite: restarting all CVNBots, not coming up cleanly === 2024-10-25 === * 02:23 AntiComposite: add cs.wikivoyage to CVNBot10 ([[phab:T370913|T370913]]) * 02:21 AntiComposite: add bdr.wikipedia to CVNBot9 ([[phab:T371760|T371760]]) * 02:18 AntiComposite: add mos.wikipedia to CVNBot8 ([[phab:T374644|T374644]]) * 02:14 AntiComposite: add kge.wikipedia to CVNBot7 ([[phab:T374815|T374815]]) * 02:11 AntiComposite: add rsk.wikipedia to CVNBot6 ([[phab:T375017|T375017]]) * 02:07 AntiComposite: add mad.wiktionary to CVNBot9 ([[phab:T375024|T375024]]) * 02:06 AntiComposite: add gor.wikiquote to CVNBot8 ([[phab:T375095|T375095]]) * 02:04 AntiComposite: add nr.wikipedia to CVNBot7 ([[phab:T375102|T375102]]) * 02:01 AntiComposite: add tdd.wikipedia to CVNBot6 ([[phab:T375424|T375424]]) * 01:54 AntiComposite: add shn.wikinews to CVNBot9 ([[phab:T375433|T375433]]) * 01:52 AntiComposite: add iba.wikipedia to CVNBot8 ([[phab:T376572|T376572]]) * 01:50 AntiComposite: add bcl.wikisource to CVNBot7 ([[phab:T377088|T377088]]) * 01:47 AntiComposite: add ann.wikipedia to CVNBot6 ([[phab:T377160|T377160]]) * 01:43 AntiComposite: add igl.wikipedia to CVNBot9 ( [[phab:T363263|T363263]] ) * 01:41 AntiComposite: add my.wikisource to CVNBot8 ([[phab:T363270|T363270]]) * 01:39 AntiComposite: add foundation.wikimedia to CVNBot19 * 01:38 AntiComposite: add wikitech.wikimedia to CVNBot19 === 2024-10-24 === * 11:36 AntiComposite: restart all CVNBots === 2024-10-23 === * 17:33 AntiComposite: restart all CVNBots === 2024-07-03 === * 02:00 AntiComposite: add kus.wikipedia to CVNBot7 ([[phab:T360303|T360303]]) * 01:57 AntiComposite: add bew.wikipedia to CVNBot6 ([[phab:T360310|T360310]]) * 01:54 AntiComposite: add ms.wikisource to CVNBot9 ([[phab:T363250|T363250]]) * 01:53 AntiComposite: add kaa.wiktionary to CVNBot8 ([[phab:T363256|T363256]]) * 01:50 AntiComposite: add dtp.wikipedia to CVNBot7 ([[phab:T365230|T365230]]) * 01:48 AntiComposite: add btm.wikipedia to CVNBot6 ([[phab:T368067|T368067]]) * 01:45 AntiComposite: add fon.wikipedia to CVNBot9 ([[phab:T347939|T347939]]) * 01:43 AntiComposite: add blk.wikisource to CVNBot8 ([[phab:T343542|T343542]]) * 01:41 AntiComposite: su.wikisource to CVNBot7 ([[phab:T343548|T343548]]) * 01:39 AntiComposite: add tly.wikipedia to CVNBot6 ([[phab:T345170|T345170]]) * 01:37 AntiComposite: add dga.wikipedia to CVNBot9 ([[phab:T350229|T350229]]) * 01:35 AntiComposite: add bjn.wikiquote to CVNBot8 ([[phab:T350235|T350235]]) * 01:32 AntiComposite: add zgh.wikipedia to CVNBot7 ([[phab:T350241|T350241]]) * 01:28 AntiComposite: add bbc.wikipedia to CVNBot6 ([[phab:T350373|T350373]]) === 2024-06-24 === * 16:40 Krinkle: cvn-clerkbot parts #cvn-unifications (not operated by CVN, renamed to #wikimedia-unifications) === 2024-06-18 === * 08:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs === 2024-03-22 === * 05:30 Operator873: /cs flags #cvn-simplewikis Drummingman +voice === 2024-02-28 === * 21:34 Krinkle: /cs flags #cvn-wp-da Sarrus local_op === 2024-01-11 === * 12:19 AntiComposite: /cs flags #cvn-meta Bsadowski1 local_op === 2023-12-01 === * 15:30 AntiComposite: restart everything after WMCS network outage === 2023-10-07 === * 14:50 AntiComposite: kill 2 CVNBot11 processes and restart, bot not joined to IRC === 2023-09-22 === * 00:06 Op873: /cs flags #cvn-wp-en Oshwah +AV === 2023-09-16 === * 10:33 JackSparrow: /cs flags #cvn-wp-fa Arian_Ar local_op === 2023-09-07 === * 01:35 AntiComposite: restart all cvn-app12 bots * 01:33 AntiComposite: restart all cvn-app10 bots === 2023-08-15 === * 14:44 AntiComposite: reboot cvn-app10 from Horizon, bots dead and not responding to SSH === 2023-08-09 === * 00:07 AntiComposite: add 9 wikis to #cvn-sw (ref [[phab:T332379|T332379]] [[phab:T336115|T336115]] [[phab:T332093|T332093]] [[phab:T332093|T332093]] [[phab:T335987|T335987]] [[phab:T334459|T334459]] [[phab:T333271|T333271]] [[phab:T334740|T334740]] [[phab:T342865|T342865]]) === 2023-08-08 === * 23:46 AntiComposite: drop wo.wikiquote from CVNBot10 (closed) [[phab:T334482|T334482]] === 2023-07-27 === * 18:15 AntiComposite: Kill and restart CVNBot29 on cvn-app12 === 2023-07-06 === * 16:21 AntiComposite: point git repos to gerrit on cvn-app10 * 16:19 AntiComposite: point git repos to gerrit on cvn-app12 * 16:03 AntiComposite: CVNBot v4.0.3 deployed to all bots ([[phab:T327126|T327126]], [[phab:T327127|T327127]]) * 16:01 AntiComposite: Upgrade CVNBot29 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot28 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot27 to v4.0.3 * 15:59 AntiComposite: Upgrade CVNBot26 to v4.0.3 * 15:58 AntiComposite: Upgrade CVNBot25 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot24 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot23 to v4.0.3 * 15:55 AntiComposite: Upgrade CVNBot22 to v4.0.3 * 15:54 AntiComposite: Upgrade CVNBot19 to v4.0.3 * 15:53 AntiComposite: Upgrade CVNBot17 to v4.0.3 * 15:46 AntiComposite: Upgrade CVNBot16 to v4.0.3 * 15:44 AntiComposite: Upgrade CVNBot10 to v4.0.3 * 15:41 AntiComposite: Upgrade CVNBot9 to v4.0.3 * 15:40 AntiComposite: Upgrade CVNBot8 to v4.0.3 * 15:39 AntiComposite: Upgrade CVNBot7 to v4.0.3 * 15:38 AntiComposite: Upgrade CVNBot6 to v4.0.3 * 04:37 AntiComposite: Upgrade CVNBot21 to v4.0.3 * 04:34 AntiComposite: Upgrade CVNBot20 to v4.0.3 * 04:33 AntiComposite: Upgrade CVNBot18 to v4.0.3 * 04:30 AntiComposite: Upgrade CVNBot15 to v4.0.3 * 04:23 AntiComposite: Upgrade CVNBot14 to v4.0.3 * 04:22 AntiComposite: Upgrade CVNBot13 to v4.0.3 * 04:14 AntiComposite: Upgrade CVNBot12 to v4.0.3 * 04:09 AntiComposite: Upgrade CVNBot11 to v4.0.3 * 04:03 AntiComposite: Upgrade CVNBot5 to v4.0.3 * 04:01 AntiComposite: Upgrade CVNBot4 to v4.0.3 * 04:00 AntiComposite: Upgrade CVNBot3 to v4.0.3 * 03:57 AntiComposite: Upgrade CVNBot2 to v4.0.3 * 03:51 AntiComposite: Upgrade CVNBot1 to v4.0.3 === 2023-06-28 === * 02:34 Operator873: /cs flags #cvn-sw Fehufanga voiced === 2023-06-16 === * 22:05 AntiComposite: manually restart cvn-clerkbot === 2023-05-15 === * 14:58 hauskater: Dropped akwiki and nawiki from CVNBot10 as closed wikis. On-wiki lists require an update. === 2023-04-26 === * 20:07 AntiComposite: /cs flags #cvn-mk-scan M4r51n voiced === 2023-04-21 === * 22:12 Operator873: granted voice to Fehufanga in #cvn-simplewikis === 2023-04-14 === * 18:28 AntiComposite: restart cvn-app10 from horizon, bots quit and ssh times out === 2023-03-22 === * 03:33 Operator873: Voiced Tulsi in #cvn-sw -meta -mediawiki -commons -simplewikis === 2023-03-13 === * 19:46 Operator873: CVNBot18 restarted === 2023-03-03 === * 14:45 AntiComposite: /cs flags #cvn-sw-spam COIBot bot === 2023-02-27 === * 22:33 herzog: Loaded gur.wikipedia to SWMT Group 4 (CVNBot9) - [[phab:T327842|T327842]] * 18:04 herzog: Loaded guc.wikipedia to CVNBot9 / Group 4 - [[phab:T326236|T326236]] === 2023-02-02 === * 00:21 ma: Added 12 new wikis to CVNBot<nowiki>{</nowiki>6,7,8<nowiki>}</nowiki>, 4 to each one. Refs.: [[phab:T321283|T321283]] [[phab:T321289|T321289]] [[phab:T321295|T321295]] [[phab:T326139|T326139]] [[phab:T305281|T305281]] [[phab:T310873|T310873]] [[phab:T312215|T312215]] [[phab:T314640|T314640]] [[phab:T314646|T314646]] [[phab:T316457|T316457]] [[phab:T317113|T317113]] [[phab:T319191|T319191]] === 2023-01-30 === * 22:50 Krinkle: Delete cvn-app8 and cvn-app9 instances, ref [[phab:T306066|T306066]] === 2023-01-28 === * 02:51 AntiComposite: /cs flags #cvn-sw Ajraddatz local_op === 2023-01-24 === * 08:54 Krinkle: Delete cvn-apache9, [[phab:T306066|T306066]] * 08:54 Krinkle: Suspend cvn-app8 and cvn-app9 (`pgrep -af cvn` is empty on both), [[phab:T306066|T306066]] === 2023-01-23 === * 16:53 AntiComposite: Deploy {{Gerrit|716e140}} to app12 ([[phab:T306066|T306066]]) * 16:50 AntiComposite: Deploy {{Gerrit|716e140}} to app9 ([[phab:T306066|T306066]]) * 16:29 AntiComposite: Deploy {{Gerrit|442f324}} to app12 ([[phab:T306066|T306066]]) * 16:25 AntiComposite: Deploy {{Gerrit|442f324}} to app9 ([[phab:T306066|T306066]]) * 16:01 AntiComposite: Deploy {{Gerrit|9024b8f}} to app12 ([[phab:T306066|T306066]]) * 15:59 AntiComposite: Deploy {{Gerrit|9024b8f}} to app9 ([[phab:T306066|T306066]]) === 2023-01-22 === * 21:40 AntiComposite: start cvndb-CVNBot14-publish on app10 * 21:07 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app10, starting bots ([[phab:T306066|T306066]]) * 20:56 AntiComposite: disable cvndb-CVNBot14-publish on app8 * 20:51 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app8, stopping bots ([[phab:T306066|T306066]]) * 19:53 AntiComposite: Deploy {{Gerrit|80ea1f5}} to cvn-app10 ([[phab:T306066|T306066]]) * 15:43 AntiComposite: restart all CVNBots on app9 * 15:42 AntiComposite: restart all CVNBots on app8 === 2023-01-17 === * 00:15 Krinkle: Suspend cvn-apache9, replaced by cvn-apache10, ref [[phab:T306066|T306066]] * 00:14 Krinkle: Switch cvn.wmflabs.org from cvn-apache9 to cvn-apache10 === 2023-01-16 === * 00:10 Krinkle: Move https://github.com/countervandalism/cvn-clerkbot to https://github.com/wikimedia/countervandalism-cvn-clerkbot (with HTTP and Git redirect preserved), and replace with Gerrit mirror === 2023-01-15 === * 23:12 Krinkle: Create 'labs-cvn' permission group in Gerrit with CVN staff members * 23:12 Krinkle: Move https://github.com/countervandalism/cvn-api to https://github.com/wikimedia/countervandalism-cvn-api (with HTTP and Git redirect preserved), and replace with Gerrit mirror * 22:02 Krinkle: Switch new cvn.wmcloud.org proxy from cvn-apache9 to cvn-apache10 (Leave main cvn.wmflabs.org as-is for now). === 2023-01-14 === * 21:45 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|4cee27a}}) * 21:22 AntiComposite: move cvn-clerbot back to cvn-app9 (deploy {{Gerrit|371ba2a}}) * 21:10 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|3f3f40f}}) === 2023-01-10 === * 23:22 Krinkle: krinkle@cvn-apache9$ update infrastructure.git, sudo apachectl graceful * 23:20 Krinkle: Create cvn.wmcloud.org web proxy (in addition to cvn.wmflabs.org) === 2023-01-07 === * 20:53 AntiComposite: apply role::labs::lvm::srv only to cvn-apache9, cvn-app8, and cvn-app9 to fix puppet failures on new instances === 2023-01-04 === * 20:47 Krinkle: Allocate new floating IPs to cvn-app10 and cvn-app11 * 20:46 Krinkle: Create new cvn-apache10, cvn-app10, cvn-app11 with Debian 11 Bullseye to replace the old Debian 9.1 Stretch instances * 20:04 taavi: bump floating ip quota from 2 to 4, [[phab:T326269|T326269]] === 2022-12-27 === * 20:11 Frosty873: /cs flags #cvn-meta xaosflux voiced * 20:11 Frosty873: /cs flags #cvn-wp-en xaosflux voiced === 2022-12-23 === * 03:25 AntiComposite: /cs flags #cvn-meta tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-mediawiki tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-sw tryvix1509 voiced === 2022-10-18 === * 23:13 Joan: CVNBot3 restarted (Last message was received on RCReader 62854.814658 seconds ag) === 2022-09-04 === * 22:21 Operator873: /cs flags #cvn-simplewikis Enfcer +AV * 02:20 Operator873: /cs flags #cvn-sw Bot873 +voiced === 2022-08-26 === * 14:09 hauskatze: Loaded pcm.wikipedia and guw.wiktionary to CVNBot8 & 9 respectively {{!}} [[phab:T310880|T310880]] [[phab:T309057|T309057]] === 2022-07-09 === * 16:42 AntiComposite: /cs flags #cvn-commons pandakekok9 voiced === 2022-07-08 === * 21:53 Krinkle: krinkle@horizon.wikimedia.org Add anticomposite as project member and project admin to cloudvps.cvn === 2022-07-01 === * 21:39 Krinkle: cvn-app8: kill CVNBot14.exe and two (!) procs for CVNBot18.exe === 2022-06-25 === * 03:25 AntiComposite: /cs flags #cvn-wp-en PhantomTech voiced === 2022-06-22 === * 21:04 op873: <+CVNBot3> Added: LuchoCR is on es.wikipedia bot list, added by Operator873{{!}}CVN until the end of time ("Mass blockiing P2P-proxies with script") * 20:34 op873: restart CVNBot3 (possibly caused by block flood) * 19:31 op873: restart CVNBot3 === 2022-06-15 === * 18:49 AntiComposite: /cs flags #cvn-wp-en Zppix voiced * 18:48 AntiComposite: /cs flags #cvn-simplewikis Zppix voiced === 2022-05-23 === * 00:24 Joan: Flags +AV were set on Sargento in cvn-wp-es * 00:23 Joan: Flags +AV were set on alhen in cvn-wp-es === 2022-05-19 === * 23:10 Joan: CVNBot3 restarted (Last message was received on RCReader 92593.747667 seconds ago) === 2022-05-11 === * 07:34 Operator873: /cs flags #cvn-wp-en Tamzin voiced === 2022-05-07 === * 17:40 Operator873: /cs flags #cvn-sw koi voiced * 17:39 Operator873: /cs flags #cvn-zh-scan koi voiced === 2022-04-28 === * 03:19 Joan: CVNBot3 restarted (Last message was received on RCReader 75273.332577 seconds ago) === 2022-04-22 === * 15:08 AntiComposite: /cs flags #cvn-meta Bsadowski1 voiced === 2022-04-18 === * 20:44 AntiComposite: /cs flags #cvn-sw Vermont voiced === 2022-04-13 === * 22:40 Operator873: /cs flags #cvn-meta Joan voiced * 22:40 Operator873: /cs flags #cvn-sw Joan voiced * 22:14 Joan: CVNBot3 restarted (Last message was received on RCReader 54942.175428 seconds ago) === 2022-04-07 === * 23:15 Operator873: /cs flags #cvn-wp-hr NovakWatchmen local_op * 23:13 Operator873: voiced Superpes (Superpes15) in #cvn-sw #cvn-sw-spam and #cvn-it-scan === 2022-04-04 === * 17:34 Operator873: Voiced Vermont in #cvn-meta and #cvn-simplewikis /cs flags #cvn-meta Vermont voiced === 2022-03-30 === * 14:33 Joan: CVNBot3 restarted (Last message was received on RCReader 26318.335196 seconds ago) === 2022-03-28 === * 02:38 AntiComposite: /cs flags #cvn-wp-en Bsoyka voiced === 2022-03-21 === * 20:22 Operator873: /cs flags #cvn-simplewikis Bsadowski1 +AfiotvV * 20:17 Operator873: Operator873{{!}}CVN (Operator873) set flags +AVfitv on Bsadowski1 * 20:03 Operator873: Operator873{{!}}CVN (Operator873) set flags +V on Bsadowski1 * 17:04 AntiComposite: /cs flags #cvn-sw Bsadowski1 local_op === 2022-03-15 === * 15:38 Joan: CVNBot3 restarted (Last message was received on RCReader 26424.279343 seconds ago) === 2022-03-14 === * 14:02 Joan: CVNBot3 restarted (Last message was received on RCReader 17096.72183 seconds ago) === 2022-03-12 === * 16:27 Joan: CVNBot3 restarted (Last message was received on RCReader 27236.775673 seconds ago) === 2022-03-11 === * 14:24 Joan: CVNBot3 restarted (Last message was received on RCReader 18853.006849 seconds ago) === 2022-03-10 === * 14:08 Joan: CVNBot3 restarted (Last message was received on RCReader 22518.614282 seconds ago) === 2022-03-08 === * 20:27 AntiComposite: /cs flags #cvn-wp-en Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-simplewikis Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-commons Sarrus voiced === 2022-03-07 === * 16:30 AntiComposite: /cs flags #cvn-meta zabe voiced * 16:25 AntiComposite: /cs flags #cvn-simplewikis DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-meta DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-sw TheresNoTime voiced * 16:07 Krinkle: /cs flags #cvn-staff Operator873 staff * 16:07 Krinkle: /cs flags #cvn-staff AntiComposite staff === 2022-03-05 === * 04:13 Joan: CVNBot3 restarted (Last message was received on RCReader 31573.894101 seconds ago) === 2022-03-03 === * 16:39 Joan: CVNBot3 restarted (Last message was received on RCReader 36578.236383 seconds ago) === 2022-03-01 === * 13:21 Joan: CVNBot3 restarted (Last message was received on RCReader 20646.781861 seconds ago) === 2022-02-15 === * 14:12 Joan: CVNBot3 restarted (Last message was received on RCReader 25001.391103 seconds ago) === 2022-02-13 === * 18:47 andrewbogott: switching to project-local nfs server cvn-nfs-1 * 17:54 andrewbogott: switching to project-local nfs server puppet-diffs-nfs-1 === 2022-02-10 === * 16:17 Joan: CVNBot3 restarted (Last message was received on RCReader 39817.871151 seconds ago) === 2022-02-08 === * 15:51 Joan: CVNBot3 restarted (Last message was received on RCReader 28868.916144 seconds ago) === 2022-02-04 === * 23:59 andrewbogott: accidentally restarted all VMs due to misreading the project purge page. sorry! === 2022-02-02 === * CVN: Several bots restarted after netsplit took nickserv and some bots with it. * 10:26 Krinkle: CVNBot1 bes del delete(?!d) — originally added by huh (reason: "widewuto") === 2022-02-01 === * 15:20 Joan: CVNBot3 restarted (Last message was received on RCReader 26990.323435 seconds ago) === 2022-01-31 === * 17:37 Joan: CVNBot3 restarted (Last message was received on RCReader 48827.882566 seconds ago) === 2022-01-27 === * 16:58 Joan: CVNBot3 restarted (Last message was received on RCReader 29206.852828 seconds ago) === 2022-01-21 === * 16:07 Joan: CVNBot3 restarted (Last message was received on RCReader 22091.557102 seconds ago) === 2022-01-20 === * 18:13 Cam11598: CVNBot15 restarted === 2022-01-19 === * 17:26 Joan: Restarted CVNBot3 (Last message was received on RCReader 28129.031916 seconds ago) === 2022-01-18 === * 16:55 Joan: Restarted CVNBot3 (Last message was received on RCReader 26283.381782 seconds ago) === 2022-01-17 === * 16:33 Joan: Restarted CVNBot3 (#cvn-wp-es) (Last message was received on RCReader 197065.877109 seconds ago) === 2022-01-15 === * 04:56 Cam11598: restarted CVNBOT18 8:55:47 PM <�25B100+ CVNBot18> Last message was received on RCReader 29723.456263 seconds ago === 2022-01-13 === * 01:29 Cam11598: restarted CVNBot2 nickserv issue * 01:29 Cam11598: restarted CVNBot18 - no response from RC feed === 2022-01-09 === * 18:18 Joan: Flags +AV were set on Hasley in cvn-wp-es (sysop at es.wikipedia) * 17:56 Krinkle: /cs flags #cvn-wp-es Joan local_op === 2022-01-07 === * 22:08 hauskatze: CVNBot9 load co.wiktionary wikt:co: * 22:04 hauskatze: CVNBot9 load ban.wikisource s:ban: * 22:04 hauskatze: CVNBot9 load ba.wikibooks b:ba: * 10:51 hauskatze: Loaded alt.wikipedia to Group 4 (CVNBot9) - small wiki not monitored === 2022-01-06 === * 19:42 hauskatze: Loaded ami.wikipedia to CVNBot8 - [[phab:T292421|T292421]] * 19:41 hauskatze: Loaded pwn.wikipedia to CVNBot7 - [[phab:T292419|T292419]] * 19:39 hauskatze: Loaded lmo.wiktionary to CVNBot6 - [[phab:T292076|T292076]] * 19:34 hauskatze: Loaded jv.wikisource to CVNBot6 refs. [[phab:T287319|T287319]] * 19:29 Krinkle: cs flags #cvn-sw hauskatze local_op * 13:57 Krinkle: Krinkle added $a:Cam11598 to the #cvn-staff I list (+I) {{SAL|Project Name=cvn}} <noinclude> ==Archives== * [[Nova Resource:Cvn/SAL/Archive 1|Archive 1]] (2006-2009) * [[Nova Resource:Cvn/SAL/Archive 2|Archive 2]] (2010-2011) * [[Nova Resource:Cvn/SAL/Archive 3|Archive 3]] (2012-2013) * [[Nova Resource:Cvn/SAL/Archive 4|Archive 4]] (2013-2021) (some parts in 2013 are not indexed) [[Category:SAL]]</noinclude> 8igpo8fj0uqezhkrb99qy62jmkais1t 2426638 2426637 2026-06-13T23:12:45Z Stashbot 7414 AntiComposite: CVNBot25 drop & purge ko.wikinews (T428622) 2426638 wikitext text/x-wiki === 2026-06-13 === * 23:12 AntiComposite: CVNBot25 drop & purge ko.wikinews ([[phab:T428622|T428622]]) * 23:12 AntiComposite: CVNBot23 drop & purge zh.wikinews ([[phab:T428622|T428622]]) * 23:11 AntiComposite: CVNBot10 drop & purge ca.wikinews, ko.wikinews, no.wikinews ([[phab:T428622|T428622]]) * 23:07 AntiComposite: CVNBot9 drop & purge bs.wikinews, el.wikinews, fa.wikinews, shn.wikinews, zh.wikinews ([[phab:T428622|T428622]]) * 23:03 AntiComposite: CVNBot8 drop & purge ar.wikinews, cs.wikinews, de.wikinews, fi.wikinews, he.wikinews, ru.wikinews, sq.wikinews, sr.wikinews, uk.wikinews ([[phab:T428622|T428622]]) * 22:58 AntiComposite: CVNBot7 drop & purge es.wikinews, guw.wikinews, pt.wikinews ([[phab:T428622|T428622]]) * 22:56 AntiComposite: CVNBot6 drop & purge eo.wikinews, fr.wikinews, pl.wikinews, ro.wikinews, sv.wikinews, ta.wikinews ([[phab:T428622|T428622]]) * 22:49 AntiComposite: CVNBot4 drop it.wikinews ([[phab:T428622|T428622]]) === 2026-06-02 === * 01:03 Krinkle: /cs flags #cvn-sw Divinations voiced === 2026-05-26 === * 18:07 AntiComposite: restart all bots -- disconnected === 2026-05-03 === * 13:39 Krinkle: Disable "Admin immed notify" for cvn-private https://lists.wikimedia.org/postorius/lists/cvn-private.lists.wikimedia.org/settings/automatic_responses. We previously removed the sub form but this is no longer supported in mailman3. We require confirm/moderate for new subs, there is no way to turn it off. But we can at least disable the noise. === 2026-04-27 === * 12:22 Krinkle: /cs flags #cvn-meta NathanVeritas voiced === 2026-04-01 === * 13:34 AntiComposite: restart all bots === 2026-02-04 === * 20:33 AntiComposite: Restart all bots === 2025-12-26 === * 15:54 Operator873: /cs flags #cvn-zh-scan nya_1F616EMO voiced === 2025-11-27 === * 13:48 AntiComposite: CVNBot10 load tok.wikipedia tok: ([[phab:T404567|T404567]]) * 13:47 AntiComposite: CVNBot9 load ms.wikiquote q:ms: ([[phab:T404700|T404700]]) * 13:45 AntiComposite: CVNBot8 load min.wikisource s:min: ([[phab:T408343|T408343]]) * 13:44 AntiComposite: CVNBot7 load pcm.wikiquote q:pcm: ([[phab:T408351|T408351]]) * 13:43 AntiComposite: CVNBot6 load tl.wikisource s:tl: ([[phab:T388654|T388654]]) * 13:42 AntiComposite: CVNBot10 load bew.wiktionary wikt:bew: ([[phab:T402134|T402134]]) * 13:41 AntiComposite: CVNBot9 load zgh.wiktionary wikt:zgh: ([[phab:T399785|T399785]]) * 13:40 AntiComposite: CVNBot8 load min.wikibooks b:min: ([[phab:T395499|T395499]]) * 13:38 AntiComposite: CVNBot7 load rki.wikipedia rki: ([[phab:T392499|T392499]]) * 13:37 AntiComposite: CVNBot6 load mad.wikisource s:mad: ([[phab:T391767|T391767]]) === 2025-10-28 === * 23:16 AntiComposite: /cs flags #cvn-commons revi local_op === 2025-08-20 === * 20:35 AntiComposite: CVNBot10 load nup.wikipedia nup: ([[phab:T390711|T390711]]) === 2025-07-11 === * 14:38 AntiComposite: cvn-app10 restart all bots * 11:10 AntiComposite: cvn-app12 restart all bots * 11:09 AntiComposite: cvn-app10 restart all bots === 2025-06-20 === * 20:49 AntiComposite: cvn-app12: restart all bots * 20:48 AntiComposite: cvn-app10: restart all bots === 2025-05-26 === * 17:59 Krinkle: Create cvn-app14 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:59 Krinkle: Create cvn-app13 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:57 Krinkle: Delete cvn-apache10 instance (replaced/shutdown 2 days ago), ref [[phab:T395164|T395164]] === 2025-05-23 === * 20:30 Krinkle: Shut off cvn-apache10, [[phab:T395164|T395164]] * 20:29 Krinkle: Change cvn.wmcloud.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 20:22 Krinkle: Change cvn.wmflabs.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 19:45 Krinkle: Create cvn-apache11 (debian-12.0-bookworm, g4.cores2.ram4.disk20), [[phab:T395164|T395164]]) === 2025-05-16 === * 18:22 Krinkle: Replace outreach.wikipedia with outreach.wikimedia in cvn-sw/CVNBot19 per https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/820245 since the source channel was renamed * 17:30 Krinkle: krinkle@cvn-apache10:/srv/cvn/git/infrastructure$ git pull -- Deploy https://gerrit.wikimedia.org/r/1146724 * 17:30 Krinkle: krinkle@cvn-apache10 Update git remote in /srv/cvn/git/infrastructure from github.com/countervandalism to https://gerrit.wikimedia.org/r/labs/countervandalism/cvn-infrastructure === 2025-04-21 === * 17:22 AntiComposite: Hard reboot cvn-app10, flapping and not responsive to ssh === 2025-03-30 === * 06:55 Krinkle: krinkle@cvn-apache10: Run `sudo chmod 644 /srv/cvn/git/infrastructure/crontab-config/*.cron`, per [[phab:T390415|T390415]] === 2025-03-12 === * 02:18 AntiComposite: CVNBot9 load id.wikivoyage voy:id: ([[phab:T381080|T381080]]) * 02:15 AntiComposite: CVNBot8 load tig.wikipedia tig: ([[phab:T381379|T381379]]) * 02:14 AntiComposite: CVNBot7 load knc.wikipedia knc: ([[phab:T385185|T385185]]) * 02:11 AntiComposite: CVNBot6 load syl.wikipedia syl: ([[phab:T386464|T386464]]) * 02:08 AntiComposite: CVNBot10 load sat.wiktionary wikt:sat: ([[phab:T386631|T386631]]) === 2025-02-03 === * 22:05 AntiComposite: Hard reboot cvn-apache10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ * 21:58 AntiComposite: Hard reboot cvn-app10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ === 2025-01-02 === * 12:46 Krinkle: /cs flags #cvn-wp-en Lordseriouspig voiced * 12:45 Krinkle: /cs flags #cvn-sw Lordseriouspig voiced === 2024-11-23 === * 00:41 AntiComposite: CVNBot9 load ka.wikisource s:ka: ([[phab:T363243|T363243]]) * 00:38 AntiComposite: CVNBot8 load tcy.wikisource s:tcy: ([[phab:T378471|T378471]]) * 00:37 AntiComposite: CVNBot7 load tcy.wiktionary wikt:tcy: ([[phab:T378463|T378463]]) * 00:25 AntiComposite: Upgrade CVNBot29 to v4.0.4 * 00:25 AntiComposite: Upgrade CVNBot28 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot27 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot26 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot25 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot24 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot23 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot22 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot19 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot17 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot16 to v4.0.4 * 00:20 AntiComposite: Upgrade CVNBot10 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot9 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot8 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot7 to v4.0.4 * 00:17 AntiComposite: Upgrade CVNBot6 to v4.0.4 === 2024-11-22 === * 23:52 AntiComposite: Upgrade CVNBot21 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot20 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot18 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot15 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot14 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot13 to v4.0.4 * 23:49 AntiComposite: Upgrade CVNBot12 to v4.0.4 * 23:48 AntiComposite: Upgrade CVNBot11 to v4.0.4 * 23:47 AntiComposite: Upgrade CVNBot5 to v4.0.4 * 23:45 AntiComposite: Upgrade CVNBot3 to v4.0.4 * 23:44 AntiComposite: Upgrade CVNBot2 to v4.0.4 * 23:41 AntiComposite: Upgrade CVNBot1 to v4.0.4 * 23:32 AntiComposite: Upgrade CVNBot4 to v4.0.4 * 17:08 AntiComposite: restart CVNBots on cvn-app12 due to simultaneous RCReader failure 91950.519949 seconds === 2024-11-08 === * 23:24 AntiComposite: Restarting all CVNBots due to simultaneous RCReader disconnect 54323.128318 seconds ago === 2024-10-29 === * 20:56 AntiComposite: add sh.wikipedia to CVNBot6 as #cvn-wp-sh didn't survive the libera migration * 14:22 AntiComposite: restart all CVNBots === 2024-10-28 === * 12:50 AntiComposite: restarting all CVNBots, not coming up cleanly === 2024-10-25 === * 02:23 AntiComposite: add cs.wikivoyage to CVNBot10 ([[phab:T370913|T370913]]) * 02:21 AntiComposite: add bdr.wikipedia to CVNBot9 ([[phab:T371760|T371760]]) * 02:18 AntiComposite: add mos.wikipedia to CVNBot8 ([[phab:T374644|T374644]]) * 02:14 AntiComposite: add kge.wikipedia to CVNBot7 ([[phab:T374815|T374815]]) * 02:11 AntiComposite: add rsk.wikipedia to CVNBot6 ([[phab:T375017|T375017]]) * 02:07 AntiComposite: add mad.wiktionary to CVNBot9 ([[phab:T375024|T375024]]) * 02:06 AntiComposite: add gor.wikiquote to CVNBot8 ([[phab:T375095|T375095]]) * 02:04 AntiComposite: add nr.wikipedia to CVNBot7 ([[phab:T375102|T375102]]) * 02:01 AntiComposite: add tdd.wikipedia to CVNBot6 ([[phab:T375424|T375424]]) * 01:54 AntiComposite: add shn.wikinews to CVNBot9 ([[phab:T375433|T375433]]) * 01:52 AntiComposite: add iba.wikipedia to CVNBot8 ([[phab:T376572|T376572]]) * 01:50 AntiComposite: add bcl.wikisource to CVNBot7 ([[phab:T377088|T377088]]) * 01:47 AntiComposite: add ann.wikipedia to CVNBot6 ([[phab:T377160|T377160]]) * 01:43 AntiComposite: add igl.wikipedia to CVNBot9 ( [[phab:T363263|T363263]] ) * 01:41 AntiComposite: add my.wikisource to CVNBot8 ([[phab:T363270|T363270]]) * 01:39 AntiComposite: add foundation.wikimedia to CVNBot19 * 01:38 AntiComposite: add wikitech.wikimedia to CVNBot19 === 2024-10-24 === * 11:36 AntiComposite: restart all CVNBots === 2024-10-23 === * 17:33 AntiComposite: restart all CVNBots === 2024-07-03 === * 02:00 AntiComposite: add kus.wikipedia to CVNBot7 ([[phab:T360303|T360303]]) * 01:57 AntiComposite: add bew.wikipedia to CVNBot6 ([[phab:T360310|T360310]]) * 01:54 AntiComposite: add ms.wikisource to CVNBot9 ([[phab:T363250|T363250]]) * 01:53 AntiComposite: add kaa.wiktionary to CVNBot8 ([[phab:T363256|T363256]]) * 01:50 AntiComposite: add dtp.wikipedia to CVNBot7 ([[phab:T365230|T365230]]) * 01:48 AntiComposite: add btm.wikipedia to CVNBot6 ([[phab:T368067|T368067]]) * 01:45 AntiComposite: add fon.wikipedia to CVNBot9 ([[phab:T347939|T347939]]) * 01:43 AntiComposite: add blk.wikisource to CVNBot8 ([[phab:T343542|T343542]]) * 01:41 AntiComposite: su.wikisource to CVNBot7 ([[phab:T343548|T343548]]) * 01:39 AntiComposite: add tly.wikipedia to CVNBot6 ([[phab:T345170|T345170]]) * 01:37 AntiComposite: add dga.wikipedia to CVNBot9 ([[phab:T350229|T350229]]) * 01:35 AntiComposite: add bjn.wikiquote to CVNBot8 ([[phab:T350235|T350235]]) * 01:32 AntiComposite: add zgh.wikipedia to CVNBot7 ([[phab:T350241|T350241]]) * 01:28 AntiComposite: add bbc.wikipedia to CVNBot6 ([[phab:T350373|T350373]]) === 2024-06-24 === * 16:40 Krinkle: cvn-clerkbot parts #cvn-unifications (not operated by CVN, renamed to #wikimedia-unifications) === 2024-06-18 === * 08:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs === 2024-03-22 === * 05:30 Operator873: /cs flags #cvn-simplewikis Drummingman +voice === 2024-02-28 === * 21:34 Krinkle: /cs flags #cvn-wp-da Sarrus local_op === 2024-01-11 === * 12:19 AntiComposite: /cs flags #cvn-meta Bsadowski1 local_op === 2023-12-01 === * 15:30 AntiComposite: restart everything after WMCS network outage === 2023-10-07 === * 14:50 AntiComposite: kill 2 CVNBot11 processes and restart, bot not joined to IRC === 2023-09-22 === * 00:06 Op873: /cs flags #cvn-wp-en Oshwah +AV === 2023-09-16 === * 10:33 JackSparrow: /cs flags #cvn-wp-fa Arian_Ar local_op === 2023-09-07 === * 01:35 AntiComposite: restart all cvn-app12 bots * 01:33 AntiComposite: restart all cvn-app10 bots === 2023-08-15 === * 14:44 AntiComposite: reboot cvn-app10 from Horizon, bots dead and not responding to SSH === 2023-08-09 === * 00:07 AntiComposite: add 9 wikis to #cvn-sw (ref [[phab:T332379|T332379]] [[phab:T336115|T336115]] [[phab:T332093|T332093]] [[phab:T332093|T332093]] [[phab:T335987|T335987]] [[phab:T334459|T334459]] [[phab:T333271|T333271]] [[phab:T334740|T334740]] [[phab:T342865|T342865]]) === 2023-08-08 === * 23:46 AntiComposite: drop wo.wikiquote from CVNBot10 (closed) [[phab:T334482|T334482]] === 2023-07-27 === * 18:15 AntiComposite: Kill and restart CVNBot29 on cvn-app12 === 2023-07-06 === * 16:21 AntiComposite: point git repos to gerrit on cvn-app10 * 16:19 AntiComposite: point git repos to gerrit on cvn-app12 * 16:03 AntiComposite: CVNBot v4.0.3 deployed to all bots ([[phab:T327126|T327126]], [[phab:T327127|T327127]]) * 16:01 AntiComposite: Upgrade CVNBot29 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot28 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot27 to v4.0.3 * 15:59 AntiComposite: Upgrade CVNBot26 to v4.0.3 * 15:58 AntiComposite: Upgrade CVNBot25 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot24 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot23 to v4.0.3 * 15:55 AntiComposite: Upgrade CVNBot22 to v4.0.3 * 15:54 AntiComposite: Upgrade CVNBot19 to v4.0.3 * 15:53 AntiComposite: Upgrade CVNBot17 to v4.0.3 * 15:46 AntiComposite: Upgrade CVNBot16 to v4.0.3 * 15:44 AntiComposite: Upgrade CVNBot10 to v4.0.3 * 15:41 AntiComposite: Upgrade CVNBot9 to v4.0.3 * 15:40 AntiComposite: Upgrade CVNBot8 to v4.0.3 * 15:39 AntiComposite: Upgrade CVNBot7 to v4.0.3 * 15:38 AntiComposite: Upgrade CVNBot6 to v4.0.3 * 04:37 AntiComposite: Upgrade CVNBot21 to v4.0.3 * 04:34 AntiComposite: Upgrade CVNBot20 to v4.0.3 * 04:33 AntiComposite: Upgrade CVNBot18 to v4.0.3 * 04:30 AntiComposite: Upgrade CVNBot15 to v4.0.3 * 04:23 AntiComposite: Upgrade CVNBot14 to v4.0.3 * 04:22 AntiComposite: Upgrade CVNBot13 to v4.0.3 * 04:14 AntiComposite: Upgrade CVNBot12 to v4.0.3 * 04:09 AntiComposite: Upgrade CVNBot11 to v4.0.3 * 04:03 AntiComposite: Upgrade CVNBot5 to v4.0.3 * 04:01 AntiComposite: Upgrade CVNBot4 to v4.0.3 * 04:00 AntiComposite: Upgrade CVNBot3 to v4.0.3 * 03:57 AntiComposite: Upgrade CVNBot2 to v4.0.3 * 03:51 AntiComposite: Upgrade CVNBot1 to v4.0.3 === 2023-06-28 === * 02:34 Operator873: /cs flags #cvn-sw Fehufanga voiced === 2023-06-16 === * 22:05 AntiComposite: manually restart cvn-clerkbot === 2023-05-15 === * 14:58 hauskater: Dropped akwiki and nawiki from CVNBot10 as closed wikis. On-wiki lists require an update. === 2023-04-26 === * 20:07 AntiComposite: /cs flags #cvn-mk-scan M4r51n voiced === 2023-04-21 === * 22:12 Operator873: granted voice to Fehufanga in #cvn-simplewikis === 2023-04-14 === * 18:28 AntiComposite: restart cvn-app10 from horizon, bots quit and ssh times out === 2023-03-22 === * 03:33 Operator873: Voiced Tulsi in #cvn-sw -meta -mediawiki -commons -simplewikis === 2023-03-13 === * 19:46 Operator873: CVNBot18 restarted === 2023-03-03 === * 14:45 AntiComposite: /cs flags #cvn-sw-spam COIBot bot === 2023-02-27 === * 22:33 herzog: Loaded gur.wikipedia to SWMT Group 4 (CVNBot9) - [[phab:T327842|T327842]] * 18:04 herzog: Loaded guc.wikipedia to CVNBot9 / Group 4 - [[phab:T326236|T326236]] === 2023-02-02 === * 00:21 ma: Added 12 new wikis to CVNBot<nowiki>{</nowiki>6,7,8<nowiki>}</nowiki>, 4 to each one. Refs.: [[phab:T321283|T321283]] [[phab:T321289|T321289]] [[phab:T321295|T321295]] [[phab:T326139|T326139]] [[phab:T305281|T305281]] [[phab:T310873|T310873]] [[phab:T312215|T312215]] [[phab:T314640|T314640]] [[phab:T314646|T314646]] [[phab:T316457|T316457]] [[phab:T317113|T317113]] [[phab:T319191|T319191]] === 2023-01-30 === * 22:50 Krinkle: Delete cvn-app8 and cvn-app9 instances, ref [[phab:T306066|T306066]] === 2023-01-28 === * 02:51 AntiComposite: /cs flags #cvn-sw Ajraddatz local_op === 2023-01-24 === * 08:54 Krinkle: Delete cvn-apache9, [[phab:T306066|T306066]] * 08:54 Krinkle: Suspend cvn-app8 and cvn-app9 (`pgrep -af cvn` is empty on both), [[phab:T306066|T306066]] === 2023-01-23 === * 16:53 AntiComposite: Deploy {{Gerrit|716e140}} to app12 ([[phab:T306066|T306066]]) * 16:50 AntiComposite: Deploy {{Gerrit|716e140}} to app9 ([[phab:T306066|T306066]]) * 16:29 AntiComposite: Deploy {{Gerrit|442f324}} to app12 ([[phab:T306066|T306066]]) * 16:25 AntiComposite: Deploy {{Gerrit|442f324}} to app9 ([[phab:T306066|T306066]]) * 16:01 AntiComposite: Deploy {{Gerrit|9024b8f}} to app12 ([[phab:T306066|T306066]]) * 15:59 AntiComposite: Deploy {{Gerrit|9024b8f}} to app9 ([[phab:T306066|T306066]]) === 2023-01-22 === * 21:40 AntiComposite: start cvndb-CVNBot14-publish on app10 * 21:07 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app10, starting bots ([[phab:T306066|T306066]]) * 20:56 AntiComposite: disable cvndb-CVNBot14-publish on app8 * 20:51 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app8, stopping bots ([[phab:T306066|T306066]]) * 19:53 AntiComposite: Deploy {{Gerrit|80ea1f5}} to cvn-app10 ([[phab:T306066|T306066]]) * 15:43 AntiComposite: restart all CVNBots on app9 * 15:42 AntiComposite: restart all CVNBots on app8 === 2023-01-17 === * 00:15 Krinkle: Suspend cvn-apache9, replaced by cvn-apache10, ref [[phab:T306066|T306066]] * 00:14 Krinkle: Switch cvn.wmflabs.org from cvn-apache9 to cvn-apache10 === 2023-01-16 === * 00:10 Krinkle: Move https://github.com/countervandalism/cvn-clerkbot to https://github.com/wikimedia/countervandalism-cvn-clerkbot (with HTTP and Git redirect preserved), and replace with Gerrit mirror === 2023-01-15 === * 23:12 Krinkle: Create 'labs-cvn' permission group in Gerrit with CVN staff members * 23:12 Krinkle: Move https://github.com/countervandalism/cvn-api to https://github.com/wikimedia/countervandalism-cvn-api (with HTTP and Git redirect preserved), and replace with Gerrit mirror * 22:02 Krinkle: Switch new cvn.wmcloud.org proxy from cvn-apache9 to cvn-apache10 (Leave main cvn.wmflabs.org as-is for now). === 2023-01-14 === * 21:45 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|4cee27a}}) * 21:22 AntiComposite: move cvn-clerbot back to cvn-app9 (deploy {{Gerrit|371ba2a}}) * 21:10 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|3f3f40f}}) === 2023-01-10 === * 23:22 Krinkle: krinkle@cvn-apache9$ update infrastructure.git, sudo apachectl graceful * 23:20 Krinkle: Create cvn.wmcloud.org web proxy (in addition to cvn.wmflabs.org) === 2023-01-07 === * 20:53 AntiComposite: apply role::labs::lvm::srv only to cvn-apache9, cvn-app8, and cvn-app9 to fix puppet failures on new instances === 2023-01-04 === * 20:47 Krinkle: Allocate new floating IPs to cvn-app10 and cvn-app11 * 20:46 Krinkle: Create new cvn-apache10, cvn-app10, cvn-app11 with Debian 11 Bullseye to replace the old Debian 9.1 Stretch instances * 20:04 taavi: bump floating ip quota from 2 to 4, [[phab:T326269|T326269]] === 2022-12-27 === * 20:11 Frosty873: /cs flags #cvn-meta xaosflux voiced * 20:11 Frosty873: /cs flags #cvn-wp-en xaosflux voiced === 2022-12-23 === * 03:25 AntiComposite: /cs flags #cvn-meta tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-mediawiki tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-sw tryvix1509 voiced === 2022-10-18 === * 23:13 Joan: CVNBot3 restarted (Last message was received on RCReader 62854.814658 seconds ag) === 2022-09-04 === * 22:21 Operator873: /cs flags #cvn-simplewikis Enfcer +AV * 02:20 Operator873: /cs flags #cvn-sw Bot873 +voiced === 2022-08-26 === * 14:09 hauskatze: Loaded pcm.wikipedia and guw.wiktionary to CVNBot8 & 9 respectively {{!}} [[phab:T310880|T310880]] [[phab:T309057|T309057]] === 2022-07-09 === * 16:42 AntiComposite: /cs flags #cvn-commons pandakekok9 voiced === 2022-07-08 === * 21:53 Krinkle: krinkle@horizon.wikimedia.org Add anticomposite as project member and project admin to cloudvps.cvn === 2022-07-01 === * 21:39 Krinkle: cvn-app8: kill CVNBot14.exe and two (!) procs for CVNBot18.exe === 2022-06-25 === * 03:25 AntiComposite: /cs flags #cvn-wp-en PhantomTech voiced === 2022-06-22 === * 21:04 op873: <+CVNBot3> Added: LuchoCR is on es.wikipedia bot list, added by Operator873{{!}}CVN until the end of time ("Mass blockiing P2P-proxies with script") * 20:34 op873: restart CVNBot3 (possibly caused by block flood) * 19:31 op873: restart CVNBot3 === 2022-06-15 === * 18:49 AntiComposite: /cs flags #cvn-wp-en Zppix voiced * 18:48 AntiComposite: /cs flags #cvn-simplewikis Zppix voiced === 2022-05-23 === * 00:24 Joan: Flags +AV were set on Sargento in cvn-wp-es * 00:23 Joan: Flags +AV were set on alhen in cvn-wp-es === 2022-05-19 === * 23:10 Joan: CVNBot3 restarted (Last message was received on RCReader 92593.747667 seconds ago) === 2022-05-11 === * 07:34 Operator873: /cs flags #cvn-wp-en Tamzin voiced === 2022-05-07 === * 17:40 Operator873: /cs flags #cvn-sw koi voiced * 17:39 Operator873: /cs flags #cvn-zh-scan koi voiced === 2022-04-28 === * 03:19 Joan: CVNBot3 restarted (Last message was received on RCReader 75273.332577 seconds ago) === 2022-04-22 === * 15:08 AntiComposite: /cs flags #cvn-meta Bsadowski1 voiced === 2022-04-18 === * 20:44 AntiComposite: /cs flags #cvn-sw Vermont voiced === 2022-04-13 === * 22:40 Operator873: /cs flags #cvn-meta Joan voiced * 22:40 Operator873: /cs flags #cvn-sw Joan voiced * 22:14 Joan: CVNBot3 restarted (Last message was received on RCReader 54942.175428 seconds ago) === 2022-04-07 === * 23:15 Operator873: /cs flags #cvn-wp-hr NovakWatchmen local_op * 23:13 Operator873: voiced Superpes (Superpes15) in #cvn-sw #cvn-sw-spam and #cvn-it-scan === 2022-04-04 === * 17:34 Operator873: Voiced Vermont in #cvn-meta and #cvn-simplewikis /cs flags #cvn-meta Vermont voiced === 2022-03-30 === * 14:33 Joan: CVNBot3 restarted (Last message was received on RCReader 26318.335196 seconds ago) === 2022-03-28 === * 02:38 AntiComposite: /cs flags #cvn-wp-en Bsoyka voiced === 2022-03-21 === * 20:22 Operator873: /cs flags #cvn-simplewikis Bsadowski1 +AfiotvV * 20:17 Operator873: Operator873{{!}}CVN (Operator873) set flags +AVfitv on Bsadowski1 * 20:03 Operator873: Operator873{{!}}CVN (Operator873) set flags +V on Bsadowski1 * 17:04 AntiComposite: /cs flags #cvn-sw Bsadowski1 local_op === 2022-03-15 === * 15:38 Joan: CVNBot3 restarted (Last message was received on RCReader 26424.279343 seconds ago) === 2022-03-14 === * 14:02 Joan: CVNBot3 restarted (Last message was received on RCReader 17096.72183 seconds ago) === 2022-03-12 === * 16:27 Joan: CVNBot3 restarted (Last message was received on RCReader 27236.775673 seconds ago) === 2022-03-11 === * 14:24 Joan: CVNBot3 restarted (Last message was received on RCReader 18853.006849 seconds ago) === 2022-03-10 === * 14:08 Joan: CVNBot3 restarted (Last message was received on RCReader 22518.614282 seconds ago) === 2022-03-08 === * 20:27 AntiComposite: /cs flags #cvn-wp-en Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-simplewikis Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-commons Sarrus voiced === 2022-03-07 === * 16:30 AntiComposite: /cs flags #cvn-meta zabe voiced * 16:25 AntiComposite: /cs flags #cvn-simplewikis DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-meta DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-sw TheresNoTime voiced * 16:07 Krinkle: /cs flags #cvn-staff Operator873 staff * 16:07 Krinkle: /cs flags #cvn-staff AntiComposite staff === 2022-03-05 === * 04:13 Joan: CVNBot3 restarted (Last message was received on RCReader 31573.894101 seconds ago) === 2022-03-03 === * 16:39 Joan: CVNBot3 restarted (Last message was received on RCReader 36578.236383 seconds ago) === 2022-03-01 === * 13:21 Joan: CVNBot3 restarted (Last message was received on RCReader 20646.781861 seconds ago) === 2022-02-15 === * 14:12 Joan: CVNBot3 restarted (Last message was received on RCReader 25001.391103 seconds ago) === 2022-02-13 === * 18:47 andrewbogott: switching to project-local nfs server cvn-nfs-1 * 17:54 andrewbogott: switching to project-local nfs server puppet-diffs-nfs-1 === 2022-02-10 === * 16:17 Joan: CVNBot3 restarted (Last message was received on RCReader 39817.871151 seconds ago) === 2022-02-08 === * 15:51 Joan: CVNBot3 restarted (Last message was received on RCReader 28868.916144 seconds ago) === 2022-02-04 === * 23:59 andrewbogott: accidentally restarted all VMs due to misreading the project purge page. sorry! === 2022-02-02 === * CVN: Several bots restarted after netsplit took nickserv and some bots with it. * 10:26 Krinkle: CVNBot1 bes del delete(?!d) — originally added by huh (reason: "widewuto") === 2022-02-01 === * 15:20 Joan: CVNBot3 restarted (Last message was received on RCReader 26990.323435 seconds ago) === 2022-01-31 === * 17:37 Joan: CVNBot3 restarted (Last message was received on RCReader 48827.882566 seconds ago) === 2022-01-27 === * 16:58 Joan: CVNBot3 restarted (Last message was received on RCReader 29206.852828 seconds ago) === 2022-01-21 === * 16:07 Joan: CVNBot3 restarted (Last message was received on RCReader 22091.557102 seconds ago) === 2022-01-20 === * 18:13 Cam11598: CVNBot15 restarted === 2022-01-19 === * 17:26 Joan: Restarted CVNBot3 (Last message was received on RCReader 28129.031916 seconds ago) === 2022-01-18 === * 16:55 Joan: Restarted CVNBot3 (Last message was received on RCReader 26283.381782 seconds ago) === 2022-01-17 === * 16:33 Joan: Restarted CVNBot3 (#cvn-wp-es) (Last message was received on RCReader 197065.877109 seconds ago) === 2022-01-15 === * 04:56 Cam11598: restarted CVNBOT18 8:55:47 PM <�25B100+ CVNBot18> Last message was received on RCReader 29723.456263 seconds ago === 2022-01-13 === * 01:29 Cam11598: restarted CVNBot2 nickserv issue * 01:29 Cam11598: restarted CVNBot18 - no response from RC feed === 2022-01-09 === * 18:18 Joan: Flags +AV were set on Hasley in cvn-wp-es (sysop at es.wikipedia) * 17:56 Krinkle: /cs flags #cvn-wp-es Joan local_op === 2022-01-07 === * 22:08 hauskatze: CVNBot9 load co.wiktionary wikt:co: * 22:04 hauskatze: CVNBot9 load ban.wikisource s:ban: * 22:04 hauskatze: CVNBot9 load ba.wikibooks b:ba: * 10:51 hauskatze: Loaded alt.wikipedia to Group 4 (CVNBot9) - small wiki not monitored === 2022-01-06 === * 19:42 hauskatze: Loaded ami.wikipedia to CVNBot8 - [[phab:T292421|T292421]] * 19:41 hauskatze: Loaded pwn.wikipedia to CVNBot7 - [[phab:T292419|T292419]] * 19:39 hauskatze: Loaded lmo.wiktionary to CVNBot6 - [[phab:T292076|T292076]] * 19:34 hauskatze: Loaded jv.wikisource to CVNBot6 refs. [[phab:T287319|T287319]] * 19:29 Krinkle: cs flags #cvn-sw hauskatze local_op * 13:57 Krinkle: Krinkle added $a:Cam11598 to the #cvn-staff I list (+I) {{SAL|Project Name=cvn}} <noinclude> ==Archives== * [[Nova Resource:Cvn/SAL/Archive 1|Archive 1]] (2006-2009) * [[Nova Resource:Cvn/SAL/Archive 2|Archive 2]] (2010-2011) * [[Nova Resource:Cvn/SAL/Archive 3|Archive 3]] (2012-2013) * [[Nova Resource:Cvn/SAL/Archive 4|Archive 4]] (2013-2021) (some parts in 2013 are not indexed) [[Category:SAL]]</noinclude> pe516ocwjbt8nene61r6h678s5dmwbq 2426639 2426638 2026-06-13T23:13:26Z Stashbot 7414 AntiComposite: CVNBot26 drop & purge ar.wikinews (T428622) 2426639 wikitext text/x-wiki === 2026-06-13 === * 23:13 AntiComposite: CVNBot26 drop & purge ar.wikinews ([[phab:T428622|T428622]]) * 23:12 AntiComposite: CVNBot25 drop & purge ko.wikinews ([[phab:T428622|T428622]]) * 23:12 AntiComposite: CVNBot23 drop & purge zh.wikinews ([[phab:T428622|T428622]]) * 23:11 AntiComposite: CVNBot10 drop & purge ca.wikinews, ko.wikinews, no.wikinews ([[phab:T428622|T428622]]) * 23:07 AntiComposite: CVNBot9 drop & purge bs.wikinews, el.wikinews, fa.wikinews, shn.wikinews, zh.wikinews ([[phab:T428622|T428622]]) * 23:03 AntiComposite: CVNBot8 drop & purge ar.wikinews, cs.wikinews, de.wikinews, fi.wikinews, he.wikinews, ru.wikinews, sq.wikinews, sr.wikinews, uk.wikinews ([[phab:T428622|T428622]]) * 22:58 AntiComposite: CVNBot7 drop & purge es.wikinews, guw.wikinews, pt.wikinews ([[phab:T428622|T428622]]) * 22:56 AntiComposite: CVNBot6 drop & purge eo.wikinews, fr.wikinews, pl.wikinews, ro.wikinews, sv.wikinews, ta.wikinews ([[phab:T428622|T428622]]) * 22:49 AntiComposite: CVNBot4 drop it.wikinews ([[phab:T428622|T428622]]) === 2026-06-02 === * 01:03 Krinkle: /cs flags #cvn-sw Divinations voiced === 2026-05-26 === * 18:07 AntiComposite: restart all bots -- disconnected === 2026-05-03 === * 13:39 Krinkle: Disable "Admin immed notify" for cvn-private https://lists.wikimedia.org/postorius/lists/cvn-private.lists.wikimedia.org/settings/automatic_responses. We previously removed the sub form but this is no longer supported in mailman3. We require confirm/moderate for new subs, there is no way to turn it off. But we can at least disable the noise. === 2026-04-27 === * 12:22 Krinkle: /cs flags #cvn-meta NathanVeritas voiced === 2026-04-01 === * 13:34 AntiComposite: restart all bots === 2026-02-04 === * 20:33 AntiComposite: Restart all bots === 2025-12-26 === * 15:54 Operator873: /cs flags #cvn-zh-scan nya_1F616EMO voiced === 2025-11-27 === * 13:48 AntiComposite: CVNBot10 load tok.wikipedia tok: ([[phab:T404567|T404567]]) * 13:47 AntiComposite: CVNBot9 load ms.wikiquote q:ms: ([[phab:T404700|T404700]]) * 13:45 AntiComposite: CVNBot8 load min.wikisource s:min: ([[phab:T408343|T408343]]) * 13:44 AntiComposite: CVNBot7 load pcm.wikiquote q:pcm: ([[phab:T408351|T408351]]) * 13:43 AntiComposite: CVNBot6 load tl.wikisource s:tl: ([[phab:T388654|T388654]]) * 13:42 AntiComposite: CVNBot10 load bew.wiktionary wikt:bew: ([[phab:T402134|T402134]]) * 13:41 AntiComposite: CVNBot9 load zgh.wiktionary wikt:zgh: ([[phab:T399785|T399785]]) * 13:40 AntiComposite: CVNBot8 load min.wikibooks b:min: ([[phab:T395499|T395499]]) * 13:38 AntiComposite: CVNBot7 load rki.wikipedia rki: ([[phab:T392499|T392499]]) * 13:37 AntiComposite: CVNBot6 load mad.wikisource s:mad: ([[phab:T391767|T391767]]) === 2025-10-28 === * 23:16 AntiComposite: /cs flags #cvn-commons revi local_op === 2025-08-20 === * 20:35 AntiComposite: CVNBot10 load nup.wikipedia nup: ([[phab:T390711|T390711]]) === 2025-07-11 === * 14:38 AntiComposite: cvn-app10 restart all bots * 11:10 AntiComposite: cvn-app12 restart all bots * 11:09 AntiComposite: cvn-app10 restart all bots === 2025-06-20 === * 20:49 AntiComposite: cvn-app12: restart all bots * 20:48 AntiComposite: cvn-app10: restart all bots === 2025-05-26 === * 17:59 Krinkle: Create cvn-app14 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:59 Krinkle: Create cvn-app13 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:57 Krinkle: Delete cvn-apache10 instance (replaced/shutdown 2 days ago), ref [[phab:T395164|T395164]] === 2025-05-23 === * 20:30 Krinkle: Shut off cvn-apache10, [[phab:T395164|T395164]] * 20:29 Krinkle: Change cvn.wmcloud.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 20:22 Krinkle: Change cvn.wmflabs.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 19:45 Krinkle: Create cvn-apache11 (debian-12.0-bookworm, g4.cores2.ram4.disk20), [[phab:T395164|T395164]]) === 2025-05-16 === * 18:22 Krinkle: Replace outreach.wikipedia with outreach.wikimedia in cvn-sw/CVNBot19 per https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/820245 since the source channel was renamed * 17:30 Krinkle: krinkle@cvn-apache10:/srv/cvn/git/infrastructure$ git pull -- Deploy https://gerrit.wikimedia.org/r/1146724 * 17:30 Krinkle: krinkle@cvn-apache10 Update git remote in /srv/cvn/git/infrastructure from github.com/countervandalism to https://gerrit.wikimedia.org/r/labs/countervandalism/cvn-infrastructure === 2025-04-21 === * 17:22 AntiComposite: Hard reboot cvn-app10, flapping and not responsive to ssh === 2025-03-30 === * 06:55 Krinkle: krinkle@cvn-apache10: Run `sudo chmod 644 /srv/cvn/git/infrastructure/crontab-config/*.cron`, per [[phab:T390415|T390415]] === 2025-03-12 === * 02:18 AntiComposite: CVNBot9 load id.wikivoyage voy:id: ([[phab:T381080|T381080]]) * 02:15 AntiComposite: CVNBot8 load tig.wikipedia tig: ([[phab:T381379|T381379]]) * 02:14 AntiComposite: CVNBot7 load knc.wikipedia knc: ([[phab:T385185|T385185]]) * 02:11 AntiComposite: CVNBot6 load syl.wikipedia syl: ([[phab:T386464|T386464]]) * 02:08 AntiComposite: CVNBot10 load sat.wiktionary wikt:sat: ([[phab:T386631|T386631]]) === 2025-02-03 === * 22:05 AntiComposite: Hard reboot cvn-apache10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ * 21:58 AntiComposite: Hard reboot cvn-app10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ === 2025-01-02 === * 12:46 Krinkle: /cs flags #cvn-wp-en Lordseriouspig voiced * 12:45 Krinkle: /cs flags #cvn-sw Lordseriouspig voiced === 2024-11-23 === * 00:41 AntiComposite: CVNBot9 load ka.wikisource s:ka: ([[phab:T363243|T363243]]) * 00:38 AntiComposite: CVNBot8 load tcy.wikisource s:tcy: ([[phab:T378471|T378471]]) * 00:37 AntiComposite: CVNBot7 load tcy.wiktionary wikt:tcy: ([[phab:T378463|T378463]]) * 00:25 AntiComposite: Upgrade CVNBot29 to v4.0.4 * 00:25 AntiComposite: Upgrade CVNBot28 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot27 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot26 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot25 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot24 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot23 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot22 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot19 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot17 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot16 to v4.0.4 * 00:20 AntiComposite: Upgrade CVNBot10 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot9 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot8 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot7 to v4.0.4 * 00:17 AntiComposite: Upgrade CVNBot6 to v4.0.4 === 2024-11-22 === * 23:52 AntiComposite: Upgrade CVNBot21 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot20 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot18 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot15 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot14 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot13 to v4.0.4 * 23:49 AntiComposite: Upgrade CVNBot12 to v4.0.4 * 23:48 AntiComposite: Upgrade CVNBot11 to v4.0.4 * 23:47 AntiComposite: Upgrade CVNBot5 to v4.0.4 * 23:45 AntiComposite: Upgrade CVNBot3 to v4.0.4 * 23:44 AntiComposite: Upgrade CVNBot2 to v4.0.4 * 23:41 AntiComposite: Upgrade CVNBot1 to v4.0.4 * 23:32 AntiComposite: Upgrade CVNBot4 to v4.0.4 * 17:08 AntiComposite: restart CVNBots on cvn-app12 due to simultaneous RCReader failure 91950.519949 seconds === 2024-11-08 === * 23:24 AntiComposite: Restarting all CVNBots due to simultaneous RCReader disconnect 54323.128318 seconds ago === 2024-10-29 === * 20:56 AntiComposite: add sh.wikipedia to CVNBot6 as #cvn-wp-sh didn't survive the libera migration * 14:22 AntiComposite: restart all CVNBots === 2024-10-28 === * 12:50 AntiComposite: restarting all CVNBots, not coming up cleanly === 2024-10-25 === * 02:23 AntiComposite: add cs.wikivoyage to CVNBot10 ([[phab:T370913|T370913]]) * 02:21 AntiComposite: add bdr.wikipedia to CVNBot9 ([[phab:T371760|T371760]]) * 02:18 AntiComposite: add mos.wikipedia to CVNBot8 ([[phab:T374644|T374644]]) * 02:14 AntiComposite: add kge.wikipedia to CVNBot7 ([[phab:T374815|T374815]]) * 02:11 AntiComposite: add rsk.wikipedia to CVNBot6 ([[phab:T375017|T375017]]) * 02:07 AntiComposite: add mad.wiktionary to CVNBot9 ([[phab:T375024|T375024]]) * 02:06 AntiComposite: add gor.wikiquote to CVNBot8 ([[phab:T375095|T375095]]) * 02:04 AntiComposite: add nr.wikipedia to CVNBot7 ([[phab:T375102|T375102]]) * 02:01 AntiComposite: add tdd.wikipedia to CVNBot6 ([[phab:T375424|T375424]]) * 01:54 AntiComposite: add shn.wikinews to CVNBot9 ([[phab:T375433|T375433]]) * 01:52 AntiComposite: add iba.wikipedia to CVNBot8 ([[phab:T376572|T376572]]) * 01:50 AntiComposite: add bcl.wikisource to CVNBot7 ([[phab:T377088|T377088]]) * 01:47 AntiComposite: add ann.wikipedia to CVNBot6 ([[phab:T377160|T377160]]) * 01:43 AntiComposite: add igl.wikipedia to CVNBot9 ( [[phab:T363263|T363263]] ) * 01:41 AntiComposite: add my.wikisource to CVNBot8 ([[phab:T363270|T363270]]) * 01:39 AntiComposite: add foundation.wikimedia to CVNBot19 * 01:38 AntiComposite: add wikitech.wikimedia to CVNBot19 === 2024-10-24 === * 11:36 AntiComposite: restart all CVNBots === 2024-10-23 === * 17:33 AntiComposite: restart all CVNBots === 2024-07-03 === * 02:00 AntiComposite: add kus.wikipedia to CVNBot7 ([[phab:T360303|T360303]]) * 01:57 AntiComposite: add bew.wikipedia to CVNBot6 ([[phab:T360310|T360310]]) * 01:54 AntiComposite: add ms.wikisource to CVNBot9 ([[phab:T363250|T363250]]) * 01:53 AntiComposite: add kaa.wiktionary to CVNBot8 ([[phab:T363256|T363256]]) * 01:50 AntiComposite: add dtp.wikipedia to CVNBot7 ([[phab:T365230|T365230]]) * 01:48 AntiComposite: add btm.wikipedia to CVNBot6 ([[phab:T368067|T368067]]) * 01:45 AntiComposite: add fon.wikipedia to CVNBot9 ([[phab:T347939|T347939]]) * 01:43 AntiComposite: add blk.wikisource to CVNBot8 ([[phab:T343542|T343542]]) * 01:41 AntiComposite: su.wikisource to CVNBot7 ([[phab:T343548|T343548]]) * 01:39 AntiComposite: add tly.wikipedia to CVNBot6 ([[phab:T345170|T345170]]) * 01:37 AntiComposite: add dga.wikipedia to CVNBot9 ([[phab:T350229|T350229]]) * 01:35 AntiComposite: add bjn.wikiquote to CVNBot8 ([[phab:T350235|T350235]]) * 01:32 AntiComposite: add zgh.wikipedia to CVNBot7 ([[phab:T350241|T350241]]) * 01:28 AntiComposite: add bbc.wikipedia to CVNBot6 ([[phab:T350373|T350373]]) === 2024-06-24 === * 16:40 Krinkle: cvn-clerkbot parts #cvn-unifications (not operated by CVN, renamed to #wikimedia-unifications) === 2024-06-18 === * 08:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs === 2024-03-22 === * 05:30 Operator873: /cs flags #cvn-simplewikis Drummingman +voice === 2024-02-28 === * 21:34 Krinkle: /cs flags #cvn-wp-da Sarrus local_op === 2024-01-11 === * 12:19 AntiComposite: /cs flags #cvn-meta Bsadowski1 local_op === 2023-12-01 === * 15:30 AntiComposite: restart everything after WMCS network outage === 2023-10-07 === * 14:50 AntiComposite: kill 2 CVNBot11 processes and restart, bot not joined to IRC === 2023-09-22 === * 00:06 Op873: /cs flags #cvn-wp-en Oshwah +AV === 2023-09-16 === * 10:33 JackSparrow: /cs flags #cvn-wp-fa Arian_Ar local_op === 2023-09-07 === * 01:35 AntiComposite: restart all cvn-app12 bots * 01:33 AntiComposite: restart all cvn-app10 bots === 2023-08-15 === * 14:44 AntiComposite: reboot cvn-app10 from Horizon, bots dead and not responding to SSH === 2023-08-09 === * 00:07 AntiComposite: add 9 wikis to #cvn-sw (ref [[phab:T332379|T332379]] [[phab:T336115|T336115]] [[phab:T332093|T332093]] [[phab:T332093|T332093]] [[phab:T335987|T335987]] [[phab:T334459|T334459]] [[phab:T333271|T333271]] [[phab:T334740|T334740]] [[phab:T342865|T342865]]) === 2023-08-08 === * 23:46 AntiComposite: drop wo.wikiquote from CVNBot10 (closed) [[phab:T334482|T334482]] === 2023-07-27 === * 18:15 AntiComposite: Kill and restart CVNBot29 on cvn-app12 === 2023-07-06 === * 16:21 AntiComposite: point git repos to gerrit on cvn-app10 * 16:19 AntiComposite: point git repos to gerrit on cvn-app12 * 16:03 AntiComposite: CVNBot v4.0.3 deployed to all bots ([[phab:T327126|T327126]], [[phab:T327127|T327127]]) * 16:01 AntiComposite: Upgrade CVNBot29 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot28 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot27 to v4.0.3 * 15:59 AntiComposite: Upgrade CVNBot26 to v4.0.3 * 15:58 AntiComposite: Upgrade CVNBot25 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot24 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot23 to v4.0.3 * 15:55 AntiComposite: Upgrade CVNBot22 to v4.0.3 * 15:54 AntiComposite: Upgrade CVNBot19 to v4.0.3 * 15:53 AntiComposite: Upgrade CVNBot17 to v4.0.3 * 15:46 AntiComposite: Upgrade CVNBot16 to v4.0.3 * 15:44 AntiComposite: Upgrade CVNBot10 to v4.0.3 * 15:41 AntiComposite: Upgrade CVNBot9 to v4.0.3 * 15:40 AntiComposite: Upgrade CVNBot8 to v4.0.3 * 15:39 AntiComposite: Upgrade CVNBot7 to v4.0.3 * 15:38 AntiComposite: Upgrade CVNBot6 to v4.0.3 * 04:37 AntiComposite: Upgrade CVNBot21 to v4.0.3 * 04:34 AntiComposite: Upgrade CVNBot20 to v4.0.3 * 04:33 AntiComposite: Upgrade CVNBot18 to v4.0.3 * 04:30 AntiComposite: Upgrade CVNBot15 to v4.0.3 * 04:23 AntiComposite: Upgrade CVNBot14 to v4.0.3 * 04:22 AntiComposite: Upgrade CVNBot13 to v4.0.3 * 04:14 AntiComposite: Upgrade CVNBot12 to v4.0.3 * 04:09 AntiComposite: Upgrade CVNBot11 to v4.0.3 * 04:03 AntiComposite: Upgrade CVNBot5 to v4.0.3 * 04:01 AntiComposite: Upgrade CVNBot4 to v4.0.3 * 04:00 AntiComposite: Upgrade CVNBot3 to v4.0.3 * 03:57 AntiComposite: Upgrade CVNBot2 to v4.0.3 * 03:51 AntiComposite: Upgrade CVNBot1 to v4.0.3 === 2023-06-28 === * 02:34 Operator873: /cs flags #cvn-sw Fehufanga voiced === 2023-06-16 === * 22:05 AntiComposite: manually restart cvn-clerkbot === 2023-05-15 === * 14:58 hauskater: Dropped akwiki and nawiki from CVNBot10 as closed wikis. On-wiki lists require an update. === 2023-04-26 === * 20:07 AntiComposite: /cs flags #cvn-mk-scan M4r51n voiced === 2023-04-21 === * 22:12 Operator873: granted voice to Fehufanga in #cvn-simplewikis === 2023-04-14 === * 18:28 AntiComposite: restart cvn-app10 from horizon, bots quit and ssh times out === 2023-03-22 === * 03:33 Operator873: Voiced Tulsi in #cvn-sw -meta -mediawiki -commons -simplewikis === 2023-03-13 === * 19:46 Operator873: CVNBot18 restarted === 2023-03-03 === * 14:45 AntiComposite: /cs flags #cvn-sw-spam COIBot bot === 2023-02-27 === * 22:33 herzog: Loaded gur.wikipedia to SWMT Group 4 (CVNBot9) - [[phab:T327842|T327842]] * 18:04 herzog: Loaded guc.wikipedia to CVNBot9 / Group 4 - [[phab:T326236|T326236]] === 2023-02-02 === * 00:21 ma: Added 12 new wikis to CVNBot<nowiki>{</nowiki>6,7,8<nowiki>}</nowiki>, 4 to each one. Refs.: [[phab:T321283|T321283]] [[phab:T321289|T321289]] [[phab:T321295|T321295]] [[phab:T326139|T326139]] [[phab:T305281|T305281]] [[phab:T310873|T310873]] [[phab:T312215|T312215]] [[phab:T314640|T314640]] [[phab:T314646|T314646]] [[phab:T316457|T316457]] [[phab:T317113|T317113]] [[phab:T319191|T319191]] === 2023-01-30 === * 22:50 Krinkle: Delete cvn-app8 and cvn-app9 instances, ref [[phab:T306066|T306066]] === 2023-01-28 === * 02:51 AntiComposite: /cs flags #cvn-sw Ajraddatz local_op === 2023-01-24 === * 08:54 Krinkle: Delete cvn-apache9, [[phab:T306066|T306066]] * 08:54 Krinkle: Suspend cvn-app8 and cvn-app9 (`pgrep -af cvn` is empty on both), [[phab:T306066|T306066]] === 2023-01-23 === * 16:53 AntiComposite: Deploy {{Gerrit|716e140}} to app12 ([[phab:T306066|T306066]]) * 16:50 AntiComposite: Deploy {{Gerrit|716e140}} to app9 ([[phab:T306066|T306066]]) * 16:29 AntiComposite: Deploy {{Gerrit|442f324}} to app12 ([[phab:T306066|T306066]]) * 16:25 AntiComposite: Deploy {{Gerrit|442f324}} to app9 ([[phab:T306066|T306066]]) * 16:01 AntiComposite: Deploy {{Gerrit|9024b8f}} to app12 ([[phab:T306066|T306066]]) * 15:59 AntiComposite: Deploy {{Gerrit|9024b8f}} to app9 ([[phab:T306066|T306066]]) === 2023-01-22 === * 21:40 AntiComposite: start cvndb-CVNBot14-publish on app10 * 21:07 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app10, starting bots ([[phab:T306066|T306066]]) * 20:56 AntiComposite: disable cvndb-CVNBot14-publish on app8 * 20:51 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app8, stopping bots ([[phab:T306066|T306066]]) * 19:53 AntiComposite: Deploy {{Gerrit|80ea1f5}} to cvn-app10 ([[phab:T306066|T306066]]) * 15:43 AntiComposite: restart all CVNBots on app9 * 15:42 AntiComposite: restart all CVNBots on app8 === 2023-01-17 === * 00:15 Krinkle: Suspend cvn-apache9, replaced by cvn-apache10, ref [[phab:T306066|T306066]] * 00:14 Krinkle: Switch cvn.wmflabs.org from cvn-apache9 to cvn-apache10 === 2023-01-16 === * 00:10 Krinkle: Move https://github.com/countervandalism/cvn-clerkbot to https://github.com/wikimedia/countervandalism-cvn-clerkbot (with HTTP and Git redirect preserved), and replace with Gerrit mirror === 2023-01-15 === * 23:12 Krinkle: Create 'labs-cvn' permission group in Gerrit with CVN staff members * 23:12 Krinkle: Move https://github.com/countervandalism/cvn-api to https://github.com/wikimedia/countervandalism-cvn-api (with HTTP and Git redirect preserved), and replace with Gerrit mirror * 22:02 Krinkle: Switch new cvn.wmcloud.org proxy from cvn-apache9 to cvn-apache10 (Leave main cvn.wmflabs.org as-is for now). === 2023-01-14 === * 21:45 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|4cee27a}}) * 21:22 AntiComposite: move cvn-clerbot back to cvn-app9 (deploy {{Gerrit|371ba2a}}) * 21:10 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|3f3f40f}}) === 2023-01-10 === * 23:22 Krinkle: krinkle@cvn-apache9$ update infrastructure.git, sudo apachectl graceful * 23:20 Krinkle: Create cvn.wmcloud.org web proxy (in addition to cvn.wmflabs.org) === 2023-01-07 === * 20:53 AntiComposite: apply role::labs::lvm::srv only to cvn-apache9, cvn-app8, and cvn-app9 to fix puppet failures on new instances === 2023-01-04 === * 20:47 Krinkle: Allocate new floating IPs to cvn-app10 and cvn-app11 * 20:46 Krinkle: Create new cvn-apache10, cvn-app10, cvn-app11 with Debian 11 Bullseye to replace the old Debian 9.1 Stretch instances * 20:04 taavi: bump floating ip quota from 2 to 4, [[phab:T326269|T326269]] === 2022-12-27 === * 20:11 Frosty873: /cs flags #cvn-meta xaosflux voiced * 20:11 Frosty873: /cs flags #cvn-wp-en xaosflux voiced === 2022-12-23 === * 03:25 AntiComposite: /cs flags #cvn-meta tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-mediawiki tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-sw tryvix1509 voiced === 2022-10-18 === * 23:13 Joan: CVNBot3 restarted (Last message was received on RCReader 62854.814658 seconds ag) === 2022-09-04 === * 22:21 Operator873: /cs flags #cvn-simplewikis Enfcer +AV * 02:20 Operator873: /cs flags #cvn-sw Bot873 +voiced === 2022-08-26 === * 14:09 hauskatze: Loaded pcm.wikipedia and guw.wiktionary to CVNBot8 & 9 respectively {{!}} [[phab:T310880|T310880]] [[phab:T309057|T309057]] === 2022-07-09 === * 16:42 AntiComposite: /cs flags #cvn-commons pandakekok9 voiced === 2022-07-08 === * 21:53 Krinkle: krinkle@horizon.wikimedia.org Add anticomposite as project member and project admin to cloudvps.cvn === 2022-07-01 === * 21:39 Krinkle: cvn-app8: kill CVNBot14.exe and two (!) procs for CVNBot18.exe === 2022-06-25 === * 03:25 AntiComposite: /cs flags #cvn-wp-en PhantomTech voiced === 2022-06-22 === * 21:04 op873: <+CVNBot3> Added: LuchoCR is on es.wikipedia bot list, added by Operator873{{!}}CVN until the end of time ("Mass blockiing P2P-proxies with script") * 20:34 op873: restart CVNBot3 (possibly caused by block flood) * 19:31 op873: restart CVNBot3 === 2022-06-15 === * 18:49 AntiComposite: /cs flags #cvn-wp-en Zppix voiced * 18:48 AntiComposite: /cs flags #cvn-simplewikis Zppix voiced === 2022-05-23 === * 00:24 Joan: Flags +AV were set on Sargento in cvn-wp-es * 00:23 Joan: Flags +AV were set on alhen in cvn-wp-es === 2022-05-19 === * 23:10 Joan: CVNBot3 restarted (Last message was received on RCReader 92593.747667 seconds ago) === 2022-05-11 === * 07:34 Operator873: /cs flags #cvn-wp-en Tamzin voiced === 2022-05-07 === * 17:40 Operator873: /cs flags #cvn-sw koi voiced * 17:39 Operator873: /cs flags #cvn-zh-scan koi voiced === 2022-04-28 === * 03:19 Joan: CVNBot3 restarted (Last message was received on RCReader 75273.332577 seconds ago) === 2022-04-22 === * 15:08 AntiComposite: /cs flags #cvn-meta Bsadowski1 voiced === 2022-04-18 === * 20:44 AntiComposite: /cs flags #cvn-sw Vermont voiced === 2022-04-13 === * 22:40 Operator873: /cs flags #cvn-meta Joan voiced * 22:40 Operator873: /cs flags #cvn-sw Joan voiced * 22:14 Joan: CVNBot3 restarted (Last message was received on RCReader 54942.175428 seconds ago) === 2022-04-07 === * 23:15 Operator873: /cs flags #cvn-wp-hr NovakWatchmen local_op * 23:13 Operator873: voiced Superpes (Superpes15) in #cvn-sw #cvn-sw-spam and #cvn-it-scan === 2022-04-04 === * 17:34 Operator873: Voiced Vermont in #cvn-meta and #cvn-simplewikis /cs flags #cvn-meta Vermont voiced === 2022-03-30 === * 14:33 Joan: CVNBot3 restarted (Last message was received on RCReader 26318.335196 seconds ago) === 2022-03-28 === * 02:38 AntiComposite: /cs flags #cvn-wp-en Bsoyka voiced === 2022-03-21 === * 20:22 Operator873: /cs flags #cvn-simplewikis Bsadowski1 +AfiotvV * 20:17 Operator873: Operator873{{!}}CVN (Operator873) set flags +AVfitv on Bsadowski1 * 20:03 Operator873: Operator873{{!}}CVN (Operator873) set flags +V on Bsadowski1 * 17:04 AntiComposite: /cs flags #cvn-sw Bsadowski1 local_op === 2022-03-15 === * 15:38 Joan: CVNBot3 restarted (Last message was received on RCReader 26424.279343 seconds ago) === 2022-03-14 === * 14:02 Joan: CVNBot3 restarted (Last message was received on RCReader 17096.72183 seconds ago) === 2022-03-12 === * 16:27 Joan: CVNBot3 restarted (Last message was received on RCReader 27236.775673 seconds ago) === 2022-03-11 === * 14:24 Joan: CVNBot3 restarted (Last message was received on RCReader 18853.006849 seconds ago) === 2022-03-10 === * 14:08 Joan: CVNBot3 restarted (Last message was received on RCReader 22518.614282 seconds ago) === 2022-03-08 === * 20:27 AntiComposite: /cs flags #cvn-wp-en Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-simplewikis Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-commons Sarrus voiced === 2022-03-07 === * 16:30 AntiComposite: /cs flags #cvn-meta zabe voiced * 16:25 AntiComposite: /cs flags #cvn-simplewikis DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-meta DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-sw TheresNoTime voiced * 16:07 Krinkle: /cs flags #cvn-staff Operator873 staff * 16:07 Krinkle: /cs flags #cvn-staff AntiComposite staff === 2022-03-05 === * 04:13 Joan: CVNBot3 restarted (Last message was received on RCReader 31573.894101 seconds ago) === 2022-03-03 === * 16:39 Joan: CVNBot3 restarted (Last message was received on RCReader 36578.236383 seconds ago) === 2022-03-01 === * 13:21 Joan: CVNBot3 restarted (Last message was received on RCReader 20646.781861 seconds ago) === 2022-02-15 === * 14:12 Joan: CVNBot3 restarted (Last message was received on RCReader 25001.391103 seconds ago) === 2022-02-13 === * 18:47 andrewbogott: switching to project-local nfs server cvn-nfs-1 * 17:54 andrewbogott: switching to project-local nfs server puppet-diffs-nfs-1 === 2022-02-10 === * 16:17 Joan: CVNBot3 restarted (Last message was received on RCReader 39817.871151 seconds ago) === 2022-02-08 === * 15:51 Joan: CVNBot3 restarted (Last message was received on RCReader 28868.916144 seconds ago) === 2022-02-04 === * 23:59 andrewbogott: accidentally restarted all VMs due to misreading the project purge page. sorry! === 2022-02-02 === * CVN: Several bots restarted after netsplit took nickserv and some bots with it. * 10:26 Krinkle: CVNBot1 bes del delete(?!d) — originally added by huh (reason: "widewuto") === 2022-02-01 === * 15:20 Joan: CVNBot3 restarted (Last message was received on RCReader 26990.323435 seconds ago) === 2022-01-31 === * 17:37 Joan: CVNBot3 restarted (Last message was received on RCReader 48827.882566 seconds ago) === 2022-01-27 === * 16:58 Joan: CVNBot3 restarted (Last message was received on RCReader 29206.852828 seconds ago) === 2022-01-21 === * 16:07 Joan: CVNBot3 restarted (Last message was received on RCReader 22091.557102 seconds ago) === 2022-01-20 === * 18:13 Cam11598: CVNBot15 restarted === 2022-01-19 === * 17:26 Joan: Restarted CVNBot3 (Last message was received on RCReader 28129.031916 seconds ago) === 2022-01-18 === * 16:55 Joan: Restarted CVNBot3 (Last message was received on RCReader 26283.381782 seconds ago) === 2022-01-17 === * 16:33 Joan: Restarted CVNBot3 (#cvn-wp-es) (Last message was received on RCReader 197065.877109 seconds ago) === 2022-01-15 === * 04:56 Cam11598: restarted CVNBOT18 8:55:47 PM <�25B100+ CVNBot18> Last message was received on RCReader 29723.456263 seconds ago === 2022-01-13 === * 01:29 Cam11598: restarted CVNBot2 nickserv issue * 01:29 Cam11598: restarted CVNBot18 - no response from RC feed === 2022-01-09 === * 18:18 Joan: Flags +AV were set on Hasley in cvn-wp-es (sysop at es.wikipedia) * 17:56 Krinkle: /cs flags #cvn-wp-es Joan local_op === 2022-01-07 === * 22:08 hauskatze: CVNBot9 load co.wiktionary wikt:co: * 22:04 hauskatze: CVNBot9 load ban.wikisource s:ban: * 22:04 hauskatze: CVNBot9 load ba.wikibooks b:ba: * 10:51 hauskatze: Loaded alt.wikipedia to Group 4 (CVNBot9) - small wiki not monitored === 2022-01-06 === * 19:42 hauskatze: Loaded ami.wikipedia to CVNBot8 - [[phab:T292421|T292421]] * 19:41 hauskatze: Loaded pwn.wikipedia to CVNBot7 - [[phab:T292419|T292419]] * 19:39 hauskatze: Loaded lmo.wiktionary to CVNBot6 - [[phab:T292076|T292076]] * 19:34 hauskatze: Loaded jv.wikisource to CVNBot6 refs. [[phab:T287319|T287319]] * 19:29 Krinkle: cs flags #cvn-sw hauskatze local_op * 13:57 Krinkle: Krinkle added $a:Cam11598 to the #cvn-staff I list (+I) {{SAL|Project Name=cvn}} <noinclude> ==Archives== * [[Nova Resource:Cvn/SAL/Archive 1|Archive 1]] (2006-2009) * [[Nova Resource:Cvn/SAL/Archive 2|Archive 2]] (2010-2011) * [[Nova Resource:Cvn/SAL/Archive 3|Archive 3]] (2012-2013) * [[Nova Resource:Cvn/SAL/Archive 4|Archive 4]] (2013-2021) (some parts in 2013 are not indexed) [[Category:SAL]]</noinclude> c2obepukkw90j5p67zhhj8xp9hrn67j 2426640 2426639 2026-06-13T23:17:16Z Stashbot 7414 AntiComposite: CVNBot29 drop & purge es.wikinews (T428622) 2426640 wikitext text/x-wiki === 2026-06-13 === * 23:17 AntiComposite: CVNBot29 drop & purge es.wikinews ([[phab:T428622|T428622]]) * 23:13 AntiComposite: CVNBot26 drop & purge ar.wikinews ([[phab:T428622|T428622]]) * 23:12 AntiComposite: CVNBot25 drop & purge ko.wikinews ([[phab:T428622|T428622]]) * 23:12 AntiComposite: CVNBot23 drop & purge zh.wikinews ([[phab:T428622|T428622]]) * 23:11 AntiComposite: CVNBot10 drop & purge ca.wikinews, ko.wikinews, no.wikinews ([[phab:T428622|T428622]]) * 23:07 AntiComposite: CVNBot9 drop & purge bs.wikinews, el.wikinews, fa.wikinews, shn.wikinews, zh.wikinews ([[phab:T428622|T428622]]) * 23:03 AntiComposite: CVNBot8 drop & purge ar.wikinews, cs.wikinews, de.wikinews, fi.wikinews, he.wikinews, ru.wikinews, sq.wikinews, sr.wikinews, uk.wikinews ([[phab:T428622|T428622]]) * 22:58 AntiComposite: CVNBot7 drop & purge es.wikinews, guw.wikinews, pt.wikinews ([[phab:T428622|T428622]]) * 22:56 AntiComposite: CVNBot6 drop & purge eo.wikinews, fr.wikinews, pl.wikinews, ro.wikinews, sv.wikinews, ta.wikinews ([[phab:T428622|T428622]]) * 22:49 AntiComposite: CVNBot4 drop it.wikinews ([[phab:T428622|T428622]]) === 2026-06-02 === * 01:03 Krinkle: /cs flags #cvn-sw Divinations voiced === 2026-05-26 === * 18:07 AntiComposite: restart all bots -- disconnected === 2026-05-03 === * 13:39 Krinkle: Disable "Admin immed notify" for cvn-private https://lists.wikimedia.org/postorius/lists/cvn-private.lists.wikimedia.org/settings/automatic_responses. We previously removed the sub form but this is no longer supported in mailman3. We require confirm/moderate for new subs, there is no way to turn it off. But we can at least disable the noise. === 2026-04-27 === * 12:22 Krinkle: /cs flags #cvn-meta NathanVeritas voiced === 2026-04-01 === * 13:34 AntiComposite: restart all bots === 2026-02-04 === * 20:33 AntiComposite: Restart all bots === 2025-12-26 === * 15:54 Operator873: /cs flags #cvn-zh-scan nya_1F616EMO voiced === 2025-11-27 === * 13:48 AntiComposite: CVNBot10 load tok.wikipedia tok: ([[phab:T404567|T404567]]) * 13:47 AntiComposite: CVNBot9 load ms.wikiquote q:ms: ([[phab:T404700|T404700]]) * 13:45 AntiComposite: CVNBot8 load min.wikisource s:min: ([[phab:T408343|T408343]]) * 13:44 AntiComposite: CVNBot7 load pcm.wikiquote q:pcm: ([[phab:T408351|T408351]]) * 13:43 AntiComposite: CVNBot6 load tl.wikisource s:tl: ([[phab:T388654|T388654]]) * 13:42 AntiComposite: CVNBot10 load bew.wiktionary wikt:bew: ([[phab:T402134|T402134]]) * 13:41 AntiComposite: CVNBot9 load zgh.wiktionary wikt:zgh: ([[phab:T399785|T399785]]) * 13:40 AntiComposite: CVNBot8 load min.wikibooks b:min: ([[phab:T395499|T395499]]) * 13:38 AntiComposite: CVNBot7 load rki.wikipedia rki: ([[phab:T392499|T392499]]) * 13:37 AntiComposite: CVNBot6 load mad.wikisource s:mad: ([[phab:T391767|T391767]]) === 2025-10-28 === * 23:16 AntiComposite: /cs flags #cvn-commons revi local_op === 2025-08-20 === * 20:35 AntiComposite: CVNBot10 load nup.wikipedia nup: ([[phab:T390711|T390711]]) === 2025-07-11 === * 14:38 AntiComposite: cvn-app10 restart all bots * 11:10 AntiComposite: cvn-app12 restart all bots * 11:09 AntiComposite: cvn-app10 restart all bots === 2025-06-20 === * 20:49 AntiComposite: cvn-app12: restart all bots * 20:48 AntiComposite: cvn-app10: restart all bots === 2025-05-26 === * 17:59 Krinkle: Create cvn-app14 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:59 Krinkle: Create cvn-app13 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:57 Krinkle: Delete cvn-apache10 instance (replaced/shutdown 2 days ago), ref [[phab:T395164|T395164]] === 2025-05-23 === * 20:30 Krinkle: Shut off cvn-apache10, [[phab:T395164|T395164]] * 20:29 Krinkle: Change cvn.wmcloud.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 20:22 Krinkle: Change cvn.wmflabs.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 19:45 Krinkle: Create cvn-apache11 (debian-12.0-bookworm, g4.cores2.ram4.disk20), [[phab:T395164|T395164]]) === 2025-05-16 === * 18:22 Krinkle: Replace outreach.wikipedia with outreach.wikimedia in cvn-sw/CVNBot19 per https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/820245 since the source channel was renamed * 17:30 Krinkle: krinkle@cvn-apache10:/srv/cvn/git/infrastructure$ git pull -- Deploy https://gerrit.wikimedia.org/r/1146724 * 17:30 Krinkle: krinkle@cvn-apache10 Update git remote in /srv/cvn/git/infrastructure from github.com/countervandalism to https://gerrit.wikimedia.org/r/labs/countervandalism/cvn-infrastructure === 2025-04-21 === * 17:22 AntiComposite: Hard reboot cvn-app10, flapping and not responsive to ssh === 2025-03-30 === * 06:55 Krinkle: krinkle@cvn-apache10: Run `sudo chmod 644 /srv/cvn/git/infrastructure/crontab-config/*.cron`, per [[phab:T390415|T390415]] === 2025-03-12 === * 02:18 AntiComposite: CVNBot9 load id.wikivoyage voy:id: ([[phab:T381080|T381080]]) * 02:15 AntiComposite: CVNBot8 load tig.wikipedia tig: ([[phab:T381379|T381379]]) * 02:14 AntiComposite: CVNBot7 load knc.wikipedia knc: ([[phab:T385185|T385185]]) * 02:11 AntiComposite: CVNBot6 load syl.wikipedia syl: ([[phab:T386464|T386464]]) * 02:08 AntiComposite: CVNBot10 load sat.wiktionary wikt:sat: ([[phab:T386631|T386631]]) === 2025-02-03 === * 22:05 AntiComposite: Hard reboot cvn-apache10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ * 21:58 AntiComposite: Hard reboot cvn-app10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ === 2025-01-02 === * 12:46 Krinkle: /cs flags #cvn-wp-en Lordseriouspig voiced * 12:45 Krinkle: /cs flags #cvn-sw Lordseriouspig voiced === 2024-11-23 === * 00:41 AntiComposite: CVNBot9 load ka.wikisource s:ka: ([[phab:T363243|T363243]]) * 00:38 AntiComposite: CVNBot8 load tcy.wikisource s:tcy: ([[phab:T378471|T378471]]) * 00:37 AntiComposite: CVNBot7 load tcy.wiktionary wikt:tcy: ([[phab:T378463|T378463]]) * 00:25 AntiComposite: Upgrade CVNBot29 to v4.0.4 * 00:25 AntiComposite: Upgrade CVNBot28 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot27 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot26 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot25 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot24 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot23 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot22 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot19 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot17 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot16 to v4.0.4 * 00:20 AntiComposite: Upgrade CVNBot10 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot9 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot8 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot7 to v4.0.4 * 00:17 AntiComposite: Upgrade CVNBot6 to v4.0.4 === 2024-11-22 === * 23:52 AntiComposite: Upgrade CVNBot21 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot20 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot18 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot15 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot14 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot13 to v4.0.4 * 23:49 AntiComposite: Upgrade CVNBot12 to v4.0.4 * 23:48 AntiComposite: Upgrade CVNBot11 to v4.0.4 * 23:47 AntiComposite: Upgrade CVNBot5 to v4.0.4 * 23:45 AntiComposite: Upgrade CVNBot3 to v4.0.4 * 23:44 AntiComposite: Upgrade CVNBot2 to v4.0.4 * 23:41 AntiComposite: Upgrade CVNBot1 to v4.0.4 * 23:32 AntiComposite: Upgrade CVNBot4 to v4.0.4 * 17:08 AntiComposite: restart CVNBots on cvn-app12 due to simultaneous RCReader failure 91950.519949 seconds === 2024-11-08 === * 23:24 AntiComposite: Restarting all CVNBots due to simultaneous RCReader disconnect 54323.128318 seconds ago === 2024-10-29 === * 20:56 AntiComposite: add sh.wikipedia to CVNBot6 as #cvn-wp-sh didn't survive the libera migration * 14:22 AntiComposite: restart all CVNBots === 2024-10-28 === * 12:50 AntiComposite: restarting all CVNBots, not coming up cleanly === 2024-10-25 === * 02:23 AntiComposite: add cs.wikivoyage to CVNBot10 ([[phab:T370913|T370913]]) * 02:21 AntiComposite: add bdr.wikipedia to CVNBot9 ([[phab:T371760|T371760]]) * 02:18 AntiComposite: add mos.wikipedia to CVNBot8 ([[phab:T374644|T374644]]) * 02:14 AntiComposite: add kge.wikipedia to CVNBot7 ([[phab:T374815|T374815]]) * 02:11 AntiComposite: add rsk.wikipedia to CVNBot6 ([[phab:T375017|T375017]]) * 02:07 AntiComposite: add mad.wiktionary to CVNBot9 ([[phab:T375024|T375024]]) * 02:06 AntiComposite: add gor.wikiquote to CVNBot8 ([[phab:T375095|T375095]]) * 02:04 AntiComposite: add nr.wikipedia to CVNBot7 ([[phab:T375102|T375102]]) * 02:01 AntiComposite: add tdd.wikipedia to CVNBot6 ([[phab:T375424|T375424]]) * 01:54 AntiComposite: add shn.wikinews to CVNBot9 ([[phab:T375433|T375433]]) * 01:52 AntiComposite: add iba.wikipedia to CVNBot8 ([[phab:T376572|T376572]]) * 01:50 AntiComposite: add bcl.wikisource to CVNBot7 ([[phab:T377088|T377088]]) * 01:47 AntiComposite: add ann.wikipedia to CVNBot6 ([[phab:T377160|T377160]]) * 01:43 AntiComposite: add igl.wikipedia to CVNBot9 ( [[phab:T363263|T363263]] ) * 01:41 AntiComposite: add my.wikisource to CVNBot8 ([[phab:T363270|T363270]]) * 01:39 AntiComposite: add foundation.wikimedia to CVNBot19 * 01:38 AntiComposite: add wikitech.wikimedia to CVNBot19 === 2024-10-24 === * 11:36 AntiComposite: restart all CVNBots === 2024-10-23 === * 17:33 AntiComposite: restart all CVNBots === 2024-07-03 === * 02:00 AntiComposite: add kus.wikipedia to CVNBot7 ([[phab:T360303|T360303]]) * 01:57 AntiComposite: add bew.wikipedia to CVNBot6 ([[phab:T360310|T360310]]) * 01:54 AntiComposite: add ms.wikisource to CVNBot9 ([[phab:T363250|T363250]]) * 01:53 AntiComposite: add kaa.wiktionary to CVNBot8 ([[phab:T363256|T363256]]) * 01:50 AntiComposite: add dtp.wikipedia to CVNBot7 ([[phab:T365230|T365230]]) * 01:48 AntiComposite: add btm.wikipedia to CVNBot6 ([[phab:T368067|T368067]]) * 01:45 AntiComposite: add fon.wikipedia to CVNBot9 ([[phab:T347939|T347939]]) * 01:43 AntiComposite: add blk.wikisource to CVNBot8 ([[phab:T343542|T343542]]) * 01:41 AntiComposite: su.wikisource to CVNBot7 ([[phab:T343548|T343548]]) * 01:39 AntiComposite: add tly.wikipedia to CVNBot6 ([[phab:T345170|T345170]]) * 01:37 AntiComposite: add dga.wikipedia to CVNBot9 ([[phab:T350229|T350229]]) * 01:35 AntiComposite: add bjn.wikiquote to CVNBot8 ([[phab:T350235|T350235]]) * 01:32 AntiComposite: add zgh.wikipedia to CVNBot7 ([[phab:T350241|T350241]]) * 01:28 AntiComposite: add bbc.wikipedia to CVNBot6 ([[phab:T350373|T350373]]) === 2024-06-24 === * 16:40 Krinkle: cvn-clerkbot parts #cvn-unifications (not operated by CVN, renamed to #wikimedia-unifications) === 2024-06-18 === * 08:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs === 2024-03-22 === * 05:30 Operator873: /cs flags #cvn-simplewikis Drummingman +voice === 2024-02-28 === * 21:34 Krinkle: /cs flags #cvn-wp-da Sarrus local_op === 2024-01-11 === * 12:19 AntiComposite: /cs flags #cvn-meta Bsadowski1 local_op === 2023-12-01 === * 15:30 AntiComposite: restart everything after WMCS network outage === 2023-10-07 === * 14:50 AntiComposite: kill 2 CVNBot11 processes and restart, bot not joined to IRC === 2023-09-22 === * 00:06 Op873: /cs flags #cvn-wp-en Oshwah +AV === 2023-09-16 === * 10:33 JackSparrow: /cs flags #cvn-wp-fa Arian_Ar local_op === 2023-09-07 === * 01:35 AntiComposite: restart all cvn-app12 bots * 01:33 AntiComposite: restart all cvn-app10 bots === 2023-08-15 === * 14:44 AntiComposite: reboot cvn-app10 from Horizon, bots dead and not responding to SSH === 2023-08-09 === * 00:07 AntiComposite: add 9 wikis to #cvn-sw (ref [[phab:T332379|T332379]] [[phab:T336115|T336115]] [[phab:T332093|T332093]] [[phab:T332093|T332093]] [[phab:T335987|T335987]] [[phab:T334459|T334459]] [[phab:T333271|T333271]] [[phab:T334740|T334740]] [[phab:T342865|T342865]]) === 2023-08-08 === * 23:46 AntiComposite: drop wo.wikiquote from CVNBot10 (closed) [[phab:T334482|T334482]] === 2023-07-27 === * 18:15 AntiComposite: Kill and restart CVNBot29 on cvn-app12 === 2023-07-06 === * 16:21 AntiComposite: point git repos to gerrit on cvn-app10 * 16:19 AntiComposite: point git repos to gerrit on cvn-app12 * 16:03 AntiComposite: CVNBot v4.0.3 deployed to all bots ([[phab:T327126|T327126]], [[phab:T327127|T327127]]) * 16:01 AntiComposite: Upgrade CVNBot29 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot28 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot27 to v4.0.3 * 15:59 AntiComposite: Upgrade CVNBot26 to v4.0.3 * 15:58 AntiComposite: Upgrade CVNBot25 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot24 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot23 to v4.0.3 * 15:55 AntiComposite: Upgrade CVNBot22 to v4.0.3 * 15:54 AntiComposite: Upgrade CVNBot19 to v4.0.3 * 15:53 AntiComposite: Upgrade CVNBot17 to v4.0.3 * 15:46 AntiComposite: Upgrade CVNBot16 to v4.0.3 * 15:44 AntiComposite: Upgrade CVNBot10 to v4.0.3 * 15:41 AntiComposite: Upgrade CVNBot9 to v4.0.3 * 15:40 AntiComposite: Upgrade CVNBot8 to v4.0.3 * 15:39 AntiComposite: Upgrade CVNBot7 to v4.0.3 * 15:38 AntiComposite: Upgrade CVNBot6 to v4.0.3 * 04:37 AntiComposite: Upgrade CVNBot21 to v4.0.3 * 04:34 AntiComposite: Upgrade CVNBot20 to v4.0.3 * 04:33 AntiComposite: Upgrade CVNBot18 to v4.0.3 * 04:30 AntiComposite: Upgrade CVNBot15 to v4.0.3 * 04:23 AntiComposite: Upgrade CVNBot14 to v4.0.3 * 04:22 AntiComposite: Upgrade CVNBot13 to v4.0.3 * 04:14 AntiComposite: Upgrade CVNBot12 to v4.0.3 * 04:09 AntiComposite: Upgrade CVNBot11 to v4.0.3 * 04:03 AntiComposite: Upgrade CVNBot5 to v4.0.3 * 04:01 AntiComposite: Upgrade CVNBot4 to v4.0.3 * 04:00 AntiComposite: Upgrade CVNBot3 to v4.0.3 * 03:57 AntiComposite: Upgrade CVNBot2 to v4.0.3 * 03:51 AntiComposite: Upgrade CVNBot1 to v4.0.3 === 2023-06-28 === * 02:34 Operator873: /cs flags #cvn-sw Fehufanga voiced === 2023-06-16 === * 22:05 AntiComposite: manually restart cvn-clerkbot === 2023-05-15 === * 14:58 hauskater: Dropped akwiki and nawiki from CVNBot10 as closed wikis. On-wiki lists require an update. === 2023-04-26 === * 20:07 AntiComposite: /cs flags #cvn-mk-scan M4r51n voiced === 2023-04-21 === * 22:12 Operator873: granted voice to Fehufanga in #cvn-simplewikis === 2023-04-14 === * 18:28 AntiComposite: restart cvn-app10 from horizon, bots quit and ssh times out === 2023-03-22 === * 03:33 Operator873: Voiced Tulsi in #cvn-sw -meta -mediawiki -commons -simplewikis === 2023-03-13 === * 19:46 Operator873: CVNBot18 restarted === 2023-03-03 === * 14:45 AntiComposite: /cs flags #cvn-sw-spam COIBot bot === 2023-02-27 === * 22:33 herzog: Loaded gur.wikipedia to SWMT Group 4 (CVNBot9) - [[phab:T327842|T327842]] * 18:04 herzog: Loaded guc.wikipedia to CVNBot9 / Group 4 - [[phab:T326236|T326236]] === 2023-02-02 === * 00:21 ma: Added 12 new wikis to CVNBot<nowiki>{</nowiki>6,7,8<nowiki>}</nowiki>, 4 to each one. Refs.: [[phab:T321283|T321283]] [[phab:T321289|T321289]] [[phab:T321295|T321295]] [[phab:T326139|T326139]] [[phab:T305281|T305281]] [[phab:T310873|T310873]] [[phab:T312215|T312215]] [[phab:T314640|T314640]] [[phab:T314646|T314646]] [[phab:T316457|T316457]] [[phab:T317113|T317113]] [[phab:T319191|T319191]] === 2023-01-30 === * 22:50 Krinkle: Delete cvn-app8 and cvn-app9 instances, ref [[phab:T306066|T306066]] === 2023-01-28 === * 02:51 AntiComposite: /cs flags #cvn-sw Ajraddatz local_op === 2023-01-24 === * 08:54 Krinkle: Delete cvn-apache9, [[phab:T306066|T306066]] * 08:54 Krinkle: Suspend cvn-app8 and cvn-app9 (`pgrep -af cvn` is empty on both), [[phab:T306066|T306066]] === 2023-01-23 === * 16:53 AntiComposite: Deploy {{Gerrit|716e140}} to app12 ([[phab:T306066|T306066]]) * 16:50 AntiComposite: Deploy {{Gerrit|716e140}} to app9 ([[phab:T306066|T306066]]) * 16:29 AntiComposite: Deploy {{Gerrit|442f324}} to app12 ([[phab:T306066|T306066]]) * 16:25 AntiComposite: Deploy {{Gerrit|442f324}} to app9 ([[phab:T306066|T306066]]) * 16:01 AntiComposite: Deploy {{Gerrit|9024b8f}} to app12 ([[phab:T306066|T306066]]) * 15:59 AntiComposite: Deploy {{Gerrit|9024b8f}} to app9 ([[phab:T306066|T306066]]) === 2023-01-22 === * 21:40 AntiComposite: start cvndb-CVNBot14-publish on app10 * 21:07 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app10, starting bots ([[phab:T306066|T306066]]) * 20:56 AntiComposite: disable cvndb-CVNBot14-publish on app8 * 20:51 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app8, stopping bots ([[phab:T306066|T306066]]) * 19:53 AntiComposite: Deploy {{Gerrit|80ea1f5}} to cvn-app10 ([[phab:T306066|T306066]]) * 15:43 AntiComposite: restart all CVNBots on app9 * 15:42 AntiComposite: restart all CVNBots on app8 === 2023-01-17 === * 00:15 Krinkle: Suspend cvn-apache9, replaced by cvn-apache10, ref [[phab:T306066|T306066]] * 00:14 Krinkle: Switch cvn.wmflabs.org from cvn-apache9 to cvn-apache10 === 2023-01-16 === * 00:10 Krinkle: Move https://github.com/countervandalism/cvn-clerkbot to https://github.com/wikimedia/countervandalism-cvn-clerkbot (with HTTP and Git redirect preserved), and replace with Gerrit mirror === 2023-01-15 === * 23:12 Krinkle: Create 'labs-cvn' permission group in Gerrit with CVN staff members * 23:12 Krinkle: Move https://github.com/countervandalism/cvn-api to https://github.com/wikimedia/countervandalism-cvn-api (with HTTP and Git redirect preserved), and replace with Gerrit mirror * 22:02 Krinkle: Switch new cvn.wmcloud.org proxy from cvn-apache9 to cvn-apache10 (Leave main cvn.wmflabs.org as-is for now). === 2023-01-14 === * 21:45 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|4cee27a}}) * 21:22 AntiComposite: move cvn-clerbot back to cvn-app9 (deploy {{Gerrit|371ba2a}}) * 21:10 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|3f3f40f}}) === 2023-01-10 === * 23:22 Krinkle: krinkle@cvn-apache9$ update infrastructure.git, sudo apachectl graceful * 23:20 Krinkle: Create cvn.wmcloud.org web proxy (in addition to cvn.wmflabs.org) === 2023-01-07 === * 20:53 AntiComposite: apply role::labs::lvm::srv only to cvn-apache9, cvn-app8, and cvn-app9 to fix puppet failures on new instances === 2023-01-04 === * 20:47 Krinkle: Allocate new floating IPs to cvn-app10 and cvn-app11 * 20:46 Krinkle: Create new cvn-apache10, cvn-app10, cvn-app11 with Debian 11 Bullseye to replace the old Debian 9.1 Stretch instances * 20:04 taavi: bump floating ip quota from 2 to 4, [[phab:T326269|T326269]] === 2022-12-27 === * 20:11 Frosty873: /cs flags #cvn-meta xaosflux voiced * 20:11 Frosty873: /cs flags #cvn-wp-en xaosflux voiced === 2022-12-23 === * 03:25 AntiComposite: /cs flags #cvn-meta tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-mediawiki tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-sw tryvix1509 voiced === 2022-10-18 === * 23:13 Joan: CVNBot3 restarted (Last message was received on RCReader 62854.814658 seconds ag) === 2022-09-04 === * 22:21 Operator873: /cs flags #cvn-simplewikis Enfcer +AV * 02:20 Operator873: /cs flags #cvn-sw Bot873 +voiced === 2022-08-26 === * 14:09 hauskatze: Loaded pcm.wikipedia and guw.wiktionary to CVNBot8 & 9 respectively {{!}} [[phab:T310880|T310880]] [[phab:T309057|T309057]] === 2022-07-09 === * 16:42 AntiComposite: /cs flags #cvn-commons pandakekok9 voiced === 2022-07-08 === * 21:53 Krinkle: krinkle@horizon.wikimedia.org Add anticomposite as project member and project admin to cloudvps.cvn === 2022-07-01 === * 21:39 Krinkle: cvn-app8: kill CVNBot14.exe and two (!) procs for CVNBot18.exe === 2022-06-25 === * 03:25 AntiComposite: /cs flags #cvn-wp-en PhantomTech voiced === 2022-06-22 === * 21:04 op873: <+CVNBot3> Added: LuchoCR is on es.wikipedia bot list, added by Operator873{{!}}CVN until the end of time ("Mass blockiing P2P-proxies with script") * 20:34 op873: restart CVNBot3 (possibly caused by block flood) * 19:31 op873: restart CVNBot3 === 2022-06-15 === * 18:49 AntiComposite: /cs flags #cvn-wp-en Zppix voiced * 18:48 AntiComposite: /cs flags #cvn-simplewikis Zppix voiced === 2022-05-23 === * 00:24 Joan: Flags +AV were set on Sargento in cvn-wp-es * 00:23 Joan: Flags +AV were set on alhen in cvn-wp-es === 2022-05-19 === * 23:10 Joan: CVNBot3 restarted (Last message was received on RCReader 92593.747667 seconds ago) === 2022-05-11 === * 07:34 Operator873: /cs flags #cvn-wp-en Tamzin voiced === 2022-05-07 === * 17:40 Operator873: /cs flags #cvn-sw koi voiced * 17:39 Operator873: /cs flags #cvn-zh-scan koi voiced === 2022-04-28 === * 03:19 Joan: CVNBot3 restarted (Last message was received on RCReader 75273.332577 seconds ago) === 2022-04-22 === * 15:08 AntiComposite: /cs flags #cvn-meta Bsadowski1 voiced === 2022-04-18 === * 20:44 AntiComposite: /cs flags #cvn-sw Vermont voiced === 2022-04-13 === * 22:40 Operator873: /cs flags #cvn-meta Joan voiced * 22:40 Operator873: /cs flags #cvn-sw Joan voiced * 22:14 Joan: CVNBot3 restarted (Last message was received on RCReader 54942.175428 seconds ago) === 2022-04-07 === * 23:15 Operator873: /cs flags #cvn-wp-hr NovakWatchmen local_op * 23:13 Operator873: voiced Superpes (Superpes15) in #cvn-sw #cvn-sw-spam and #cvn-it-scan === 2022-04-04 === * 17:34 Operator873: Voiced Vermont in #cvn-meta and #cvn-simplewikis /cs flags #cvn-meta Vermont voiced === 2022-03-30 === * 14:33 Joan: CVNBot3 restarted (Last message was received on RCReader 26318.335196 seconds ago) === 2022-03-28 === * 02:38 AntiComposite: /cs flags #cvn-wp-en Bsoyka voiced === 2022-03-21 === * 20:22 Operator873: /cs flags #cvn-simplewikis Bsadowski1 +AfiotvV * 20:17 Operator873: Operator873{{!}}CVN (Operator873) set flags +AVfitv on Bsadowski1 * 20:03 Operator873: Operator873{{!}}CVN (Operator873) set flags +V on Bsadowski1 * 17:04 AntiComposite: /cs flags #cvn-sw Bsadowski1 local_op === 2022-03-15 === * 15:38 Joan: CVNBot3 restarted (Last message was received on RCReader 26424.279343 seconds ago) === 2022-03-14 === * 14:02 Joan: CVNBot3 restarted (Last message was received on RCReader 17096.72183 seconds ago) === 2022-03-12 === * 16:27 Joan: CVNBot3 restarted (Last message was received on RCReader 27236.775673 seconds ago) === 2022-03-11 === * 14:24 Joan: CVNBot3 restarted (Last message was received on RCReader 18853.006849 seconds ago) === 2022-03-10 === * 14:08 Joan: CVNBot3 restarted (Last message was received on RCReader 22518.614282 seconds ago) === 2022-03-08 === * 20:27 AntiComposite: /cs flags #cvn-wp-en Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-simplewikis Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-commons Sarrus voiced === 2022-03-07 === * 16:30 AntiComposite: /cs flags #cvn-meta zabe voiced * 16:25 AntiComposite: /cs flags #cvn-simplewikis DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-meta DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-sw TheresNoTime voiced * 16:07 Krinkle: /cs flags #cvn-staff Operator873 staff * 16:07 Krinkle: /cs flags #cvn-staff AntiComposite staff === 2022-03-05 === * 04:13 Joan: CVNBot3 restarted (Last message was received on RCReader 31573.894101 seconds ago) === 2022-03-03 === * 16:39 Joan: CVNBot3 restarted (Last message was received on RCReader 36578.236383 seconds ago) === 2022-03-01 === * 13:21 Joan: CVNBot3 restarted (Last message was received on RCReader 20646.781861 seconds ago) === 2022-02-15 === * 14:12 Joan: CVNBot3 restarted (Last message was received on RCReader 25001.391103 seconds ago) === 2022-02-13 === * 18:47 andrewbogott: switching to project-local nfs server cvn-nfs-1 * 17:54 andrewbogott: switching to project-local nfs server puppet-diffs-nfs-1 === 2022-02-10 === * 16:17 Joan: CVNBot3 restarted (Last message was received on RCReader 39817.871151 seconds ago) === 2022-02-08 === * 15:51 Joan: CVNBot3 restarted (Last message was received on RCReader 28868.916144 seconds ago) === 2022-02-04 === * 23:59 andrewbogott: accidentally restarted all VMs due to misreading the project purge page. sorry! === 2022-02-02 === * CVN: Several bots restarted after netsplit took nickserv and some bots with it. * 10:26 Krinkle: CVNBot1 bes del delete(?!d) — originally added by huh (reason: "widewuto") === 2022-02-01 === * 15:20 Joan: CVNBot3 restarted (Last message was received on RCReader 26990.323435 seconds ago) === 2022-01-31 === * 17:37 Joan: CVNBot3 restarted (Last message was received on RCReader 48827.882566 seconds ago) === 2022-01-27 === * 16:58 Joan: CVNBot3 restarted (Last message was received on RCReader 29206.852828 seconds ago) === 2022-01-21 === * 16:07 Joan: CVNBot3 restarted (Last message was received on RCReader 22091.557102 seconds ago) === 2022-01-20 === * 18:13 Cam11598: CVNBot15 restarted === 2022-01-19 === * 17:26 Joan: Restarted CVNBot3 (Last message was received on RCReader 28129.031916 seconds ago) === 2022-01-18 === * 16:55 Joan: Restarted CVNBot3 (Last message was received on RCReader 26283.381782 seconds ago) === 2022-01-17 === * 16:33 Joan: Restarted CVNBot3 (#cvn-wp-es) (Last message was received on RCReader 197065.877109 seconds ago) === 2022-01-15 === * 04:56 Cam11598: restarted CVNBOT18 8:55:47 PM <�25B100+ CVNBot18> Last message was received on RCReader 29723.456263 seconds ago === 2022-01-13 === * 01:29 Cam11598: restarted CVNBot2 nickserv issue * 01:29 Cam11598: restarted CVNBot18 - no response from RC feed === 2022-01-09 === * 18:18 Joan: Flags +AV were set on Hasley in cvn-wp-es (sysop at es.wikipedia) * 17:56 Krinkle: /cs flags #cvn-wp-es Joan local_op === 2022-01-07 === * 22:08 hauskatze: CVNBot9 load co.wiktionary wikt:co: * 22:04 hauskatze: CVNBot9 load ban.wikisource s:ban: * 22:04 hauskatze: CVNBot9 load ba.wikibooks b:ba: * 10:51 hauskatze: Loaded alt.wikipedia to Group 4 (CVNBot9) - small wiki not monitored === 2022-01-06 === * 19:42 hauskatze: Loaded ami.wikipedia to CVNBot8 - [[phab:T292421|T292421]] * 19:41 hauskatze: Loaded pwn.wikipedia to CVNBot7 - [[phab:T292419|T292419]] * 19:39 hauskatze: Loaded lmo.wiktionary to CVNBot6 - [[phab:T292076|T292076]] * 19:34 hauskatze: Loaded jv.wikisource to CVNBot6 refs. [[phab:T287319|T287319]] * 19:29 Krinkle: cs flags #cvn-sw hauskatze local_op * 13:57 Krinkle: Krinkle added $a:Cam11598 to the #cvn-staff I list (+I) {{SAL|Project Name=cvn}} <noinclude> ==Archives== * [[Nova Resource:Cvn/SAL/Archive 1|Archive 1]] (2006-2009) * [[Nova Resource:Cvn/SAL/Archive 2|Archive 2]] (2010-2011) * [[Nova Resource:Cvn/SAL/Archive 3|Archive 3]] (2012-2013) * [[Nova Resource:Cvn/SAL/Archive 4|Archive 4]] (2013-2021) (some parts in 2013 are not indexed) [[Category:SAL]]</noinclude> 2q76qh13npjknj0p6pnkua41e9u1e72 2426641 2426640 2026-06-13T23:18:38Z Stashbot 7414 AntiComposite: zhswMonitor drop zh.wikinews 2426641 wikitext text/x-wiki === 2026-06-13 === * 23:18 AntiComposite: zhswMonitor drop zh.wikinews * 23:17 AntiComposite: CVNBot29 drop & purge es.wikinews ([[phab:T428622|T428622]]) * 23:13 AntiComposite: CVNBot26 drop & purge ar.wikinews ([[phab:T428622|T428622]]) * 23:12 AntiComposite: CVNBot25 drop & purge ko.wikinews ([[phab:T428622|T428622]]) * 23:12 AntiComposite: CVNBot23 drop & purge zh.wikinews ([[phab:T428622|T428622]]) * 23:11 AntiComposite: CVNBot10 drop & purge ca.wikinews, ko.wikinews, no.wikinews ([[phab:T428622|T428622]]) * 23:07 AntiComposite: CVNBot9 drop & purge bs.wikinews, el.wikinews, fa.wikinews, shn.wikinews, zh.wikinews ([[phab:T428622|T428622]]) * 23:03 AntiComposite: CVNBot8 drop & purge ar.wikinews, cs.wikinews, de.wikinews, fi.wikinews, he.wikinews, ru.wikinews, sq.wikinews, sr.wikinews, uk.wikinews ([[phab:T428622|T428622]]) * 22:58 AntiComposite: CVNBot7 drop & purge es.wikinews, guw.wikinews, pt.wikinews ([[phab:T428622|T428622]]) * 22:56 AntiComposite: CVNBot6 drop & purge eo.wikinews, fr.wikinews, pl.wikinews, ro.wikinews, sv.wikinews, ta.wikinews ([[phab:T428622|T428622]]) * 22:49 AntiComposite: CVNBot4 drop it.wikinews ([[phab:T428622|T428622]]) === 2026-06-02 === * 01:03 Krinkle: /cs flags #cvn-sw Divinations voiced === 2026-05-26 === * 18:07 AntiComposite: restart all bots -- disconnected === 2026-05-03 === * 13:39 Krinkle: Disable "Admin immed notify" for cvn-private https://lists.wikimedia.org/postorius/lists/cvn-private.lists.wikimedia.org/settings/automatic_responses. We previously removed the sub form but this is no longer supported in mailman3. We require confirm/moderate for new subs, there is no way to turn it off. But we can at least disable the noise. === 2026-04-27 === * 12:22 Krinkle: /cs flags #cvn-meta NathanVeritas voiced === 2026-04-01 === * 13:34 AntiComposite: restart all bots === 2026-02-04 === * 20:33 AntiComposite: Restart all bots === 2025-12-26 === * 15:54 Operator873: /cs flags #cvn-zh-scan nya_1F616EMO voiced === 2025-11-27 === * 13:48 AntiComposite: CVNBot10 load tok.wikipedia tok: ([[phab:T404567|T404567]]) * 13:47 AntiComposite: CVNBot9 load ms.wikiquote q:ms: ([[phab:T404700|T404700]]) * 13:45 AntiComposite: CVNBot8 load min.wikisource s:min: ([[phab:T408343|T408343]]) * 13:44 AntiComposite: CVNBot7 load pcm.wikiquote q:pcm: ([[phab:T408351|T408351]]) * 13:43 AntiComposite: CVNBot6 load tl.wikisource s:tl: ([[phab:T388654|T388654]]) * 13:42 AntiComposite: CVNBot10 load bew.wiktionary wikt:bew: ([[phab:T402134|T402134]]) * 13:41 AntiComposite: CVNBot9 load zgh.wiktionary wikt:zgh: ([[phab:T399785|T399785]]) * 13:40 AntiComposite: CVNBot8 load min.wikibooks b:min: ([[phab:T395499|T395499]]) * 13:38 AntiComposite: CVNBot7 load rki.wikipedia rki: ([[phab:T392499|T392499]]) * 13:37 AntiComposite: CVNBot6 load mad.wikisource s:mad: ([[phab:T391767|T391767]]) === 2025-10-28 === * 23:16 AntiComposite: /cs flags #cvn-commons revi local_op === 2025-08-20 === * 20:35 AntiComposite: CVNBot10 load nup.wikipedia nup: ([[phab:T390711|T390711]]) === 2025-07-11 === * 14:38 AntiComposite: cvn-app10 restart all bots * 11:10 AntiComposite: cvn-app12 restart all bots * 11:09 AntiComposite: cvn-app10 restart all bots === 2025-06-20 === * 20:49 AntiComposite: cvn-app12: restart all bots * 20:48 AntiComposite: cvn-app10: restart all bots === 2025-05-26 === * 17:59 Krinkle: Create cvn-app14 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:59 Krinkle: Create cvn-app13 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:57 Krinkle: Delete cvn-apache10 instance (replaced/shutdown 2 days ago), ref [[phab:T395164|T395164]] === 2025-05-23 === * 20:30 Krinkle: Shut off cvn-apache10, [[phab:T395164|T395164]] * 20:29 Krinkle: Change cvn.wmcloud.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 20:22 Krinkle: Change cvn.wmflabs.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 19:45 Krinkle: Create cvn-apache11 (debian-12.0-bookworm, g4.cores2.ram4.disk20), [[phab:T395164|T395164]]) === 2025-05-16 === * 18:22 Krinkle: Replace outreach.wikipedia with outreach.wikimedia in cvn-sw/CVNBot19 per https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/820245 since the source channel was renamed * 17:30 Krinkle: krinkle@cvn-apache10:/srv/cvn/git/infrastructure$ git pull -- Deploy https://gerrit.wikimedia.org/r/1146724 * 17:30 Krinkle: krinkle@cvn-apache10 Update git remote in /srv/cvn/git/infrastructure from github.com/countervandalism to https://gerrit.wikimedia.org/r/labs/countervandalism/cvn-infrastructure === 2025-04-21 === * 17:22 AntiComposite: Hard reboot cvn-app10, flapping and not responsive to ssh === 2025-03-30 === * 06:55 Krinkle: krinkle@cvn-apache10: Run `sudo chmod 644 /srv/cvn/git/infrastructure/crontab-config/*.cron`, per [[phab:T390415|T390415]] === 2025-03-12 === * 02:18 AntiComposite: CVNBot9 load id.wikivoyage voy:id: ([[phab:T381080|T381080]]) * 02:15 AntiComposite: CVNBot8 load tig.wikipedia tig: ([[phab:T381379|T381379]]) * 02:14 AntiComposite: CVNBot7 load knc.wikipedia knc: ([[phab:T385185|T385185]]) * 02:11 AntiComposite: CVNBot6 load syl.wikipedia syl: ([[phab:T386464|T386464]]) * 02:08 AntiComposite: CVNBot10 load sat.wiktionary wikt:sat: ([[phab:T386631|T386631]]) === 2025-02-03 === * 22:05 AntiComposite: Hard reboot cvn-apache10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ * 21:58 AntiComposite: Hard reboot cvn-app10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ === 2025-01-02 === * 12:46 Krinkle: /cs flags #cvn-wp-en Lordseriouspig voiced * 12:45 Krinkle: /cs flags #cvn-sw Lordseriouspig voiced === 2024-11-23 === * 00:41 AntiComposite: CVNBot9 load ka.wikisource s:ka: ([[phab:T363243|T363243]]) * 00:38 AntiComposite: CVNBot8 load tcy.wikisource s:tcy: ([[phab:T378471|T378471]]) * 00:37 AntiComposite: CVNBot7 load tcy.wiktionary wikt:tcy: ([[phab:T378463|T378463]]) * 00:25 AntiComposite: Upgrade CVNBot29 to v4.0.4 * 00:25 AntiComposite: Upgrade CVNBot28 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot27 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot26 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot25 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot24 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot23 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot22 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot19 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot17 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot16 to v4.0.4 * 00:20 AntiComposite: Upgrade CVNBot10 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot9 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot8 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot7 to v4.0.4 * 00:17 AntiComposite: Upgrade CVNBot6 to v4.0.4 === 2024-11-22 === * 23:52 AntiComposite: Upgrade CVNBot21 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot20 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot18 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot15 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot14 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot13 to v4.0.4 * 23:49 AntiComposite: Upgrade CVNBot12 to v4.0.4 * 23:48 AntiComposite: Upgrade CVNBot11 to v4.0.4 * 23:47 AntiComposite: Upgrade CVNBot5 to v4.0.4 * 23:45 AntiComposite: Upgrade CVNBot3 to v4.0.4 * 23:44 AntiComposite: Upgrade CVNBot2 to v4.0.4 * 23:41 AntiComposite: Upgrade CVNBot1 to v4.0.4 * 23:32 AntiComposite: Upgrade CVNBot4 to v4.0.4 * 17:08 AntiComposite: restart CVNBots on cvn-app12 due to simultaneous RCReader failure 91950.519949 seconds === 2024-11-08 === * 23:24 AntiComposite: Restarting all CVNBots due to simultaneous RCReader disconnect 54323.128318 seconds ago === 2024-10-29 === * 20:56 AntiComposite: add sh.wikipedia to CVNBot6 as #cvn-wp-sh didn't survive the libera migration * 14:22 AntiComposite: restart all CVNBots === 2024-10-28 === * 12:50 AntiComposite: restarting all CVNBots, not coming up cleanly === 2024-10-25 === * 02:23 AntiComposite: add cs.wikivoyage to CVNBot10 ([[phab:T370913|T370913]]) * 02:21 AntiComposite: add bdr.wikipedia to CVNBot9 ([[phab:T371760|T371760]]) * 02:18 AntiComposite: add mos.wikipedia to CVNBot8 ([[phab:T374644|T374644]]) * 02:14 AntiComposite: add kge.wikipedia to CVNBot7 ([[phab:T374815|T374815]]) * 02:11 AntiComposite: add rsk.wikipedia to CVNBot6 ([[phab:T375017|T375017]]) * 02:07 AntiComposite: add mad.wiktionary to CVNBot9 ([[phab:T375024|T375024]]) * 02:06 AntiComposite: add gor.wikiquote to CVNBot8 ([[phab:T375095|T375095]]) * 02:04 AntiComposite: add nr.wikipedia to CVNBot7 ([[phab:T375102|T375102]]) * 02:01 AntiComposite: add tdd.wikipedia to CVNBot6 ([[phab:T375424|T375424]]) * 01:54 AntiComposite: add shn.wikinews to CVNBot9 ([[phab:T375433|T375433]]) * 01:52 AntiComposite: add iba.wikipedia to CVNBot8 ([[phab:T376572|T376572]]) * 01:50 AntiComposite: add bcl.wikisource to CVNBot7 ([[phab:T377088|T377088]]) * 01:47 AntiComposite: add ann.wikipedia to CVNBot6 ([[phab:T377160|T377160]]) * 01:43 AntiComposite: add igl.wikipedia to CVNBot9 ( [[phab:T363263|T363263]] ) * 01:41 AntiComposite: add my.wikisource to CVNBot8 ([[phab:T363270|T363270]]) * 01:39 AntiComposite: add foundation.wikimedia to CVNBot19 * 01:38 AntiComposite: add wikitech.wikimedia to CVNBot19 === 2024-10-24 === * 11:36 AntiComposite: restart all CVNBots === 2024-10-23 === * 17:33 AntiComposite: restart all CVNBots === 2024-07-03 === * 02:00 AntiComposite: add kus.wikipedia to CVNBot7 ([[phab:T360303|T360303]]) * 01:57 AntiComposite: add bew.wikipedia to CVNBot6 ([[phab:T360310|T360310]]) * 01:54 AntiComposite: add ms.wikisource to CVNBot9 ([[phab:T363250|T363250]]) * 01:53 AntiComposite: add kaa.wiktionary to CVNBot8 ([[phab:T363256|T363256]]) * 01:50 AntiComposite: add dtp.wikipedia to CVNBot7 ([[phab:T365230|T365230]]) * 01:48 AntiComposite: add btm.wikipedia to CVNBot6 ([[phab:T368067|T368067]]) * 01:45 AntiComposite: add fon.wikipedia to CVNBot9 ([[phab:T347939|T347939]]) * 01:43 AntiComposite: add blk.wikisource to CVNBot8 ([[phab:T343542|T343542]]) * 01:41 AntiComposite: su.wikisource to CVNBot7 ([[phab:T343548|T343548]]) * 01:39 AntiComposite: add tly.wikipedia to CVNBot6 ([[phab:T345170|T345170]]) * 01:37 AntiComposite: add dga.wikipedia to CVNBot9 ([[phab:T350229|T350229]]) * 01:35 AntiComposite: add bjn.wikiquote to CVNBot8 ([[phab:T350235|T350235]]) * 01:32 AntiComposite: add zgh.wikipedia to CVNBot7 ([[phab:T350241|T350241]]) * 01:28 AntiComposite: add bbc.wikipedia to CVNBot6 ([[phab:T350373|T350373]]) === 2024-06-24 === * 16:40 Krinkle: cvn-clerkbot parts #cvn-unifications (not operated by CVN, renamed to #wikimedia-unifications) === 2024-06-18 === * 08:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs === 2024-03-22 === * 05:30 Operator873: /cs flags #cvn-simplewikis Drummingman +voice === 2024-02-28 === * 21:34 Krinkle: /cs flags #cvn-wp-da Sarrus local_op === 2024-01-11 === * 12:19 AntiComposite: /cs flags #cvn-meta Bsadowski1 local_op === 2023-12-01 === * 15:30 AntiComposite: restart everything after WMCS network outage === 2023-10-07 === * 14:50 AntiComposite: kill 2 CVNBot11 processes and restart, bot not joined to IRC === 2023-09-22 === * 00:06 Op873: /cs flags #cvn-wp-en Oshwah +AV === 2023-09-16 === * 10:33 JackSparrow: /cs flags #cvn-wp-fa Arian_Ar local_op === 2023-09-07 === * 01:35 AntiComposite: restart all cvn-app12 bots * 01:33 AntiComposite: restart all cvn-app10 bots === 2023-08-15 === * 14:44 AntiComposite: reboot cvn-app10 from Horizon, bots dead and not responding to SSH === 2023-08-09 === * 00:07 AntiComposite: add 9 wikis to #cvn-sw (ref [[phab:T332379|T332379]] [[phab:T336115|T336115]] [[phab:T332093|T332093]] [[phab:T332093|T332093]] [[phab:T335987|T335987]] [[phab:T334459|T334459]] [[phab:T333271|T333271]] [[phab:T334740|T334740]] [[phab:T342865|T342865]]) === 2023-08-08 === * 23:46 AntiComposite: drop wo.wikiquote from CVNBot10 (closed) [[phab:T334482|T334482]] === 2023-07-27 === * 18:15 AntiComposite: Kill and restart CVNBot29 on cvn-app12 === 2023-07-06 === * 16:21 AntiComposite: point git repos to gerrit on cvn-app10 * 16:19 AntiComposite: point git repos to gerrit on cvn-app12 * 16:03 AntiComposite: CVNBot v4.0.3 deployed to all bots ([[phab:T327126|T327126]], [[phab:T327127|T327127]]) * 16:01 AntiComposite: Upgrade CVNBot29 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot28 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot27 to v4.0.3 * 15:59 AntiComposite: Upgrade CVNBot26 to v4.0.3 * 15:58 AntiComposite: Upgrade CVNBot25 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot24 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot23 to v4.0.3 * 15:55 AntiComposite: Upgrade CVNBot22 to v4.0.3 * 15:54 AntiComposite: Upgrade CVNBot19 to v4.0.3 * 15:53 AntiComposite: Upgrade CVNBot17 to v4.0.3 * 15:46 AntiComposite: Upgrade CVNBot16 to v4.0.3 * 15:44 AntiComposite: Upgrade CVNBot10 to v4.0.3 * 15:41 AntiComposite: Upgrade CVNBot9 to v4.0.3 * 15:40 AntiComposite: Upgrade CVNBot8 to v4.0.3 * 15:39 AntiComposite: Upgrade CVNBot7 to v4.0.3 * 15:38 AntiComposite: Upgrade CVNBot6 to v4.0.3 * 04:37 AntiComposite: Upgrade CVNBot21 to v4.0.3 * 04:34 AntiComposite: Upgrade CVNBot20 to v4.0.3 * 04:33 AntiComposite: Upgrade CVNBot18 to v4.0.3 * 04:30 AntiComposite: Upgrade CVNBot15 to v4.0.3 * 04:23 AntiComposite: Upgrade CVNBot14 to v4.0.3 * 04:22 AntiComposite: Upgrade CVNBot13 to v4.0.3 * 04:14 AntiComposite: Upgrade CVNBot12 to v4.0.3 * 04:09 AntiComposite: Upgrade CVNBot11 to v4.0.3 * 04:03 AntiComposite: Upgrade CVNBot5 to v4.0.3 * 04:01 AntiComposite: Upgrade CVNBot4 to v4.0.3 * 04:00 AntiComposite: Upgrade CVNBot3 to v4.0.3 * 03:57 AntiComposite: Upgrade CVNBot2 to v4.0.3 * 03:51 AntiComposite: Upgrade CVNBot1 to v4.0.3 === 2023-06-28 === * 02:34 Operator873: /cs flags #cvn-sw Fehufanga voiced === 2023-06-16 === * 22:05 AntiComposite: manually restart cvn-clerkbot === 2023-05-15 === * 14:58 hauskater: Dropped akwiki and nawiki from CVNBot10 as closed wikis. On-wiki lists require an update. === 2023-04-26 === * 20:07 AntiComposite: /cs flags #cvn-mk-scan M4r51n voiced === 2023-04-21 === * 22:12 Operator873: granted voice to Fehufanga in #cvn-simplewikis === 2023-04-14 === * 18:28 AntiComposite: restart cvn-app10 from horizon, bots quit and ssh times out === 2023-03-22 === * 03:33 Operator873: Voiced Tulsi in #cvn-sw -meta -mediawiki -commons -simplewikis === 2023-03-13 === * 19:46 Operator873: CVNBot18 restarted === 2023-03-03 === * 14:45 AntiComposite: /cs flags #cvn-sw-spam COIBot bot === 2023-02-27 === * 22:33 herzog: Loaded gur.wikipedia to SWMT Group 4 (CVNBot9) - [[phab:T327842|T327842]] * 18:04 herzog: Loaded guc.wikipedia to CVNBot9 / Group 4 - [[phab:T326236|T326236]] === 2023-02-02 === * 00:21 ma: Added 12 new wikis to CVNBot<nowiki>{</nowiki>6,7,8<nowiki>}</nowiki>, 4 to each one. Refs.: [[phab:T321283|T321283]] [[phab:T321289|T321289]] [[phab:T321295|T321295]] [[phab:T326139|T326139]] [[phab:T305281|T305281]] [[phab:T310873|T310873]] [[phab:T312215|T312215]] [[phab:T314640|T314640]] [[phab:T314646|T314646]] [[phab:T316457|T316457]] [[phab:T317113|T317113]] [[phab:T319191|T319191]] === 2023-01-30 === * 22:50 Krinkle: Delete cvn-app8 and cvn-app9 instances, ref [[phab:T306066|T306066]] === 2023-01-28 === * 02:51 AntiComposite: /cs flags #cvn-sw Ajraddatz local_op === 2023-01-24 === * 08:54 Krinkle: Delete cvn-apache9, [[phab:T306066|T306066]] * 08:54 Krinkle: Suspend cvn-app8 and cvn-app9 (`pgrep -af cvn` is empty on both), [[phab:T306066|T306066]] === 2023-01-23 === * 16:53 AntiComposite: Deploy {{Gerrit|716e140}} to app12 ([[phab:T306066|T306066]]) * 16:50 AntiComposite: Deploy {{Gerrit|716e140}} to app9 ([[phab:T306066|T306066]]) * 16:29 AntiComposite: Deploy {{Gerrit|442f324}} to app12 ([[phab:T306066|T306066]]) * 16:25 AntiComposite: Deploy {{Gerrit|442f324}} to app9 ([[phab:T306066|T306066]]) * 16:01 AntiComposite: Deploy {{Gerrit|9024b8f}} to app12 ([[phab:T306066|T306066]]) * 15:59 AntiComposite: Deploy {{Gerrit|9024b8f}} to app9 ([[phab:T306066|T306066]]) === 2023-01-22 === * 21:40 AntiComposite: start cvndb-CVNBot14-publish on app10 * 21:07 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app10, starting bots ([[phab:T306066|T306066]]) * 20:56 AntiComposite: disable cvndb-CVNBot14-publish on app8 * 20:51 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app8, stopping bots ([[phab:T306066|T306066]]) * 19:53 AntiComposite: Deploy {{Gerrit|80ea1f5}} to cvn-app10 ([[phab:T306066|T306066]]) * 15:43 AntiComposite: restart all CVNBots on app9 * 15:42 AntiComposite: restart all CVNBots on app8 === 2023-01-17 === * 00:15 Krinkle: Suspend cvn-apache9, replaced by cvn-apache10, ref [[phab:T306066|T306066]] * 00:14 Krinkle: Switch cvn.wmflabs.org from cvn-apache9 to cvn-apache10 === 2023-01-16 === * 00:10 Krinkle: Move https://github.com/countervandalism/cvn-clerkbot to https://github.com/wikimedia/countervandalism-cvn-clerkbot (with HTTP and Git redirect preserved), and replace with Gerrit mirror === 2023-01-15 === * 23:12 Krinkle: Create 'labs-cvn' permission group in Gerrit with CVN staff members * 23:12 Krinkle: Move https://github.com/countervandalism/cvn-api to https://github.com/wikimedia/countervandalism-cvn-api (with HTTP and Git redirect preserved), and replace with Gerrit mirror * 22:02 Krinkle: Switch new cvn.wmcloud.org proxy from cvn-apache9 to cvn-apache10 (Leave main cvn.wmflabs.org as-is for now). === 2023-01-14 === * 21:45 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|4cee27a}}) * 21:22 AntiComposite: move cvn-clerbot back to cvn-app9 (deploy {{Gerrit|371ba2a}}) * 21:10 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|3f3f40f}}) === 2023-01-10 === * 23:22 Krinkle: krinkle@cvn-apache9$ update infrastructure.git, sudo apachectl graceful * 23:20 Krinkle: Create cvn.wmcloud.org web proxy (in addition to cvn.wmflabs.org) === 2023-01-07 === * 20:53 AntiComposite: apply role::labs::lvm::srv only to cvn-apache9, cvn-app8, and cvn-app9 to fix puppet failures on new instances === 2023-01-04 === * 20:47 Krinkle: Allocate new floating IPs to cvn-app10 and cvn-app11 * 20:46 Krinkle: Create new cvn-apache10, cvn-app10, cvn-app11 with Debian 11 Bullseye to replace the old Debian 9.1 Stretch instances * 20:04 taavi: bump floating ip quota from 2 to 4, [[phab:T326269|T326269]] === 2022-12-27 === * 20:11 Frosty873: /cs flags #cvn-meta xaosflux voiced * 20:11 Frosty873: /cs flags #cvn-wp-en xaosflux voiced === 2022-12-23 === * 03:25 AntiComposite: /cs flags #cvn-meta tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-mediawiki tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-sw tryvix1509 voiced === 2022-10-18 === * 23:13 Joan: CVNBot3 restarted (Last message was received on RCReader 62854.814658 seconds ag) === 2022-09-04 === * 22:21 Operator873: /cs flags #cvn-simplewikis Enfcer +AV * 02:20 Operator873: /cs flags #cvn-sw Bot873 +voiced === 2022-08-26 === * 14:09 hauskatze: Loaded pcm.wikipedia and guw.wiktionary to CVNBot8 & 9 respectively {{!}} [[phab:T310880|T310880]] [[phab:T309057|T309057]] === 2022-07-09 === * 16:42 AntiComposite: /cs flags #cvn-commons pandakekok9 voiced === 2022-07-08 === * 21:53 Krinkle: krinkle@horizon.wikimedia.org Add anticomposite as project member and project admin to cloudvps.cvn === 2022-07-01 === * 21:39 Krinkle: cvn-app8: kill CVNBot14.exe and two (!) procs for CVNBot18.exe === 2022-06-25 === * 03:25 AntiComposite: /cs flags #cvn-wp-en PhantomTech voiced === 2022-06-22 === * 21:04 op873: <+CVNBot3> Added: LuchoCR is on es.wikipedia bot list, added by Operator873{{!}}CVN until the end of time ("Mass blockiing P2P-proxies with script") * 20:34 op873: restart CVNBot3 (possibly caused by block flood) * 19:31 op873: restart CVNBot3 === 2022-06-15 === * 18:49 AntiComposite: /cs flags #cvn-wp-en Zppix voiced * 18:48 AntiComposite: /cs flags #cvn-simplewikis Zppix voiced === 2022-05-23 === * 00:24 Joan: Flags +AV were set on Sargento in cvn-wp-es * 00:23 Joan: Flags +AV were set on alhen in cvn-wp-es === 2022-05-19 === * 23:10 Joan: CVNBot3 restarted (Last message was received on RCReader 92593.747667 seconds ago) === 2022-05-11 === * 07:34 Operator873: /cs flags #cvn-wp-en Tamzin voiced === 2022-05-07 === * 17:40 Operator873: /cs flags #cvn-sw koi voiced * 17:39 Operator873: /cs flags #cvn-zh-scan koi voiced === 2022-04-28 === * 03:19 Joan: CVNBot3 restarted (Last message was received on RCReader 75273.332577 seconds ago) === 2022-04-22 === * 15:08 AntiComposite: /cs flags #cvn-meta Bsadowski1 voiced === 2022-04-18 === * 20:44 AntiComposite: /cs flags #cvn-sw Vermont voiced === 2022-04-13 === * 22:40 Operator873: /cs flags #cvn-meta Joan voiced * 22:40 Operator873: /cs flags #cvn-sw Joan voiced * 22:14 Joan: CVNBot3 restarted (Last message was received on RCReader 54942.175428 seconds ago) === 2022-04-07 === * 23:15 Operator873: /cs flags #cvn-wp-hr NovakWatchmen local_op * 23:13 Operator873: voiced Superpes (Superpes15) in #cvn-sw #cvn-sw-spam and #cvn-it-scan === 2022-04-04 === * 17:34 Operator873: Voiced Vermont in #cvn-meta and #cvn-simplewikis /cs flags #cvn-meta Vermont voiced === 2022-03-30 === * 14:33 Joan: CVNBot3 restarted (Last message was received on RCReader 26318.335196 seconds ago) === 2022-03-28 === * 02:38 AntiComposite: /cs flags #cvn-wp-en Bsoyka voiced === 2022-03-21 === * 20:22 Operator873: /cs flags #cvn-simplewikis Bsadowski1 +AfiotvV * 20:17 Operator873: Operator873{{!}}CVN (Operator873) set flags +AVfitv on Bsadowski1 * 20:03 Operator873: Operator873{{!}}CVN (Operator873) set flags +V on Bsadowski1 * 17:04 AntiComposite: /cs flags #cvn-sw Bsadowski1 local_op === 2022-03-15 === * 15:38 Joan: CVNBot3 restarted (Last message was received on RCReader 26424.279343 seconds ago) === 2022-03-14 === * 14:02 Joan: CVNBot3 restarted (Last message was received on RCReader 17096.72183 seconds ago) === 2022-03-12 === * 16:27 Joan: CVNBot3 restarted (Last message was received on RCReader 27236.775673 seconds ago) === 2022-03-11 === * 14:24 Joan: CVNBot3 restarted (Last message was received on RCReader 18853.006849 seconds ago) === 2022-03-10 === * 14:08 Joan: CVNBot3 restarted (Last message was received on RCReader 22518.614282 seconds ago) === 2022-03-08 === * 20:27 AntiComposite: /cs flags #cvn-wp-en Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-simplewikis Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-commons Sarrus voiced === 2022-03-07 === * 16:30 AntiComposite: /cs flags #cvn-meta zabe voiced * 16:25 AntiComposite: /cs flags #cvn-simplewikis DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-meta DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-sw TheresNoTime voiced * 16:07 Krinkle: /cs flags #cvn-staff Operator873 staff * 16:07 Krinkle: /cs flags #cvn-staff AntiComposite staff === 2022-03-05 === * 04:13 Joan: CVNBot3 restarted (Last message was received on RCReader 31573.894101 seconds ago) === 2022-03-03 === * 16:39 Joan: CVNBot3 restarted (Last message was received on RCReader 36578.236383 seconds ago) === 2022-03-01 === * 13:21 Joan: CVNBot3 restarted (Last message was received on RCReader 20646.781861 seconds ago) === 2022-02-15 === * 14:12 Joan: CVNBot3 restarted (Last message was received on RCReader 25001.391103 seconds ago) === 2022-02-13 === * 18:47 andrewbogott: switching to project-local nfs server cvn-nfs-1 * 17:54 andrewbogott: switching to project-local nfs server puppet-diffs-nfs-1 === 2022-02-10 === * 16:17 Joan: CVNBot3 restarted (Last message was received on RCReader 39817.871151 seconds ago) === 2022-02-08 === * 15:51 Joan: CVNBot3 restarted (Last message was received on RCReader 28868.916144 seconds ago) === 2022-02-04 === * 23:59 andrewbogott: accidentally restarted all VMs due to misreading the project purge page. sorry! === 2022-02-02 === * CVN: Several bots restarted after netsplit took nickserv and some bots with it. * 10:26 Krinkle: CVNBot1 bes del delete(?!d) — originally added by huh (reason: "widewuto") === 2022-02-01 === * 15:20 Joan: CVNBot3 restarted (Last message was received on RCReader 26990.323435 seconds ago) === 2022-01-31 === * 17:37 Joan: CVNBot3 restarted (Last message was received on RCReader 48827.882566 seconds ago) === 2022-01-27 === * 16:58 Joan: CVNBot3 restarted (Last message was received on RCReader 29206.852828 seconds ago) === 2022-01-21 === * 16:07 Joan: CVNBot3 restarted (Last message was received on RCReader 22091.557102 seconds ago) === 2022-01-20 === * 18:13 Cam11598: CVNBot15 restarted === 2022-01-19 === * 17:26 Joan: Restarted CVNBot3 (Last message was received on RCReader 28129.031916 seconds ago) === 2022-01-18 === * 16:55 Joan: Restarted CVNBot3 (Last message was received on RCReader 26283.381782 seconds ago) === 2022-01-17 === * 16:33 Joan: Restarted CVNBot3 (#cvn-wp-es) (Last message was received on RCReader 197065.877109 seconds ago) === 2022-01-15 === * 04:56 Cam11598: restarted CVNBOT18 8:55:47 PM <�25B100+ CVNBot18> Last message was received on RCReader 29723.456263 seconds ago === 2022-01-13 === * 01:29 Cam11598: restarted CVNBot2 nickserv issue * 01:29 Cam11598: restarted CVNBot18 - no response from RC feed === 2022-01-09 === * 18:18 Joan: Flags +AV were set on Hasley in cvn-wp-es (sysop at es.wikipedia) * 17:56 Krinkle: /cs flags #cvn-wp-es Joan local_op === 2022-01-07 === * 22:08 hauskatze: CVNBot9 load co.wiktionary wikt:co: * 22:04 hauskatze: CVNBot9 load ban.wikisource s:ban: * 22:04 hauskatze: CVNBot9 load ba.wikibooks b:ba: * 10:51 hauskatze: Loaded alt.wikipedia to Group 4 (CVNBot9) - small wiki not monitored === 2022-01-06 === * 19:42 hauskatze: Loaded ami.wikipedia to CVNBot8 - [[phab:T292421|T292421]] * 19:41 hauskatze: Loaded pwn.wikipedia to CVNBot7 - [[phab:T292419|T292419]] * 19:39 hauskatze: Loaded lmo.wiktionary to CVNBot6 - [[phab:T292076|T292076]] * 19:34 hauskatze: Loaded jv.wikisource to CVNBot6 refs. [[phab:T287319|T287319]] * 19:29 Krinkle: cs flags #cvn-sw hauskatze local_op * 13:57 Krinkle: Krinkle added $a:Cam11598 to the #cvn-staff I list (+I) {{SAL|Project Name=cvn}} <noinclude> ==Archives== * [[Nova Resource:Cvn/SAL/Archive 1|Archive 1]] (2006-2009) * [[Nova Resource:Cvn/SAL/Archive 2|Archive 2]] (2010-2011) * [[Nova Resource:Cvn/SAL/Archive 3|Archive 3]] (2012-2013) * [[Nova Resource:Cvn/SAL/Archive 4|Archive 4]] (2013-2021) (some parts in 2013 are not indexed) [[Category:SAL]]</noinclude> mv692a3pwabjdb9all7esjas9ji2u54 2426642 2426641 2026-06-13T23:19:49Z Stashbot 7414 AntiComposite: zhswMonitor drop zh.wikinews (T428622) 2426642 wikitext text/x-wiki === 2026-06-13 === * 23:19 AntiComposite: zhswMonitor drop zh.wikinews ([[phab:T428622|T428622]]) * 23:18 AntiComposite: zhswMonitor drop zh.wikinews * 23:17 AntiComposite: CVNBot29 drop & purge es.wikinews ([[phab:T428622|T428622]]) * 23:13 AntiComposite: CVNBot26 drop & purge ar.wikinews ([[phab:T428622|T428622]]) * 23:12 AntiComposite: CVNBot25 drop & purge ko.wikinews ([[phab:T428622|T428622]]) * 23:12 AntiComposite: CVNBot23 drop & purge zh.wikinews ([[phab:T428622|T428622]]) * 23:11 AntiComposite: CVNBot10 drop & purge ca.wikinews, ko.wikinews, no.wikinews ([[phab:T428622|T428622]]) * 23:07 AntiComposite: CVNBot9 drop & purge bs.wikinews, el.wikinews, fa.wikinews, shn.wikinews, zh.wikinews ([[phab:T428622|T428622]]) * 23:03 AntiComposite: CVNBot8 drop & purge ar.wikinews, cs.wikinews, de.wikinews, fi.wikinews, he.wikinews, ru.wikinews, sq.wikinews, sr.wikinews, uk.wikinews ([[phab:T428622|T428622]]) * 22:58 AntiComposite: CVNBot7 drop & purge es.wikinews, guw.wikinews, pt.wikinews ([[phab:T428622|T428622]]) * 22:56 AntiComposite: CVNBot6 drop & purge eo.wikinews, fr.wikinews, pl.wikinews, ro.wikinews, sv.wikinews, ta.wikinews ([[phab:T428622|T428622]]) * 22:49 AntiComposite: CVNBot4 drop it.wikinews ([[phab:T428622|T428622]]) === 2026-06-02 === * 01:03 Krinkle: /cs flags #cvn-sw Divinations voiced === 2026-05-26 === * 18:07 AntiComposite: restart all bots -- disconnected === 2026-05-03 === * 13:39 Krinkle: Disable "Admin immed notify" for cvn-private https://lists.wikimedia.org/postorius/lists/cvn-private.lists.wikimedia.org/settings/automatic_responses. We previously removed the sub form but this is no longer supported in mailman3. We require confirm/moderate for new subs, there is no way to turn it off. But we can at least disable the noise. === 2026-04-27 === * 12:22 Krinkle: /cs flags #cvn-meta NathanVeritas voiced === 2026-04-01 === * 13:34 AntiComposite: restart all bots === 2026-02-04 === * 20:33 AntiComposite: Restart all bots === 2025-12-26 === * 15:54 Operator873: /cs flags #cvn-zh-scan nya_1F616EMO voiced === 2025-11-27 === * 13:48 AntiComposite: CVNBot10 load tok.wikipedia tok: ([[phab:T404567|T404567]]) * 13:47 AntiComposite: CVNBot9 load ms.wikiquote q:ms: ([[phab:T404700|T404700]]) * 13:45 AntiComposite: CVNBot8 load min.wikisource s:min: ([[phab:T408343|T408343]]) * 13:44 AntiComposite: CVNBot7 load pcm.wikiquote q:pcm: ([[phab:T408351|T408351]]) * 13:43 AntiComposite: CVNBot6 load tl.wikisource s:tl: ([[phab:T388654|T388654]]) * 13:42 AntiComposite: CVNBot10 load bew.wiktionary wikt:bew: ([[phab:T402134|T402134]]) * 13:41 AntiComposite: CVNBot9 load zgh.wiktionary wikt:zgh: ([[phab:T399785|T399785]]) * 13:40 AntiComposite: CVNBot8 load min.wikibooks b:min: ([[phab:T395499|T395499]]) * 13:38 AntiComposite: CVNBot7 load rki.wikipedia rki: ([[phab:T392499|T392499]]) * 13:37 AntiComposite: CVNBot6 load mad.wikisource s:mad: ([[phab:T391767|T391767]]) === 2025-10-28 === * 23:16 AntiComposite: /cs flags #cvn-commons revi local_op === 2025-08-20 === * 20:35 AntiComposite: CVNBot10 load nup.wikipedia nup: ([[phab:T390711|T390711]]) === 2025-07-11 === * 14:38 AntiComposite: cvn-app10 restart all bots * 11:10 AntiComposite: cvn-app12 restart all bots * 11:09 AntiComposite: cvn-app10 restart all bots === 2025-06-20 === * 20:49 AntiComposite: cvn-app12: restart all bots * 20:48 AntiComposite: cvn-app10: restart all bots === 2025-05-26 === * 17:59 Krinkle: Create cvn-app14 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:59 Krinkle: Create cvn-app13 (debian-12.0-bookworm, g4.cores2.ram4.disk20) * 17:57 Krinkle: Delete cvn-apache10 instance (replaced/shutdown 2 days ago), ref [[phab:T395164|T395164]] === 2025-05-23 === * 20:30 Krinkle: Shut off cvn-apache10, [[phab:T395164|T395164]] * 20:29 Krinkle: Change cvn.wmcloud.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 20:22 Krinkle: Change cvn.wmflabs.org web proxy from cvn-apache10 to cvn-apache11, [[phab:T395164|T395164]] * 19:45 Krinkle: Create cvn-apache11 (debian-12.0-bookworm, g4.cores2.ram4.disk20), [[phab:T395164|T395164]]) === 2025-05-16 === * 18:22 Krinkle: Replace outreach.wikipedia with outreach.wikimedia in cvn-sw/CVNBot19 per https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/820245 since the source channel was renamed * 17:30 Krinkle: krinkle@cvn-apache10:/srv/cvn/git/infrastructure$ git pull -- Deploy https://gerrit.wikimedia.org/r/1146724 * 17:30 Krinkle: krinkle@cvn-apache10 Update git remote in /srv/cvn/git/infrastructure from github.com/countervandalism to https://gerrit.wikimedia.org/r/labs/countervandalism/cvn-infrastructure === 2025-04-21 === * 17:22 AntiComposite: Hard reboot cvn-app10, flapping and not responsive to ssh === 2025-03-30 === * 06:55 Krinkle: krinkle@cvn-apache10: Run `sudo chmod 644 /srv/cvn/git/infrastructure/crontab-config/*.cron`, per [[phab:T390415|T390415]] === 2025-03-12 === * 02:18 AntiComposite: CVNBot9 load id.wikivoyage voy:id: ([[phab:T381080|T381080]]) * 02:15 AntiComposite: CVNBot8 load tig.wikipedia tig: ([[phab:T381379|T381379]]) * 02:14 AntiComposite: CVNBot7 load knc.wikipedia knc: ([[phab:T385185|T385185]]) * 02:11 AntiComposite: CVNBot6 load syl.wikipedia syl: ([[phab:T386464|T386464]]) * 02:08 AntiComposite: CVNBot10 load sat.wiktionary wikt:sat: ([[phab:T386631|T386631]]) === 2025-02-03 === * 22:05 AntiComposite: Hard reboot cvn-apache10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ * 21:58 AntiComposite: Hard reboot cvn-app10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ === 2025-01-02 === * 12:46 Krinkle: /cs flags #cvn-wp-en Lordseriouspig voiced * 12:45 Krinkle: /cs flags #cvn-sw Lordseriouspig voiced === 2024-11-23 === * 00:41 AntiComposite: CVNBot9 load ka.wikisource s:ka: ([[phab:T363243|T363243]]) * 00:38 AntiComposite: CVNBot8 load tcy.wikisource s:tcy: ([[phab:T378471|T378471]]) * 00:37 AntiComposite: CVNBot7 load tcy.wiktionary wikt:tcy: ([[phab:T378463|T378463]]) * 00:25 AntiComposite: Upgrade CVNBot29 to v4.0.4 * 00:25 AntiComposite: Upgrade CVNBot28 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot27 to v4.0.4 * 00:24 AntiComposite: Upgrade CVNBot26 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot25 to v4.0.4 * 00:23 AntiComposite: Upgrade CVNBot24 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot23 to v4.0.4 * 00:22 AntiComposite: Upgrade CVNBot22 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot19 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot17 to v4.0.4 * 00:21 AntiComposite: Upgrade CVNBot16 to v4.0.4 * 00:20 AntiComposite: Upgrade CVNBot10 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot9 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot8 to v4.0.4 * 00:19 AntiComposite: Upgrade CVNBot7 to v4.0.4 * 00:17 AntiComposite: Upgrade CVNBot6 to v4.0.4 === 2024-11-22 === * 23:52 AntiComposite: Upgrade CVNBot21 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot20 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot18 to v4.0.4 * 23:51 AntiComposite: Upgrade CVNBot15 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot14 to v4.0.4 * 23:50 AntiComposite: Upgrade CVNBot13 to v4.0.4 * 23:49 AntiComposite: Upgrade CVNBot12 to v4.0.4 * 23:48 AntiComposite: Upgrade CVNBot11 to v4.0.4 * 23:47 AntiComposite: Upgrade CVNBot5 to v4.0.4 * 23:45 AntiComposite: Upgrade CVNBot3 to v4.0.4 * 23:44 AntiComposite: Upgrade CVNBot2 to v4.0.4 * 23:41 AntiComposite: Upgrade CVNBot1 to v4.0.4 * 23:32 AntiComposite: Upgrade CVNBot4 to v4.0.4 * 17:08 AntiComposite: restart CVNBots on cvn-app12 due to simultaneous RCReader failure 91950.519949 seconds === 2024-11-08 === * 23:24 AntiComposite: Restarting all CVNBots due to simultaneous RCReader disconnect 54323.128318 seconds ago === 2024-10-29 === * 20:56 AntiComposite: add sh.wikipedia to CVNBot6 as #cvn-wp-sh didn't survive the libera migration * 14:22 AntiComposite: restart all CVNBots === 2024-10-28 === * 12:50 AntiComposite: restarting all CVNBots, not coming up cleanly === 2024-10-25 === * 02:23 AntiComposite: add cs.wikivoyage to CVNBot10 ([[phab:T370913|T370913]]) * 02:21 AntiComposite: add bdr.wikipedia to CVNBot9 ([[phab:T371760|T371760]]) * 02:18 AntiComposite: add mos.wikipedia to CVNBot8 ([[phab:T374644|T374644]]) * 02:14 AntiComposite: add kge.wikipedia to CVNBot7 ([[phab:T374815|T374815]]) * 02:11 AntiComposite: add rsk.wikipedia to CVNBot6 ([[phab:T375017|T375017]]) * 02:07 AntiComposite: add mad.wiktionary to CVNBot9 ([[phab:T375024|T375024]]) * 02:06 AntiComposite: add gor.wikiquote to CVNBot8 ([[phab:T375095|T375095]]) * 02:04 AntiComposite: add nr.wikipedia to CVNBot7 ([[phab:T375102|T375102]]) * 02:01 AntiComposite: add tdd.wikipedia to CVNBot6 ([[phab:T375424|T375424]]) * 01:54 AntiComposite: add shn.wikinews to CVNBot9 ([[phab:T375433|T375433]]) * 01:52 AntiComposite: add iba.wikipedia to CVNBot8 ([[phab:T376572|T376572]]) * 01:50 AntiComposite: add bcl.wikisource to CVNBot7 ([[phab:T377088|T377088]]) * 01:47 AntiComposite: add ann.wikipedia to CVNBot6 ([[phab:T377160|T377160]]) * 01:43 AntiComposite: add igl.wikipedia to CVNBot9 ( [[phab:T363263|T363263]] ) * 01:41 AntiComposite: add my.wikisource to CVNBot8 ([[phab:T363270|T363270]]) * 01:39 AntiComposite: add foundation.wikimedia to CVNBot19 * 01:38 AntiComposite: add wikitech.wikimedia to CVNBot19 === 2024-10-24 === * 11:36 AntiComposite: restart all CVNBots === 2024-10-23 === * 17:33 AntiComposite: restart all CVNBots === 2024-07-03 === * 02:00 AntiComposite: add kus.wikipedia to CVNBot7 ([[phab:T360303|T360303]]) * 01:57 AntiComposite: add bew.wikipedia to CVNBot6 ([[phab:T360310|T360310]]) * 01:54 AntiComposite: add ms.wikisource to CVNBot9 ([[phab:T363250|T363250]]) * 01:53 AntiComposite: add kaa.wiktionary to CVNBot8 ([[phab:T363256|T363256]]) * 01:50 AntiComposite: add dtp.wikipedia to CVNBot7 ([[phab:T365230|T365230]]) * 01:48 AntiComposite: add btm.wikipedia to CVNBot6 ([[phab:T368067|T368067]]) * 01:45 AntiComposite: add fon.wikipedia to CVNBot9 ([[phab:T347939|T347939]]) * 01:43 AntiComposite: add blk.wikisource to CVNBot8 ([[phab:T343542|T343542]]) * 01:41 AntiComposite: su.wikisource to CVNBot7 ([[phab:T343548|T343548]]) * 01:39 AntiComposite: add tly.wikipedia to CVNBot6 ([[phab:T345170|T345170]]) * 01:37 AntiComposite: add dga.wikipedia to CVNBot9 ([[phab:T350229|T350229]]) * 01:35 AntiComposite: add bjn.wikiquote to CVNBot8 ([[phab:T350235|T350235]]) * 01:32 AntiComposite: add zgh.wikipedia to CVNBot7 ([[phab:T350241|T350241]]) * 01:28 AntiComposite: add bbc.wikipedia to CVNBot6 ([[phab:T350373|T350373]]) === 2024-06-24 === * 16:40 Krinkle: cvn-clerkbot parts #cvn-unifications (not operated by CVN, renamed to #wikimedia-unifications) === 2024-06-18 === * 08:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs === 2024-03-22 === * 05:30 Operator873: /cs flags #cvn-simplewikis Drummingman +voice === 2024-02-28 === * 21:34 Krinkle: /cs flags #cvn-wp-da Sarrus local_op === 2024-01-11 === * 12:19 AntiComposite: /cs flags #cvn-meta Bsadowski1 local_op === 2023-12-01 === * 15:30 AntiComposite: restart everything after WMCS network outage === 2023-10-07 === * 14:50 AntiComposite: kill 2 CVNBot11 processes and restart, bot not joined to IRC === 2023-09-22 === * 00:06 Op873: /cs flags #cvn-wp-en Oshwah +AV === 2023-09-16 === * 10:33 JackSparrow: /cs flags #cvn-wp-fa Arian_Ar local_op === 2023-09-07 === * 01:35 AntiComposite: restart all cvn-app12 bots * 01:33 AntiComposite: restart all cvn-app10 bots === 2023-08-15 === * 14:44 AntiComposite: reboot cvn-app10 from Horizon, bots dead and not responding to SSH === 2023-08-09 === * 00:07 AntiComposite: add 9 wikis to #cvn-sw (ref [[phab:T332379|T332379]] [[phab:T336115|T336115]] [[phab:T332093|T332093]] [[phab:T332093|T332093]] [[phab:T335987|T335987]] [[phab:T334459|T334459]] [[phab:T333271|T333271]] [[phab:T334740|T334740]] [[phab:T342865|T342865]]) === 2023-08-08 === * 23:46 AntiComposite: drop wo.wikiquote from CVNBot10 (closed) [[phab:T334482|T334482]] === 2023-07-27 === * 18:15 AntiComposite: Kill and restart CVNBot29 on cvn-app12 === 2023-07-06 === * 16:21 AntiComposite: point git repos to gerrit on cvn-app10 * 16:19 AntiComposite: point git repos to gerrit on cvn-app12 * 16:03 AntiComposite: CVNBot v4.0.3 deployed to all bots ([[phab:T327126|T327126]], [[phab:T327127|T327127]]) * 16:01 AntiComposite: Upgrade CVNBot29 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot28 to v4.0.3 * 16:00 AntiComposite: Upgrade CVNBot27 to v4.0.3 * 15:59 AntiComposite: Upgrade CVNBot26 to v4.0.3 * 15:58 AntiComposite: Upgrade CVNBot25 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot24 to v4.0.3 * 15:57 AntiComposite: Upgrade CVNBot23 to v4.0.3 * 15:55 AntiComposite: Upgrade CVNBot22 to v4.0.3 * 15:54 AntiComposite: Upgrade CVNBot19 to v4.0.3 * 15:53 AntiComposite: Upgrade CVNBot17 to v4.0.3 * 15:46 AntiComposite: Upgrade CVNBot16 to v4.0.3 * 15:44 AntiComposite: Upgrade CVNBot10 to v4.0.3 * 15:41 AntiComposite: Upgrade CVNBot9 to v4.0.3 * 15:40 AntiComposite: Upgrade CVNBot8 to v4.0.3 * 15:39 AntiComposite: Upgrade CVNBot7 to v4.0.3 * 15:38 AntiComposite: Upgrade CVNBot6 to v4.0.3 * 04:37 AntiComposite: Upgrade CVNBot21 to v4.0.3 * 04:34 AntiComposite: Upgrade CVNBot20 to v4.0.3 * 04:33 AntiComposite: Upgrade CVNBot18 to v4.0.3 * 04:30 AntiComposite: Upgrade CVNBot15 to v4.0.3 * 04:23 AntiComposite: Upgrade CVNBot14 to v4.0.3 * 04:22 AntiComposite: Upgrade CVNBot13 to v4.0.3 * 04:14 AntiComposite: Upgrade CVNBot12 to v4.0.3 * 04:09 AntiComposite: Upgrade CVNBot11 to v4.0.3 * 04:03 AntiComposite: Upgrade CVNBot5 to v4.0.3 * 04:01 AntiComposite: Upgrade CVNBot4 to v4.0.3 * 04:00 AntiComposite: Upgrade CVNBot3 to v4.0.3 * 03:57 AntiComposite: Upgrade CVNBot2 to v4.0.3 * 03:51 AntiComposite: Upgrade CVNBot1 to v4.0.3 === 2023-06-28 === * 02:34 Operator873: /cs flags #cvn-sw Fehufanga voiced === 2023-06-16 === * 22:05 AntiComposite: manually restart cvn-clerkbot === 2023-05-15 === * 14:58 hauskater: Dropped akwiki and nawiki from CVNBot10 as closed wikis. On-wiki lists require an update. === 2023-04-26 === * 20:07 AntiComposite: /cs flags #cvn-mk-scan M4r51n voiced === 2023-04-21 === * 22:12 Operator873: granted voice to Fehufanga in #cvn-simplewikis === 2023-04-14 === * 18:28 AntiComposite: restart cvn-app10 from horizon, bots quit and ssh times out === 2023-03-22 === * 03:33 Operator873: Voiced Tulsi in #cvn-sw -meta -mediawiki -commons -simplewikis === 2023-03-13 === * 19:46 Operator873: CVNBot18 restarted === 2023-03-03 === * 14:45 AntiComposite: /cs flags #cvn-sw-spam COIBot bot === 2023-02-27 === * 22:33 herzog: Loaded gur.wikipedia to SWMT Group 4 (CVNBot9) - [[phab:T327842|T327842]] * 18:04 herzog: Loaded guc.wikipedia to CVNBot9 / Group 4 - [[phab:T326236|T326236]] === 2023-02-02 === * 00:21 ma: Added 12 new wikis to CVNBot<nowiki>{</nowiki>6,7,8<nowiki>}</nowiki>, 4 to each one. Refs.: [[phab:T321283|T321283]] [[phab:T321289|T321289]] [[phab:T321295|T321295]] [[phab:T326139|T326139]] [[phab:T305281|T305281]] [[phab:T310873|T310873]] [[phab:T312215|T312215]] [[phab:T314640|T314640]] [[phab:T314646|T314646]] [[phab:T316457|T316457]] [[phab:T317113|T317113]] [[phab:T319191|T319191]] === 2023-01-30 === * 22:50 Krinkle: Delete cvn-app8 and cvn-app9 instances, ref [[phab:T306066|T306066]] === 2023-01-28 === * 02:51 AntiComposite: /cs flags #cvn-sw Ajraddatz local_op === 2023-01-24 === * 08:54 Krinkle: Delete cvn-apache9, [[phab:T306066|T306066]] * 08:54 Krinkle: Suspend cvn-app8 and cvn-app9 (`pgrep -af cvn` is empty on both), [[phab:T306066|T306066]] === 2023-01-23 === * 16:53 AntiComposite: Deploy {{Gerrit|716e140}} to app12 ([[phab:T306066|T306066]]) * 16:50 AntiComposite: Deploy {{Gerrit|716e140}} to app9 ([[phab:T306066|T306066]]) * 16:29 AntiComposite: Deploy {{Gerrit|442f324}} to app12 ([[phab:T306066|T306066]]) * 16:25 AntiComposite: Deploy {{Gerrit|442f324}} to app9 ([[phab:T306066|T306066]]) * 16:01 AntiComposite: Deploy {{Gerrit|9024b8f}} to app12 ([[phab:T306066|T306066]]) * 15:59 AntiComposite: Deploy {{Gerrit|9024b8f}} to app9 ([[phab:T306066|T306066]]) === 2023-01-22 === * 21:40 AntiComposite: start cvndb-CVNBot14-publish on app10 * 21:07 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app10, starting bots ([[phab:T306066|T306066]]) * 20:56 AntiComposite: disable cvndb-CVNBot14-publish on app8 * 20:51 AntiComposite: Deploy {{Gerrit|1acdb8e}} to cvn-app8, stopping bots ([[phab:T306066|T306066]]) * 19:53 AntiComposite: Deploy {{Gerrit|80ea1f5}} to cvn-app10 ([[phab:T306066|T306066]]) * 15:43 AntiComposite: restart all CVNBots on app9 * 15:42 AntiComposite: restart all CVNBots on app8 === 2023-01-17 === * 00:15 Krinkle: Suspend cvn-apache9, replaced by cvn-apache10, ref [[phab:T306066|T306066]] * 00:14 Krinkle: Switch cvn.wmflabs.org from cvn-apache9 to cvn-apache10 === 2023-01-16 === * 00:10 Krinkle: Move https://github.com/countervandalism/cvn-clerkbot to https://github.com/wikimedia/countervandalism-cvn-clerkbot (with HTTP and Git redirect preserved), and replace with Gerrit mirror === 2023-01-15 === * 23:12 Krinkle: Create 'labs-cvn' permission group in Gerrit with CVN staff members * 23:12 Krinkle: Move https://github.com/countervandalism/cvn-api to https://github.com/wikimedia/countervandalism-cvn-api (with HTTP and Git redirect preserved), and replace with Gerrit mirror * 22:02 Krinkle: Switch new cvn.wmcloud.org proxy from cvn-apache9 to cvn-apache10 (Leave main cvn.wmflabs.org as-is for now). === 2023-01-14 === * 21:45 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|4cee27a}}) * 21:22 AntiComposite: move cvn-clerbot back to cvn-app9 (deploy {{Gerrit|371ba2a}}) * 21:10 AntiComposite: move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy {{Gerrit|3f3f40f}}) === 2023-01-10 === * 23:22 Krinkle: krinkle@cvn-apache9$ update infrastructure.git, sudo apachectl graceful * 23:20 Krinkle: Create cvn.wmcloud.org web proxy (in addition to cvn.wmflabs.org) === 2023-01-07 === * 20:53 AntiComposite: apply role::labs::lvm::srv only to cvn-apache9, cvn-app8, and cvn-app9 to fix puppet failures on new instances === 2023-01-04 === * 20:47 Krinkle: Allocate new floating IPs to cvn-app10 and cvn-app11 * 20:46 Krinkle: Create new cvn-apache10, cvn-app10, cvn-app11 with Debian 11 Bullseye to replace the old Debian 9.1 Stretch instances * 20:04 taavi: bump floating ip quota from 2 to 4, [[phab:T326269|T326269]] === 2022-12-27 === * 20:11 Frosty873: /cs flags #cvn-meta xaosflux voiced * 20:11 Frosty873: /cs flags #cvn-wp-en xaosflux voiced === 2022-12-23 === * 03:25 AntiComposite: /cs flags #cvn-meta tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-mediawiki tryvix1509 voiced * 03:24 AntiComposite: /cs flags #cvn-sw tryvix1509 voiced === 2022-10-18 === * 23:13 Joan: CVNBot3 restarted (Last message was received on RCReader 62854.814658 seconds ag) === 2022-09-04 === * 22:21 Operator873: /cs flags #cvn-simplewikis Enfcer +AV * 02:20 Operator873: /cs flags #cvn-sw Bot873 +voiced === 2022-08-26 === * 14:09 hauskatze: Loaded pcm.wikipedia and guw.wiktionary to CVNBot8 & 9 respectively {{!}} [[phab:T310880|T310880]] [[phab:T309057|T309057]] === 2022-07-09 === * 16:42 AntiComposite: /cs flags #cvn-commons pandakekok9 voiced === 2022-07-08 === * 21:53 Krinkle: krinkle@horizon.wikimedia.org Add anticomposite as project member and project admin to cloudvps.cvn === 2022-07-01 === * 21:39 Krinkle: cvn-app8: kill CVNBot14.exe and two (!) procs for CVNBot18.exe === 2022-06-25 === * 03:25 AntiComposite: /cs flags #cvn-wp-en PhantomTech voiced === 2022-06-22 === * 21:04 op873: <+CVNBot3> Added: LuchoCR is on es.wikipedia bot list, added by Operator873{{!}}CVN until the end of time ("Mass blockiing P2P-proxies with script") * 20:34 op873: restart CVNBot3 (possibly caused by block flood) * 19:31 op873: restart CVNBot3 === 2022-06-15 === * 18:49 AntiComposite: /cs flags #cvn-wp-en Zppix voiced * 18:48 AntiComposite: /cs flags #cvn-simplewikis Zppix voiced === 2022-05-23 === * 00:24 Joan: Flags +AV were set on Sargento in cvn-wp-es * 00:23 Joan: Flags +AV were set on alhen in cvn-wp-es === 2022-05-19 === * 23:10 Joan: CVNBot3 restarted (Last message was received on RCReader 92593.747667 seconds ago) === 2022-05-11 === * 07:34 Operator873: /cs flags #cvn-wp-en Tamzin voiced === 2022-05-07 === * 17:40 Operator873: /cs flags #cvn-sw koi voiced * 17:39 Operator873: /cs flags #cvn-zh-scan koi voiced === 2022-04-28 === * 03:19 Joan: CVNBot3 restarted (Last message was received on RCReader 75273.332577 seconds ago) === 2022-04-22 === * 15:08 AntiComposite: /cs flags #cvn-meta Bsadowski1 voiced === 2022-04-18 === * 20:44 AntiComposite: /cs flags #cvn-sw Vermont voiced === 2022-04-13 === * 22:40 Operator873: /cs flags #cvn-meta Joan voiced * 22:40 Operator873: /cs flags #cvn-sw Joan voiced * 22:14 Joan: CVNBot3 restarted (Last message was received on RCReader 54942.175428 seconds ago) === 2022-04-07 === * 23:15 Operator873: /cs flags #cvn-wp-hr NovakWatchmen local_op * 23:13 Operator873: voiced Superpes (Superpes15) in #cvn-sw #cvn-sw-spam and #cvn-it-scan === 2022-04-04 === * 17:34 Operator873: Voiced Vermont in #cvn-meta and #cvn-simplewikis /cs flags #cvn-meta Vermont voiced === 2022-03-30 === * 14:33 Joan: CVNBot3 restarted (Last message was received on RCReader 26318.335196 seconds ago) === 2022-03-28 === * 02:38 AntiComposite: /cs flags #cvn-wp-en Bsoyka voiced === 2022-03-21 === * 20:22 Operator873: /cs flags #cvn-simplewikis Bsadowski1 +AfiotvV * 20:17 Operator873: Operator873{{!}}CVN (Operator873) set flags +AVfitv on Bsadowski1 * 20:03 Operator873: Operator873{{!}}CVN (Operator873) set flags +V on Bsadowski1 * 17:04 AntiComposite: /cs flags #cvn-sw Bsadowski1 local_op === 2022-03-15 === * 15:38 Joan: CVNBot3 restarted (Last message was received on RCReader 26424.279343 seconds ago) === 2022-03-14 === * 14:02 Joan: CVNBot3 restarted (Last message was received on RCReader 17096.72183 seconds ago) === 2022-03-12 === * 16:27 Joan: CVNBot3 restarted (Last message was received on RCReader 27236.775673 seconds ago) === 2022-03-11 === * 14:24 Joan: CVNBot3 restarted (Last message was received on RCReader 18853.006849 seconds ago) === 2022-03-10 === * 14:08 Joan: CVNBot3 restarted (Last message was received on RCReader 22518.614282 seconds ago) === 2022-03-08 === * 20:27 AntiComposite: /cs flags #cvn-wp-en Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-simplewikis Sarrus voiced * 20:27 AntiComposite: /cs flags #cvn-commons Sarrus voiced === 2022-03-07 === * 16:30 AntiComposite: /cs flags #cvn-meta zabe voiced * 16:25 AntiComposite: /cs flags #cvn-simplewikis DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-meta DannyS712 voiced * 16:25 AntiComposite: /cs flags #cvn-sw TheresNoTime voiced * 16:07 Krinkle: /cs flags #cvn-staff Operator873 staff * 16:07 Krinkle: /cs flags #cvn-staff AntiComposite staff === 2022-03-05 === * 04:13 Joan: CVNBot3 restarted (Last message was received on RCReader 31573.894101 seconds ago) === 2022-03-03 === * 16:39 Joan: CVNBot3 restarted (Last message was received on RCReader 36578.236383 seconds ago) === 2022-03-01 === * 13:21 Joan: CVNBot3 restarted (Last message was received on RCReader 20646.781861 seconds ago) === 2022-02-15 === * 14:12 Joan: CVNBot3 restarted (Last message was received on RCReader 25001.391103 seconds ago) === 2022-02-13 === * 18:47 andrewbogott: switching to project-local nfs server cvn-nfs-1 * 17:54 andrewbogott: switching to project-local nfs server puppet-diffs-nfs-1 === 2022-02-10 === * 16:17 Joan: CVNBot3 restarted (Last message was received on RCReader 39817.871151 seconds ago) === 2022-02-08 === * 15:51 Joan: CVNBot3 restarted (Last message was received on RCReader 28868.916144 seconds ago) === 2022-02-04 === * 23:59 andrewbogott: accidentally restarted all VMs due to misreading the project purge page. sorry! === 2022-02-02 === * CVN: Several bots restarted after netsplit took nickserv and some bots with it. * 10:26 Krinkle: CVNBot1 bes del delete(?!d) — originally added by huh (reason: "widewuto") === 2022-02-01 === * 15:20 Joan: CVNBot3 restarted (Last message was received on RCReader 26990.323435 seconds ago) === 2022-01-31 === * 17:37 Joan: CVNBot3 restarted (Last message was received on RCReader 48827.882566 seconds ago) === 2022-01-27 === * 16:58 Joan: CVNBot3 restarted (Last message was received on RCReader 29206.852828 seconds ago) === 2022-01-21 === * 16:07 Joan: CVNBot3 restarted (Last message was received on RCReader 22091.557102 seconds ago) === 2022-01-20 === * 18:13 Cam11598: CVNBot15 restarted === 2022-01-19 === * 17:26 Joan: Restarted CVNBot3 (Last message was received on RCReader 28129.031916 seconds ago) === 2022-01-18 === * 16:55 Joan: Restarted CVNBot3 (Last message was received on RCReader 26283.381782 seconds ago) === 2022-01-17 === * 16:33 Joan: Restarted CVNBot3 (#cvn-wp-es) (Last message was received on RCReader 197065.877109 seconds ago) === 2022-01-15 === * 04:56 Cam11598: restarted CVNBOT18 8:55:47 PM <�25B100+ CVNBot18> Last message was received on RCReader 29723.456263 seconds ago === 2022-01-13 === * 01:29 Cam11598: restarted CVNBot2 nickserv issue * 01:29 Cam11598: restarted CVNBot18 - no response from RC feed === 2022-01-09 === * 18:18 Joan: Flags +AV were set on Hasley in cvn-wp-es (sysop at es.wikipedia) * 17:56 Krinkle: /cs flags #cvn-wp-es Joan local_op === 2022-01-07 === * 22:08 hauskatze: CVNBot9 load co.wiktionary wikt:co: * 22:04 hauskatze: CVNBot9 load ban.wikisource s:ban: * 22:04 hauskatze: CVNBot9 load ba.wikibooks b:ba: * 10:51 hauskatze: Loaded alt.wikipedia to Group 4 (CVNBot9) - small wiki not monitored === 2022-01-06 === * 19:42 hauskatze: Loaded ami.wikipedia to CVNBot8 - [[phab:T292421|T292421]] * 19:41 hauskatze: Loaded pwn.wikipedia to CVNBot7 - [[phab:T292419|T292419]] * 19:39 hauskatze: Loaded lmo.wiktionary to CVNBot6 - [[phab:T292076|T292076]] * 19:34 hauskatze: Loaded jv.wikisource to CVNBot6 refs. [[phab:T287319|T287319]] * 19:29 Krinkle: cs flags #cvn-sw hauskatze local_op * 13:57 Krinkle: Krinkle added $a:Cam11598 to the #cvn-staff I list (+I) {{SAL|Project Name=cvn}} <noinclude> ==Archives== * [[Nova Resource:Cvn/SAL/Archive 1|Archive 1]] (2006-2009) * [[Nova Resource:Cvn/SAL/Archive 2|Archive 2]] (2010-2011) * [[Nova Resource:Cvn/SAL/Archive 3|Archive 3]] (2012-2013) * [[Nova Resource:Cvn/SAL/Archive 4|Archive 4]] (2013-2021) (some parts in 2013 are not indexed) [[Category:SAL]]</noinclude> lhqtes61sacieurf8kcxg34szof3rj9 0 7919 2426644 2426607 2026-06-14T02:00:20Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2426644 wikitext text/x-wiki == 2026-06-14 == * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-06-13 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 35s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-06-12 == * 19:54 dwisehaupt@dns1004: END - running authdns-update * 19:52 dwisehaupt@dns1004: START - running authdns-update * 18:33 dwisehaupt@dns1006: END - running authdns-update * 18:32 dwisehaupt@dns1006: START - running authdns-update * 16:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:10 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:10 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:59 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 15:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:43 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1301371{{!}}Hotfix for T428620 (T428620)]] (duration: 11m 17s) * 14:36 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 14:35 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1301371{{!}}Hotfix for T428620 (T428620)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:31 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1301371{{!}}Hotfix for T428620 (T428620)]] * 14:29 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:28 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 13:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 12:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 12:22 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:22 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:22 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:22 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 12:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 12:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 12:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 12:04 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:04 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:04 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 12:02 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of prometheus5003.eqsin.wmnet to drbd * 12:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus5003.eqsin.wmnet to drbd * 11:40 moritzm: installing Linux 5.10.257 on Bullseye hosts * 11:36 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 11:35 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:24 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 11:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:56 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 10:56 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 10:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:49 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 10:49 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 10:40 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply * 10:37 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply * 10:36 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply * 10:35 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply * 10:35 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply * 10:35 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply * 10:12 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 10:12 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 10:08 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 09:59 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 09:58 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 09:57 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 06:13 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) * 06:11 jmm@cumin2002: START - Cookbook sre.puppet.disable-merges * 03:07 ryankemper: [[phab:T427951|T427951]] sorry, `[eqiad,codfw].mediawiki.page_html_content_change.rc0` (accidentally a word) * 03:06 ryankemper: [[phab:T427951|T427951]] Deleted all 20 unused dev/test topics on kafka-jumbo (verified empty first); 2 (`[eqiad,codfw]page_html_content_change.rc0`) were immediately auto-recreated empty by a still-running `dse-k8s` enrichment consumer; awaiting owner confirmation before final re-delete * 02:01 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 01m 13s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:00 bblack@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () == 2026-06-11 == * 22:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 22:26 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 22:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 22:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 22:05 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300906{{!}}Restore MediaViewer toggle in Special:Preferences (T428742)]] (duration: 30m 51s) * 21:58 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host releases2003.codfw.wmnet with OS trixie * 21:52 egardner@deploy1003: egardner: Continuing with deployment * 21:51 egardner@deploy1003: egardner: Backport for [[gerrit:1300906{{!}}Restore MediaViewer toggle in Special:Preferences (T428742)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:34 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1300906{{!}}Restore MediaViewer toggle in Special:Preferences (T428742)]] * 21:34 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet with reason: host reimage * 21:29 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300913{{!}}Avoid the escaping from nowiki processing (T398967)]] (duration: 09m 09s) * 21:28 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on releases2003.codfw.wmnet with reason: host reimage * 21:25 arlolra@deploy1003: arlolra: Continuing with deployment * 21:22 arlolra@deploy1003: arlolra: Backport for [[gerrit:1300913{{!}}Avoid the escaping from nowiki processing (T398967)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:20 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1300913{{!}}Avoid the escaping from nowiki processing (T398967)]] * 21:07 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300911{{!}}hCaptcha: Enable for badlogin for all small wikis (T426875)]], [[gerrit:1300905{{!}}RadioRangeBallot: Fix strict mode issue (T428947)]] (duration: 10m 43s) * 21:06 bblack@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text and not P<nowiki>{</nowiki>cp7008*<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 21:01 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 21:00 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300911{{!}}hCaptcha: Enable for badlogin for all small wikis (T426875)]], [[gerrit:1300905{{!}}RadioRangeBallot: Fix strict mode issue (T428947)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:56 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300911{{!}}hCaptcha: Enable for badlogin for all small wikis (T426875)]], [[gerrit:1300905{{!}}RadioRangeBallot: Fix strict mode issue (T428947)]] * 20:51 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300842{{!}}Donor Delight Badge: Unify on "Remove badge" language across treatments (T427313)]], [[gerrit:1300843{{!}}[A11y] Donor Badge: Remove Badge button disappears too quickly (T428646)]], [[gerrit:1300896{{!}}Donor Delight Badge, styles: Amending to final design review feedback (T427313)]] (duration: 34m 10s) * 20:39 jdrewniak@deploy1003: annet, jdrewniak: Continuing with deployment * 20:35 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host releases2003.codfw.wmnet with OS trixie * 20:34 jdrewniak@deploy1003: annet, jdrewniak: Backport for [[gerrit:1300842{{!}}Donor Delight Badge: Unify on "Remove badge" language across treatments (T427313)]], [[gerrit:1300843{{!}}[A11y] Donor Badge: Remove Badge button disappears too quickly (T428646)]], [[gerrit:1300896{{!}}Donor Delight Badge, styles: Amending to final design review feedback (T427313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug * 20:17 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1300842{{!}}Donor Delight Badge: Unify on "Remove badge" language across treatments (T427313)]], [[gerrit:1300843{{!}}[A11y] Donor Badge: Remove Badge button disappears too quickly (T428646)]], [[gerrit:1300896{{!}}Donor Delight Badge, styles: Amending to final design review feedback (T427313)]] * 19:12 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 18:12 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 18:12 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 17:52 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300865{{!}}UploadWizard.config.php: Fix cc-by-4.0-heirs msg issue (T428935 T405146)]] (duration: 08m 15s) * 17:48 reedy@deploy1003: reedy: Continuing with deployment * 17:46 reedy@deploy1003: reedy: Backport for [[gerrit:1300865{{!}}UploadWizard.config.php: Fix cc-by-4.0-heirs msg issue (T428935 T405146)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:44 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1300865{{!}}UploadWizard.config.php: Fix cc-by-4.0-heirs msg issue (T428935 T405146)]] * 17:26 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 17:25 blake@deploy1003: Scap cancelled without rolling back. * 17:25 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 17:24 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 17:24 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 17:24 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 17:24 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 17:23 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 17:23 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 17:23 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 17:23 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 17:23 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 17:23 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 17:20 blake@deploy1003: blake: apache config update ([[phab:T428772|T428772]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:20 blake@deploy1003: Started scap sync-world: apache config update ([[phab:T428772|T428772]]) * 17:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 17:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2212: Migration of db2212.codfw.wmnet completed * 17:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 17:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1235: Migration of db1235.eqiad.wmnet completed * 17:08 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 16:45 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:43 dzahn@dns1005: END - running authdns-update * 16:42 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:41 dzahn@dns1005: START - running authdns-update * 16:41 mutante: releases.wikimedia.org - switching backend from codfw to eqiad - releases1003 is now the source of rsync for uploaded releases files (use releases.discovery.wmnet to not have to think about it) - [[phab:T418299|T418299]] * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts rdb2007.codfw.wmnet * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts rdb1011.eqiad.wmnet * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) * 16:34 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb2009.codfw.wmnet * 16:34 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:34 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2009.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:33 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Migration of db2212.codfw.wmnet completed * 16:27 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2009.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:27 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1235: Migration of db1235.eqiad.wmnet completed * 16:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2212.codfw.wmnet with OS trixie * 16:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1235.eqiad.wmnet with OS trixie * 16:13 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:07 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:06 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 16:05 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 16:05 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 16:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 16:04 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2212.codfw.wmnet with reason: host reimage * 16:01 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 16:01 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:01 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 16:01 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 16:00 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 16:00 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 16:00 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 16:00 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2212.codfw.wmnet with reason: host reimage * 15:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1235.eqiad.wmnet with reason: host reimage * 15:58 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 15:58 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 15:57 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 15:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 15:57 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 15:57 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 15:56 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2009.codfw.wmnet * 15:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:55 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb1011.eqiad.wmnet * 15:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 15:55 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2007.codfw.wmnet * 15:54 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 15:54 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1235.eqiad.wmnet with reason: host reimage * 15:54 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 15:53 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 15:53 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 15:40 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 15:40 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2212.codfw.wmnet with OS trixie * 15:39 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 15:39 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1235.eqiad.wmnet with OS trixie * 15:36 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 15:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1235: Upgrading db1235.eqiad.wmnet * 15:35 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 15:35 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1235: Upgrading db1235.eqiad.wmnet * 15:35 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:32 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 15:32 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:31 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 15:30 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300822{{!}}T428849: temporarily disable noisy warnings in HandleParsoidSectionLinks (T428849 T417530)]] (duration: 11m 29s) * 15:27 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2212: Upgrading db2212.codfw.wmnet * 15:26 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2212: Upgrading db2212.codfw.wmnet * 15:26 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:26 cscott@deploy1003: cscott: Continuing with deployment * 15:26 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1235: Upgrading db1235.eqiad.wmnet * 15:25 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1235: Upgrading db1235.eqiad.wmnet * 15:25 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:21 cscott@deploy1003: cscott: Backport for [[gerrit:1300822{{!}}T428849: temporarily disable noisy warnings in HandleParsoidSectionLinks (T428849 T417530)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:19 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1300822{{!}}T428849: temporarily disable noisy warnings in HandleParsoidSectionLinks (T428849 T417530)]] * 15:18 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 15:17 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 15:13 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 15:13 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 15:13 moritzm: installing libdbi-perl security updates * 14:53 moritzm: installing Bind security updates (just client-side tools/libraries) * 14:51 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry * 14:48 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry * 14:43 moritzm: installing Poppler security updates * 14:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:33 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 14:32 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 14:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1234: Migration of db1234.eqiad.wmnet completed * 14:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin02 and group 01 * 14:24 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin02 and group 01 * 14:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:18 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet * 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet * 14:00 Lucas_WMDE: UTC afternoon backport+config window done * 13:58 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300733{{!}}stream: webrequest.page_view_stats.dev0 (T428725)]] (duration: 08m 12s) * 13:57 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp5024.* * 13:55 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: name=cp5024.* * 13:55 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp5020.* * 13:54 javiermonton@deploy1003: javiermonton: Continuing with deployment * 13:52 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1300733{{!}}stream: webrequest.page_view_stats.dev0 (T428725)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:51 slyngshede@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P<nowiki>{</nowiki>lvs5004*<nowiki>}</nowiki> and A:liberica * 13:50 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1300733{{!}}stream: webrequest.page_view_stats.dev0 (T428725)]] * 13:50 slyngshede@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P<nowiki>{</nowiki>lvs5004*<nowiki>}</nowiki> and A:liberica * 13:50 slyngs: reloading liberica config on lvs5004 * 13:50 moritzm: installing openssl security updates * 13:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:46 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 13:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm * 13:46 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1234: Migration of db1234.eqiad.wmnet completed * 13:46 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 13:45 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 13:45 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 13:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2202.codfw.wmnet with OS trixie * 13:43 alexsanford@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298890{{!}}Add 2FA enforcement demotion config for phase 3 groups (T423120)]] (duration: 07m 19s) * 13:39 alexsanford@deploy1003: alexsanford: Continuing with deployment * 13:38 alexsanford@deploy1003: alexsanford: Backport for [[gerrit:1298890{{!}}Add 2FA enforcement demotion config for phase 3 groups (T423120)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:36 alexsanford@deploy1003: Started scap sync-world: Backport for [[gerrit:1298890{{!}}Add 2FA enforcement demotion config for phase 3 groups (T423120)]] * 13:36 slyngshede@dns1004: END - running authdns-update * 13:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1234.eqiad.wmnet with OS trixie * 13:34 moritzm: installing dovecot security updates * 13:34 slyngshede@dns1004: START - running authdns-update * 13:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300787{{!}}hCaptcha: Enable for MobileFrontend on all group1 wikis (T425940)]] (duration: 06m 59s) * 13:29 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 13:29 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 13:29 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 13:29 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 13:28 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 13:28 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 13:28 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 13:27 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300787{{!}}hCaptcha: Enable for MobileFrontend on all group1 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:26 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2202.codfw.wmnet with reason: host reimage * 13:25 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300787{{!}}hCaptcha: Enable for MobileFrontend on all group1 wikis (T425940)]] * 13:25 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/Android_FAQ 'Wikimedia Apps/FAQ/Android' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:24 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/Android_FAQ 'Wikimedia Apps/FAQ/Android' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:22 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300736{{!}}fix: correct intake-url and payload type for NCS experiment events (T422295)]] (duration: 06m 51s) * 13:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage * 13:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1234.eqiad.wmnet with reason: host reimage * 13:18 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Continuing with deployment * 13:18 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2202.codfw.wmnet with reason: host reimage * 13:18 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Backport for [[gerrit:1300736{{!}}fix: correct intake-url and payload type for NCS experiment events (T422295)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:18 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 13:17 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 13:16 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1300736{{!}}fix: correct intake-url and payload type for NCS experiment events (T422295)]] * 13:15 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage * 13:14 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/Android_FAQ 'Wikimedia Apps/FAQ/Android' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:13 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:13 gkyziridis@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300731{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] (duration: 08m 47s) * 13:13 andrewbogott: sudo -i reprepro --noskipold --component thirdparty/openstack-trixie-flamingo-backports update trixie-wikimedia * 13:12 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1234.eqiad.wmnet with reason: host reimage * 13:12 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 13:12 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/iOS_FAQ 'Wikimedia Apps/FAQ/iOS' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:12 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 13:12 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 13:11 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 13:11 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 13:11 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 13:11 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 13:11 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 13:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 13:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 13:09 gkyziridis@deploy1003: gkyziridis: Continuing with deployment * 13:06 gkyziridis@deploy1003: gkyziridis: Backport for [[gerrit:1300731{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:06 claime: echo 'https://api.wikimedia.org/service/lw/specs/openapi.yaml' {{!}} mwscript-k8s --attach -- purgeList.php * 13:04 gkyziridis@deploy1003: Started scap sync-world: Backport for [[gerrit:1300731{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] * 13:02 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2202.codfw.wmnet with OS trixie * 13:00 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:57 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1234.eqiad.wmnet with OS trixie * 12:55 moritzm: installing Exim security updates on Bullseye * 12:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ganeti5006 * 12:47 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti5006 * 12:46 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti5006 * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti5006.eqsin.wmnet 9.0.132.10.in-addr.arpa 9.0.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 12:46 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache ganeti5006.eqsin.wmnet 9.0.132.10.in-addr.arpa 9.0.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5006 - jmm@cumin2002" * 12:46 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5006 - jmm@cumin2002" * 12:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1234: Upgrading db1234.eqiad.wmnet * 12:44 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1234: Upgrading db1234.eqiad.wmnet * 12:44 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2188: Migration of db2188.codfw.wmnet completed * 12:29 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "UX improvements - oblivian@cumin1003" * 12:29 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: UX improvements - oblivian@cumin1003 * 12:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1232: Migration of db1232.eqiad.wmnet completed * 12:28 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: UX improvements - oblivian@cumin1003 * 12:28 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "UX improvements - oblivian@cumin1003" * 12:27 jmm@cumin2002: START - Cookbook sre.dns.netbox * 12:26 jmm@cumin2002: START - Cookbook sre.hosts.move-vlan for host ganeti5006 * 12:26 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm * 12:21 moritzm: remove ganeti5006 from eqsin cluster for reimage [[phab:T428229|T428229]] * 12:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet * 12:10 moritzm: installing openjdk-21 security updates on Bookworm * 12:03 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300764{{!}}Remove GrowthExperiments extension from closed wikis (T428884)]] (duration: 06m 53s) * 11:59 urbanecm@deploy1003: urbanecm: Continuing with deployment * 11:58 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1300764{{!}}Remove GrowthExperiments extension from closed wikis (T428884)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:56 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1300764{{!}}Remove GrowthExperiments extension from closed wikis (T428884)]] * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb1012.eqiad.wmnet * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb2010.codfw.wmnet * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:48 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2010.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 11:46 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:46 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb2008.codfw.wmnet * 11:46 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:46 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2188: Migration of db2188.codfw.wmnet completed * 11:44 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 11:43 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:43 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2010.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 11:43 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1232: Migration of db1232.eqiad.wmnet completed * 11:38 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:37 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 11:37 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 11:36 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 11:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2188.codfw.wmnet with OS trixie * 11:35 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb1012.eqiad.wmnet * 11:34 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2008.codfw.wmnet * 11:34 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2010.codfw.wmnet * 11:33 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 11:32 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 11:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1232.eqiad.wmnet with OS trixie * 11:27 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2002.codfw.wmnet * 11:25 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300749{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300751{{!}}hCaptcha: Enable for DiscussionTools on all wikis (T426039)]] (duration: 08m 38s) * 11:21 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 11:19 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300749{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300751{{!}}hCaptcha: Enable for DiscussionTools on all wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2188.codfw.wmnet with reason: host reimage * 11:17 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300749{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300751{{!}}hCaptcha: Enable for DiscussionTools on all wikis (T426039)]] * 11:15 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2188.codfw.wmnet with reason: host reimage * 11:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1232.eqiad.wmnet with reason: host reimage * 11:13 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc2002.codfw.wmnet * 11:13 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet * 11:11 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 11:09 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2001.codfw.wmnet * 11:09 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1232.eqiad.wmnet with reason: host reimage * 11:08 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet * 11:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:04 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc2001.codfw.wmnet * 11:04 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host testreduce1002.eqiad.wmnet * 11:04 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:02 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1262.eqiad.wmnet with reason: crash * 11:00 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 11:00 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host testreduce1002.eqiad.wmnet * 10:59 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 10:59 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 10:58 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 10:55 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2188.codfw.wmnet with OS trixie * 10:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2188: Upgrading db2188.codfw.wmnet * 10:52 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2188: Upgrading db2188.codfw.wmnet * 10:52 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:52 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1232.eqiad.wmnet with OS trixie * 10:48 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1232: Upgrading db1232.eqiad.wmnet * 10:48 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1232: Upgrading db1232.eqiad.wmnet * 10:48 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:33 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:32 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:31 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300734{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300727{{!}}hCaptcha: Enable for DiscussionTools on group 1 wikis (T426039)]] (duration: 11m 01s) * 10:26 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:23 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:23 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:22 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300734{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300727{{!}}hCaptcha: Enable for DiscussionTools on group 1 wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:20 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300734{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300727{{!}}hCaptcha: Enable for DiscussionTools on group 1 wikis (T426039)]] * 10:18 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:18 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:10 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 10:10 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 10:09 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2045.codfw.wmnet with OS trixie * 10:09 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repool es2046', diff saved to https://phabricator.wikimedia.org/P94069 and previous config saved to /var/cache/conftool/dbconfig/20260611-100221-marostegui.json * 10:01 marostegui@cumin1003: dbctl commit (dc=all): 'Depool es2046', diff saved to https://phabricator.wikimedia.org/P94068 and previous config saved to /var/cache/conftool/dbconfig/20260611-100145-marostegui.json * 10:01 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:59 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300580{{!}}ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] (duration: 15m 41s) * 09:54 jiji@deploy1003: jiji: Continuing with deployment * 09:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2045.codfw.wmnet with reason: host reimage * 09:45 jiji@deploy1003: jiji: Backport for [[gerrit:1300580{{!}}ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:43 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1300580{{!}}ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] * 09:42 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2045.codfw.wmnet with reason: host reimage * 09:37 elukey: uploaded spicerack_12.8.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia * 09:26 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS trixie * 09:26 marostegui@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host es2045.codfw.wmnet with OS bookworm * 09:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2176: Migration of db2176.codfw.wmnet completed * 09:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1219: Migration of db1219.eqiad.wmnet completed * 09:11 claime: cumin -x 'A:swift-fe' "disable-puppet 'Disabling puppet for ratelimit deploy - cgoubert'" * 08:57 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS bookworm * 08:39 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2176: Migration of db2176.codfw.wmnet completed * 08:34 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94055) * 08:34 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1219: Migration of db1219.eqiad.wmnet completed * 08:33 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94053) * 08:30 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T428823|T428823]] (duration: 01m 18s) * 08:29 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T428823|T428823]] * 08:27 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2176.codfw.wmnet with OS trixie * 08:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Migration to 10.11.17 * 08:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 08:25 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 08:25 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool pc1021: Migration to 10.11.17 * 08:25 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94052) * 08:24 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): Testing upgrade for [[phab:T428823|T428823]] (duration: 01m 17s) * 08:23 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): Testing upgrade for [[phab:T428823|T428823]] * 08:22 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94051) * 08:22 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1219.eqiad.wmnet with OS trixie * 08:17 moritzm: installing PHP 8.2 security updates * 08:15 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 08:14 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 08:11 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 08:11 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 08:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2176.codfw.wmnet with reason: host reimage * 08:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1013.eqiad.wmnet with OS trixie * 08:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5004.eqsin.wmnet to cluster eqsin02 and group 01 * 08:06 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 08:06 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 08:05 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2021.codfw.wmnet,pc1021.eqiad.wmnet with reason: upgrade * 08:05 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1219.eqiad.wmnet with reason: host reimage * 08:05 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5004.eqsin.wmnet to cluster eqsin02 and group 01 * 08:05 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 08:05 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 08:04 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2176.codfw.wmnet with reason: host reimage * 08:04 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 08:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 08:03 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 08:03 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5004.eqsin.wmnet * 07:58 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1219.eqiad.wmnet with reason: host reimage * 07:56 marostegui: install mariadb 10.11.17 on pc1 [[phab:T427345|T427345]] * 07:54 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1013.eqiad.wmnet with reason: host reimage * 07:50 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1013.eqiad.wmnet with reason: host reimage * 07:49 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 07:49 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 07:49 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5004.eqsin.wmnet * 07:47 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 07:47 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 07:46 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2176.codfw.wmnet with OS trixie * 07:43 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1219.eqiad.wmnet with OS trixie * 07:43 moritzm: imported Jenkins 2.541.3 for thirdparty/ci (Bullseye) and thirdparty/jenkins (Bookworm, Trixie) * 07:42 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 07:35 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1013.eqiad.wmnet with OS trixie * 07:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2176: Upgrading db2176.codfw.wmnet * 07:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1219: Upgrading db1219.eqiad.wmnet * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2176: Upgrading db2176.codfw.wmnet * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1219: Upgrading db1219.eqiad.wmnet * 07:31 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:30 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 07:29 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1163: Repooling * 07:19 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 06:51 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS trixie * 06:50 marostegui@cumin1003: dbctl commit (dc=all): 'Repool es2042', diff saved to https://phabricator.wikimedia.org/P94044 and previous config saved to /var/cache/conftool/dbconfig/20260611-065049-marostegui.json * 06:50 marostegui@cumin1003: dbctl commit (dc=all): 'Depool es2042', diff saved to https://phabricator.wikimedia.org/P94043 and previous config saved to /var/cache/conftool/dbconfig/20260611-065027-marostegui.json * 06:44 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1163: Repooling * 06:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1163 [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94041 and previous config saved to /var/cache/conftool/dbconfig/20260611-064319-fceratto.json * 06:42 fceratto@dns1005: END - running authdns-update * 06:40 fceratto@dns1005: START - running authdns-update * 06:33 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:33 fceratto@cumin1003: MariaDB change: Setting sections s1 as read-write for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:33 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:33 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1184 to s1 primary and set section read-write [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94040 and previous config saved to /var/cache/conftool/dbconfig/20260611-063323-fceratto.json * 06:32 fceratto@cumin1003: dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94039 and previous config saved to /var/cache/conftool/dbconfig/20260611-063251-fceratto.json * 06:32 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:32 fceratto@cumin1003: Dbctl change: Setting sections s1 as read-write for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:32 fceratto@cumin1003: MariaDB change: Setting sections s1 as read-write for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:31 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:31 fceratto@cumin1003: dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94037 and previous config saved to /var/cache/conftool/dbconfig/20260611-063100-fceratto.json * 06:30 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:30 fceratto@cumin1003: MariaDB change: Setting sections s1 as read-only for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:30 fceratto@cumin1003: Dbctl change: Setting sections s1 as read-only for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:29 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:29 federico3: Starting s1 eqiad failover from db1163 to db1184 - [[phab:T426083|T426083]] * 06:22 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1184 with weight 0 [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94035 and previous config saved to /var/cache/conftool/dbconfig/20260611-062224-fceratto.json * 06:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 30 hosts with reason: Primary switchover s1 [[phab:T426083|T426083]] * 05:37 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 05:28 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 05:27 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 05:18 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 05:17 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS trixie * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2045: Upgrading es2045.codfw.wmnet * 05:16 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2045: Upgrading es2045.codfw.wmnet * 05:16 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 44s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:23 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2046.* * 01:19 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync * 01:18 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync * 01:18 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1009.eqiad.wmnet with OS trixie * 01:12 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:12 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 01:12 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:12 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:11 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:11 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 01:11 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:10 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:10 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 01:09 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 01:09 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 01:08 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 01:08 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 01:08 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 01:07 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 01:07 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 01:06 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 01:06 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 01:06 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 01:05 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 01:05 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 01:05 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 01:02 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1009.eqiad.wmnet with reason: host reimage * 00:58 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1009.eqiad.wmnet with reason: host reimage * 00:54 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main1009 * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main1009 * 00:41 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main1009 * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main1009.eqiad.wmnet 37.48.64.10.in-addr.arpa 7.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:41 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main1009.eqiad.wmnet 37.48.64.10.in-addr.arpa 7.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1009 - jasmine@cumin2002" * 00:40 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1009 - jasmine@cumin2002" * 00:39 cdanis@cumin1003: dbctl commit (dc=all): 'depool db1262', diff saved to https://phabricator.wikimedia.org/P94032 and previous config saved to /var/cache/conftool/dbconfig/20260611-003950-cdanis.json * 00:36 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 00:34 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5020.* * 00:30 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main1009 * 00:30 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1009.eqiad.wmnet with OS trixie * 00:03 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5024.* == 2026-06-10 == * 23:53 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5024.* * 23:15 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300154{{!}}Disable ShortUrl on bdwikimedia, bhwiki, bnwiki, bnwikisource, eswikibooks, gomwiki (T107188)]] (duration: 11m 37s) * 23:11 krinkle@deploy1003: krinkle: Continuing with deployment * 23:06 krinkle@deploy1003: krinkle: Backport for [[gerrit:1300154{{!}}Disable ShortUrl on bdwikimedia, bhwiki, bnwiki, bnwikisource, eswikibooks, gomwiki (T107188)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:04 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1300154{{!}}Disable ShortUrl on bdwikimedia, bhwiki, bnwiki, bnwikisource, eswikibooks, gomwiki (T107188)]] * 22:57 ladsgroup@dns1004: END - running authdns-update * 22:55 ladsgroup@dns1004: START - running authdns-update * 22:13 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5024.eqsin.wmnet with OS trixie * 22:13 mutante: gerrit - restarting service for logging change * 22:11 dzahn@cumin2002: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 0:10:00 on gerrit.wikimedia.org with reason: service restart * 22:09 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on gerrit2003.wikimedia.org with reason: service restart * 22:06 mutante: gerrit-spare: restarting gerrit * 22:06 mutante: gerrit-replica: restarting gerrit * 21:44 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage * 21:37 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage * 21:22 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300250{{!}}ExecuteTestAndCacheJob: Fix stdClasses serialised wrongly by JobQueue (T428801)]], [[gerrit:1300248{{!}}tests: Fix StandaloneHooksTest ordering, now broken by DB upgrade]] (duration: 08m 23s) * 21:17 jforrester@deploy1003: jforrester: Continuing with deployment * 21:15 jforrester@deploy1003: jforrester: Backport for [[gerrit:1300250{{!}}ExecuteTestAndCacheJob: Fix stdClasses serialised wrongly by JobQueue (T428801)]], [[gerrit:1300248{{!}}tests: Fix StandaloneHooksTest ordering, now broken by DB upgrade]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:13 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1300250{{!}}ExecuteTestAndCacheJob: Fix stdClasses serialised wrongly by JobQueue (T428801)]], [[gerrit:1300248{{!}}tests: Fix StandaloneHooksTest ordering, now broken by DB upgrade]] * 21:03 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5024 * 21:02 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5024 * 21:02 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300247{{!}}Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] (duration: 06m 51s) * 21:00 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp5024 * 21:00 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5024.eqsin.wmnet 35.0.132.10.in-addr.arpa 5.3.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 21:00 brett@cumin2002: START - Cookbook sre.dns.wipe-cache cp5024.eqsin.wmnet 35.0.132.10.in-addr.arpa 5.3.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 21:00 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:00 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5024 - brett@cumin2002" * 20:59 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5024 - brett@cumin2002" * 20:57 catrope@deploy1003: catrope: Continuing with deployment * 20:57 catrope@deploy1003: catrope: Backport for [[gerrit:1300247{{!}}Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:55 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1300247{{!}}Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] * 20:54 brett@cumin2002: START - Cookbook sre.dns.netbox * 20:50 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5024 * 20:49 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5024.eqsin.wmnet with OS trixie * 20:48 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5020.* * 20:44 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300073{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] (duration: 11m 55s) * 20:40 catrope@deploy1003: catrope, gkyziridis: Continuing with deployment * 20:34 catrope@deploy1003: catrope, gkyziridis: Backport for [[gerrit:1300073{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:32 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1300073{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] * 20:30 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5020.eqsin.wmnet with OS trixie * 20:30 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300226{{!}}[arzwiki] Change the wordmark (T427720)]] (duration: 09m 49s) * 20:25 catrope@deploy1003: gergesshamon, catrope: Continuing with deployment * 20:22 catrope@deploy1003: gergesshamon, catrope: Backport for [[gerrit:1300226{{!}}[arzwiki] Change the wordmark (T427720)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1300226{{!}}[arzwiki] Change the wordmark (T427720)]] * 19:59 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5020.eqsin.wmnet with reason: host reimage * 19:53 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5020.eqsin.wmnet with reason: host reimage * 19:30 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 19:27 bblack@cumin1003: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 19:23 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2046.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:19 brett@cumin2002: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2046.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5020 * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5020 * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2044.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:18 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp5020 * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5020.eqsin.wmnet 24.0.132.10.in-addr.arpa 4.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:18 brett@cumin2002: START - Cookbook sre.dns.wipe-cache cp5020.eqsin.wmnet 24.0.132.10.in-addr.arpa 4.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:17 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5020 - brett@cumin2002" * 19:17 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5020 - brett@cumin2002" * 19:14 brett@cumin2002: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2044.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:11 brett@cumin2002: START - Cookbook sre.dns.netbox * 19:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 19:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2174: Migration of db2174.codfw.wmnet completed * 19:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 19:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1218: Migration of db1218.eqiad.wmnet completed * 18:24 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5020 * 18:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5020.eqsin.wmnet with OS trixie * 18:22 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2174: Migration of db2174.codfw.wmnet completed * 18:20 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 18:17 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1218: Migration of db1218.eqiad.wmnet completed * 18:16 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5018.* * 18:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2174.codfw.wmnet with OS trixie * 18:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1218.eqiad.wmnet with OS trixie * 17:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2174.codfw.wmnet with reason: host reimage * 17:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1218.eqiad.wmnet with reason: host reimage * 17:46 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2010.codfw.wmnet with OS trixie * 17:45 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync * 17:44 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync * 17:44 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2174.codfw.wmnet with reason: host reimage * 17:42 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1218.eqiad.wmnet with reason: host reimage * 17:33 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94021) * 17:29 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2010.codfw.wmnet with reason: host reimage * 17:26 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1218.eqiad.wmnet with OS trixie * 17:26 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2174.codfw.wmnet with OS trixie * 17:25 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:24 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 17:24 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1218: Upgrading db1218.eqiad.wmnet * 17:24 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:24 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:24 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1218: Upgrading db1218.eqiad.wmnet * 17:23 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 17:23 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2174: Upgrading db2174.codfw.wmnet * 17:23 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 17:23 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2010.codfw.wmnet with reason: host reimage * 17:23 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:22 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2174: Upgrading db2174.codfw.wmnet * 17:22 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:22 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 17:22 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 17:22 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 17:22 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text and not P<nowiki>{</nowiki>cp7008*<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 17:21 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 17:21 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 17:19 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 17:19 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 17:18 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 17:18 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 17:17 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 17:17 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 17:17 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 17:13 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 17:12 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox) * 17:03 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 17:03 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1206: Migration of db1206.eqiad.wmnet completed * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2010 * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2010 * 17:02 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2010 * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2010.codfw.wmnet 35.48.192.10.in-addr.arpa 5.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:02 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2010.codfw.wmnet 35.48.192.10.in-addr.arpa 5.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2010 - jasmine@cumin2002" * 17:01 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2010 - jasmine@cumin2002" * 16:57 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 16:50 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2010 * 16:50 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2010.codfw.wmnet with OS trixie * 16:41 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 16:39 bblack@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 16:39 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 16:34 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 16:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5018.eqsin.wmnet with OS trixie * 16:22 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 16:20 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 16:17 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1206: Migration of db1206.eqiad.wmnet completed * 16:15 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 16:15 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 16:14 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 16:12 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 16:12 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 16:11 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 16:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1206.eqiad.wmnet with OS trixie * 16:01 blblack: apt: uploaded libvmod-wmfuniq 0.3.0 for trixie * 15:59 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5018.eqsin.wmnet with reason: host reimage * 15:53 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:52 vriley@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:51 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5018.eqsin.wmnet with reason: host reimage * 15:50 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1206.eqiad.wmnet with reason: host reimage * 15:45 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1206.eqiad.wmnet with reason: host reimage * 15:43 sukhe@cumin1003: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool drmrs [reason: no reason specified, no task ID specified] * 15:42 sukhe@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: no reason specified, no task ID specified] * 15:38 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 15:38 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2173: Migration of db2173.codfw.wmnet completed * 15:34 topranks: drain traffic through cr2-drmrs to reset pic0 * 15:33 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94013) * 15:30 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1206.eqiad.wmnet with OS trixie * 15:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1206: Upgrading db1206.eqiad.wmnet * 15:28 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1206: Upgrading db1206.eqiad.wmnet * 15:27 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:25 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:24 vriley@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:24 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1009 * 15:24 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Harroyo-wmf out of all services on: 2436 hosts * 15:23 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1009 * 15:21 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:20 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist translate extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release * 15:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5018 * 15:19 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5018 * 15:18 vriley@cumin1003: START - Cookbook sre.dns.netbox * 15:18 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp5018 * 15:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5018.eqsin.wmnet 18.0.132.10.in-addr.arpa 8.1.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 15:18 brett@cumin2002: START - Cookbook sre.dns.wipe-cache cp5018.eqsin.wmnet 18.0.132.10.in-addr.arpa 8.1.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 15:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:15 brett@cumin2002: START - Cookbook sre.dns.netbox * 15:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 15:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1195: Migration of db1195.eqiad.wmnet completed * 15:12 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:11 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:11 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin1003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:11 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin1003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:08 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300169{{!}}Fix snak value display for rtl languages (T360854)]], [[gerrit:1300168{{!}}Fix snak value display for rtl languages (T360854)]] (duration: 08m 39s) * 15:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 15:01 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1300169{{!}}Fix snak value display for rtl languages (T360854)]], [[gerrit:1300168{{!}}Fix snak value display for rtl languages (T360854)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:59 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:59 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1300169{{!}}Fix snak value display for rtl languages (T360854)]], [[gerrit:1300168{{!}}Fix snak value display for rtl languages (T360854)]] * 14:58 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:55 Lucas_WMDE: lucaswerkmeister-wmde@deploy1003 $ printf 'https://www.mediawiki.org/keys/%s\n' '' 'keys.txt' 'keys.html' {{!}} mwscript-k8s --attach --comment=[[phab:T423267|T423267]] purgeList mediawikiwiki * 14:54 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist translate extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release, now with correct schema * 14:53 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2173: Migration of db2173.codfw.wmnet completed * 14:50 ayounsi@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin2003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:50 ayounsi@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:49 ayounsi@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:48 ayounsi@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:47 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299614{{!}}Add my public key to mediawiki.org/keys (T423267)]] (duration: 08m 33s) * 14:46 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:42 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, matmarex: Continuing with deployment * 14:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2173.codfw.wmnet with OS trixie * 14:40 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, matmarex: Backport for [[gerrit:1299614{{!}}Add my public key to mediawiki.org/keys (T423267)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:40 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:40 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:38 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1299614{{!}}Add my public key to mediawiki.org/keys (T423267)]] * 14:38 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox and (A:dnsbox) * 14:34 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:34 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:33 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:29 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1195: Migration of db1195.eqiad.wmnet completed * 14:28 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:27 vriley@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:26 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 14:26 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 14:24 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist translate extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release, now with dblist translate * 14:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2173.codfw.wmnet with reason: host reimage * 14:23 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 14:22 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 14:22 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply * 14:21 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply * 14:20 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox) * 14:20 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2173.codfw.wmnet with reason: host reimage * 14:20 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:19 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:19 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:18 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:18 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:18 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:18 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1195.eqiad.wmnet with OS trixie * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 14:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 14:15 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 14:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 14:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 14:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 14:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:13 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:13 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 14:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 14:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 14:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 14:09 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 14:08 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:08 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 14:07 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 14:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 14:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 14:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 14:02 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2173.codfw.wmnet with OS trixie * 14:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 14:00 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1195.eqiad.wmnet with reason: host reimage * 14:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 13:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2173: Upgrading db2173.codfw.wmnet * 13:59 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2173: Upgrading db2173.codfw.wmnet * 13:58 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:58 atsuko@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/ttmserver-export.php --wiki=default --ttmserver eqiad-test # [[phab:T425377|T425377]] populating production index on test cluster to estimate time required for the release * 13:56 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1195.eqiad.wmnet with reason: host reimage * 13:54 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Atieno out of all services on: 2436 hosts * 13:42 Lucas_WMDE: UTC afternoon backport+config window done * 13:42 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1195.eqiad.wmnet with OS trixie * 13:36 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297237{{!}}wmf-config: Update private subnets to include additions (T427393)]] (duration: 07m 20s) * 13:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1195: Upgrading db1195.eqiad.wmnet * 13:33 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling restart_daemons on A:hcaptcha-proxy and A:hcaptcha-proxy * 13:33 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum and A:durum * 13:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2170: Migration of db2170.codfw.wmnet completed * 13:33 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1195: Upgrading db1195.eqiad.wmnet * 13:32 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:32 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, brett: Continuing with deployment * 13:32 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough * 13:31 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply * 13:31 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, brett: Backport for [[gerrit:1297237{{!}}wmf-config: Update private subnets to include additions (T427393)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:31 eevans@deploy1003: helmfile [staging] START helmfile.d/services/data-gateway: apply * 13:29 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1297237{{!}}wmf-config: Update private subnets to include additions (T427393)]] * 13:28 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp5018.eqsin.wmnet with reason: host down * 13:28 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy (exit_code=0) rolling restart_daemons on A:tcpproxy and A:tcpproxy * 13:25 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5018.eqsin.wmnet,service=(cdn{{!}}ats-be) * 13:22 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox) * 13:20 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum and A:durum * 13:20 sukhe@cumin1003: START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling restart_daemons on A:hcaptcha-proxy and A:hcaptcha-proxy * 13:19 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299676{{!}}Enable ULS v2 on group0 wikis]] (duration: 17m 00s) * 13:19 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough * 13:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1186: Migration of db1186.eqiad.wmnet completed * 13:18 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-test: apply * 13:18 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-test: apply * 13:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply * 13:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply * 13:15 sbisson@deploy1003: sbisson, abi: Continuing with deployment * 13:10 sukhe@cumin1003: START - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy rolling restart_daemons on A:tcpproxy and A:tcpproxy * 13:05 sbisson@deploy1003: sbisson, abi: Backport for [[gerrit:1299676{{!}}Enable ULS v2 on group0 wikis]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:03 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1014.eqiad.wmnet with OS trixie * 13:02 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299676{{!}}Enable ULS v2 on group0 wikis]] * 12:47 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2170: Migration of db2170.codfw.wmnet completed * 12:46 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5004.eqsin.wmnet with OS bookworm * 12:46 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:46 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:46 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:45 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1014.eqiad.wmnet with reason: host reimage * 12:42 topranks: re-map DSCP AF41 from 'low' to 'normal' priority qos class on network [[phab:T424640|T424640]] * 12:41 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1014.eqiad.wmnet with reason: host reimage * 12:36 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2170.codfw.wmnet with OS trixie * 12:33 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1186: Migration of db1186.eqiad.wmnet completed * 12:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5004.eqsin.wmnet with reason: host reimage * 12:24 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1014 * 12:24 jiji@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host rdb1014 * 12:23 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1186.eqiad.wmnet with OS trixie * 12:21 jiji@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host rdb1014 * 12:21 jiji@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb1014.eqiad.wmnet 42.48.64.10.in-addr.arpa 2.4.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 12:21 jiji@cumin1003: START - Cookbook sre.dns.wipe-cache rdb1014.eqiad.wmnet 42.48.64.10.in-addr.arpa 2.4.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 12:21 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:21 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host rdb1014 - jiji@cumin1003" * 12:21 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host rdb1014 - jiji@cumin1003" * 12:20 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5004.eqsin.wmnet with reason: host reimage * 12:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2170.codfw.wmnet with reason: host reimage * 12:16 jiji@cumin1003: START - Cookbook sre.dns.netbox * 12:13 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1014 * 12:12 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1014.eqiad.wmnet with OS trixie * 12:12 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2170.codfw.wmnet with reason: host reimage * 12:08 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300104{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643{{!}}wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] (duration: 11m 06s) * 12:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1186.eqiad.wmnet with reason: host reimage * 12:03 reedy@deploy1003: reedy: Continuing with deployment * 12:02 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1186.eqiad.wmnet with reason: host reimage * 11:59 reedy@deploy1003: reedy: Backport for [[gerrit:1300104{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643{{!}}wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes c * 11:57 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1300104{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643{{!}}wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] * 11:53 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2170.codfw.wmnet with OS trixie * 11:51 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ganeti5004 * 11:51 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti5004 * 11:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2170: Upgrading db2170.codfw.wmnet * 11:49 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2170: Upgrading db2170.codfw.wmnet * 11:49 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti5004 * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti5004.eqsin.wmnet 40.0.132.10.in-addr.arpa 0.4.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 11:49 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache ganeti5004.eqsin.wmnet 40.0.132.10.in-addr.arpa 0.4.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5004 - jmm@cumin2002" * 11:49 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5004 - jmm@cumin2002" * 11:49 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:48 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1186.eqiad.wmnet with OS trixie * 11:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1186: Upgrading db1186.eqiad.wmnet * 11:45 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1186: Upgrading db1186.eqiad.wmnet * 11:45 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:38 jmm@cumin2002: START - Cookbook sre.dns.netbox * 11:35 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 11:34 jmm@cumin2002: START - Cookbook sre.hosts.move-vlan for host ganeti5004 * 11:34 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 11:34 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5004.eqsin.wmnet with OS bookworm * 11:34 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 11:33 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1151: Security updates * 11:33 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 11:33 root@cumin1003: START - Cookbook sre.mysql.parsercache * 11:33 root@cumin1003: START - Cookbook sre.mysql.pool pool db1151: Security updates * 11:31 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply * 11:30 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply * 11:30 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply * 11:30 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:23 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 11:23 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:23 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 11:23 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 11:16 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 11:15 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 11:15 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 11:15 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:09 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1151: Security updates * 11:09 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 11:09 root@cumin1003: START - Cookbook sre.mysql.parsercache * 11:09 root@cumin1003: START - Cookbook sre.mysql.depool depool db1151: Security updates * 11:08 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300092{{!}}ProductionServices: re-add poolcounter2006 (T426736)]] (duration: 06m 55s) * 11:04 blake@deploy1003: blake: Continuing with deployment * 11:04 blake@deploy1003: blake: Backport for [[gerrit:1300092{{!}}ProductionServices: re-add poolcounter2006 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:03 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:01 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300092{{!}}ProductionServices: re-add poolcounter2006 (T426736)]] * 10:59 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet * 10:57 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 10:57 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 10:57 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 10:56 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 10:56 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply * 10:56 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply * 10:56 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet * 10:56 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300087{{!}}ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] (duration: 06m 42s) * 10:51 blake@deploy1003: blake: Continuing with deployment * 10:51 moritzm: remove ganeti5004 from eqsin cluster for reimage [[phab:T428229|T428229]] * 10:51 blake@deploy1003: blake: Backport for [[gerrit:1300087{{!}}ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:49 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300087{{!}}ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] * 10:47 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet * 10:47 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 10:46 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 10:46 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 10:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 10:43 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet * 10:43 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300082{{!}}ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] (duration: 07m 38s) * 10:41 moritzm: installing nginx security updates * 10:38 blake@deploy1003: blake: Continuing with deployment * 10:38 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1152: Security updates * 10:38 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:38 root@cumin1003: START - Cookbook sre.mysql.parsercache * 10:38 root@cumin1003: START - Cookbook sre.mysql.pool pool db1152: Security updates * 10:38 blake@deploy1003: blake: Backport for [[gerrit:1300082{{!}}ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:37 moritzm: failover Ganeti master in eqsin to ganeti5007 [[phab:T428229|T428229]] * 10:35 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300082{{!}}ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] * 10:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 10:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 10:33 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet * 10:29 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet * 10:29 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300072{{!}}ProductionServices: reboot poolcounter1007 (T426736)]] (duration: 07m 45s) * 10:27 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 10:27 jmm@cumin2002: DONE (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for sretest2009.codfw.wmnet: Renew puppet certificate - jmm@cumin2002 * 10:24 blake@deploy1003: blake: Continuing with deployment * 10:23 blake@deploy1003: blake: Backport for [[gerrit:1300072{{!}}ProductionServices: reboot poolcounter1007 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:22 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 10:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 10:21 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:21 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300072{{!}}ProductionServices: reboot poolcounter1007 (T426736)]] * 10:21 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:21 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:21 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:21 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:20 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet * 10:14 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1152: Security updates * 10:14 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:14 root@cumin1003: START - Cookbook sre.mysql.parsercache * 10:14 root@cumin1003: START - Cookbook sre.mysql.depool depool db1152: Security updates * 10:13 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet * 10:12 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300064{{!}}ProductionServices: reboot poolcounter1006.eqiad (T426736)]] (duration: 07m 46s) * 10:07 blake@deploy1003: blake: Continuing with deployment * 10:06 blake@deploy1003: blake: Backport for [[gerrit:1300064{{!}}ProductionServices: reboot poolcounter1006.eqiad (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:04 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300064{{!}}ProductionServices: reboot poolcounter1006.eqiad (T426736)]] * 09:57 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300058{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] (duration: 09m 32s) * 09:52 kharlan@deploy1003: kharlan: Continuing with deployment * 09:49 kharlan@deploy1003: kharlan: Backport for [[gerrit:1300058{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:47 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1300058{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] * 09:35 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox * 09:34 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 09:32 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 09:32 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 09:26 moritzm: upgrade routinator in eqiad to 0.15.2 [[phab:T428456|T428456]] * 09:23 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 09:23 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 09:22 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 09:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus5003.eqsin.wmnet to plain * 09:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus5003.eqsin.wmnet to plain * 09:15 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:04 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:54 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:20 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:09 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:09 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:07 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 08:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:04 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:01 fceratto@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host db1215.eqiad.wmnet with OS trixie * 07:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:53 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:48 javiermonton@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply * 07:48 javiermonton@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply * 07:44 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1215.eqiad.wmnet with reason: host reimage * 07:41 javiermonton@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply * 07:40 javiermonton@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply * 07:40 moritzm: installing openssl security updates * 07:39 fceratto@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1215.eqiad.wmnet with reason: host reimage * 07:38 javiermonton@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply * 07:37 javiermonton@deploy1003: helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply * 07:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:29 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299556{{!}}ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561{{!}}ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529{{!}}translate: adding separate read/write endpoints (T425377)]] (duration: 14m 03s) * 07:25 atsuko@deploy1003: atsuko: Continuing with deployment * 07:23 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db1215.eqiad.wmnet with OS trixie * 07:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1215.eqiad.wmnet with reason: Reimage * 07:21 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:20 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:20 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:17 atsuko@deploy1003: atsuko: Backport for [[gerrit:1299556{{!}}ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561{{!}}ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529{{!}}translate: adding separate read/write endpoints (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be veri * 07:16 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:15 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1299556{{!}}ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561{{!}}ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529{{!}}translate: adding separate read/write endpoints (T425377)]] * 07:14 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:12 atsukoito: backporting extensions/Translate to wmf/1.47.0-wmf.5 and applying the config * 07:12 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 06:45 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 05:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 05:43 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 05:42 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 05:41 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 47s) * 02:07 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1008.eqiad.wmnet with OS trixie * 02:03 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync * 02:02 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:52 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:51 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 01:51 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:50 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:50 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:49 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1008.eqiad.wmnet with reason: host reimage * 01:49 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 01:49 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:49 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:49 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 01:48 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 01:48 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 01:47 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 01:47 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 01:46 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 01:46 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 01:44 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 01:44 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 01:43 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 01:43 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1008.eqiad.wmnet with reason: host reimage * 01:25 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main1008 * 01:24 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main1008 * 01:24 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main1008 * 01:24 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main1008.eqiad.wmnet 45.32.64.10.in-addr.arpa 5.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 01:23 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main1008.eqiad.wmnet 45.32.64.10.in-addr.arpa 5.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 01:23 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 01:23 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1008 - jasmine@cumin2002" * 01:23 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1008 - jasmine@cumin2002" * 01:19 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 01:12 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main1008 * 01:11 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1008.eqiad.wmnet with OS trixie * 01:00 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2009.codfw.wmnet with OS trixie * 00:54 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync * 00:53 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync * 00:43 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2009.codfw.wmnet with reason: host reimage * 00:40 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:38 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2009.codfw.wmnet with reason: host reimage * 00:38 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 00:38 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:37 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:37 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 00:36 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 00:36 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 00:34 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 00:34 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 00:33 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 00:33 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 00:32 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 00:32 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 00:32 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2009 * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2009 * 00:15 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2009 * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2009.codfw.wmnet 33.48.192.10.in-addr.arpa 3.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:15 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2009.codfw.wmnet 33.48.192.10.in-addr.arpa 3.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2009 - jasmine@cumin2002" * 00:15 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2009 - jasmine@cumin2002" * 00:10 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 00:03 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2009 * 00:03 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2009.codfw.wmnet with OS trixie == 2026-06-09 == * 22:50 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299640{{!}}HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] (duration: 08m 59s) * 22:45 cscott@deploy1003: cscott: Continuing with deployment * 22:43 cscott@deploy1003: cscott: Backport for [[gerrit:1299640{{!}}HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:41 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299640{{!}}HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] * 22:15 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299639{{!}}[Bug] Donor Badge: Remove client prefs for control group (T428501)]] (duration: 20m 57s) * 22:11 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 22:07 mutante: gerrit - apache httpd log file location moved to /srv/gerrit/site_path/review_site/logs/ [[phab:T425667|T425667]] * 22:06 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on gerrit2003.wikimedia.org with reason: debug * 21:56 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1299639{{!}}[Bug] Donor Badge: Remove client prefs for control group (T428501)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:54 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299639{{!}}[Bug] Donor Badge: Remove client prefs for control group (T428501)]] * 21:52 ryankemper: [[phab:T428241|T428241]] removed retired wdqs2009 full-graph journal dump (446G x2, ~892G) from clouddumps100[1-2]:/srv/dumps/xmldatadumps/public/other/wdqs * 21:49 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299602{{!}}Revert "Create VectorComponentPageToolbar component" (T428649)]] (duration: 08m 16s) * 21:48 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 21:45 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 21:43 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1299602{{!}}Revert "Create VectorComponentPageToolbar component" (T428649)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:41 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299602{{!}}Revert "Create VectorComponentPageToolbar component" (T428649)]] * 21:34 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gerrit1003.wikimedia.org with reason: debug * 21:27 maryum: Deployed security fix for [[phab:T428324|T428324]] * 21:24 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 21:15 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart * 21:06 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart * 20:50 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs2002.codfw.wmnet with OS trixie * 20:50 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299588{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] (duration: 11m 13s) * 20:46 cscott@deploy1003: cscott: Continuing with deployment * 20:43 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs2002.codfw.wmnet with OS trixie * 20:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:42 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:41 cscott@deploy1003: cscott: Backport for [[gerrit:1299588{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:39 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299588{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] * 20:38 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:38 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:33 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:33 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:32 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299454{{!}}wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] (duration: 22m 08s) * 20:28 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:28 cscott@deploy1003: cscott, gkyziridis: Continuing with deployment * 20:24 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2004 * 20:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2004 * 20:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2003 * 20:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2003 * 20:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2002 * 20:13 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2002 * 20:13 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2001 * 20:13 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2001 * 20:12 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:12 cscott@deploy1003: cscott, gkyziridis: Backport for [[gerrit:1299454{{!}}wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:10 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299454{{!}}wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] * 20:09 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:04 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:59 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:54 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:53 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:48 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:47 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:47 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:46 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:46 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:45 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:45 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:28 ryankemper@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts wdqs1015.eqiad.wmnet * 19:28 ryankemper@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:28 ryankemper@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002" * 19:27 ryankemper@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002" * 19:20 ryankemper@cumin2002: START - Cookbook sre.dns.netbox * 19:15 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2008.codfw.wmnet with OS trixie * 19:15 ryankemper@cumin2002: START - Cookbook sre.hosts.decommission for hosts wdqs1015.eqiad.wmnet * 19:12 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync * 19:12 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync * 19:00 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:58 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 18:58 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2008.codfw.wmnet with reason: host reimage * 18:58 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 18:58 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 18:57 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 18:57 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 18:56 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 18:56 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 18:55 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 18:55 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 18:55 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 18:54 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 18:54 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:54 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 18:53 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 18:53 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 18:53 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 18:52 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:52 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2003 to codfw - jhancock@cumin2002" * 18:52 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2003 to codfw - jhancock@cumin2002" * 18:52 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 18:52 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 18:51 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2008.codfw.wmnet with reason: host reimage * 18:51 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 18:51 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 18:51 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 18:50 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 18:50 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 18:47 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:47 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:47 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:46 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:46 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:42 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:42 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:31 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 18:29 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2008.codfw.wmnet with OS trixie * 18:26 jasmine@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2008.codfw.wmnet with OS trixie * 17:48 mutante: https://releases.wikimedia.org {{!}} https://releases-jenkins.wikimedia.org - down for maintenance [[phab:T418299|T418299]] * 17:48 cmooney@dns2005: END - running authdns-update * 17:47 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet with reason: reimage * 17:47 cmooney@dns2005: START - running authdns-update * 17:46 sukhe: sudo cumin 'A:hcaptcha-proxy' 'run-puppet-agent': rolling out CR {{Gerrit|1299427}} [[phab:T428539|T428539]] * 17:43 jayme: kafka-main2008 is down due to hardware failure [[phab:T428654|T428654]] * 17:32 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf1002.eqiad.wmnet with OS trixie * 17:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf1002.eqiad.wmnet with reason: host reimage * 17:06 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf1002.eqiad.wmnet with reason: host reimage * 17:05 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2008 * 17:05 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2008 * 17:04 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 17:04 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2008 * 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2008.codfw.wmnet 4.32.192.10.in-addr.arpa 4.0.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:04 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 17:04 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2008.codfw.wmnet 4.32.192.10.in-addr.arpa 4.0.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2008 - jasmine@cumin2002" * 17:04 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5018 * 17:04 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2008 - jasmine@cumin2002" * 17:03 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5018.eqsin.wmnet with OS trixie * 16:58 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 16:58 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 16:57 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 16:57 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 16:57 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 16:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 16:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 16:50 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf1002.eqiad.wmnet with OS trixie * 16:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 16:47 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf1001.eqiad.wmnet with OS trixie * 16:47 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply * 16:47 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply * 16:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 16:41 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 16:41 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 16:35 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2008 * 16:34 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2008.codfw.wmnet with OS trixie * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:31 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply * 16:30 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply * 16:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf1001.eqiad.wmnet with reason: host reimage * 16:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:26 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf1001.eqiad.wmnet with reason: host reimage * 16:23 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply * 16:22 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: apply * 16:20 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:16 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:15 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:13 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:13 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:12 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf1001.eqiad.wmnet with OS trixie * 16:10 jiji@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. * 16:09 jiji@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. * 16:07 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf2002.codfw.wmnet with OS trixie * 16:02 jiji@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 16:02 jiji@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 16:00 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. * 15:59 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 15:59 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. * 15:59 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 15:59 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. * 15:59 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 15:58 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] DONE helmfile.d/services/termbox: apply * 15:58 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] START helmfile.d/services/termbox: apply * 15:57 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 15:57 jiji@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 15:57 lucaswerkmeister-wmde@deploy1003: helmfile [staging] DONE helmfile.d/services/termbox: apply * 15:56 lucaswerkmeister-wmde@deploy1003: helmfile [staging] START helmfile.d/services/termbox: apply * 15:54 jiji@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 15:53 jiji@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 15:51 jiji@deploy1003: Finished scap sync-world: redeploy {{Gerrit|1299468}} (duration: 07m 23s) * 15:49 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf2002.codfw.wmnet with reason: host reimage * 15:47 jiji@deploy1003: jiji: Continuing with deployment * 15:46 jiji@deploy1003: jiji: redeploy {{Gerrit|1299468}} synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:46 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf2002.codfw.wmnet with reason: host reimage * 15:45 jiji@deploy1003: Started scap sync-world: redeploy {{Gerrit|1299468}} * 15:43 brouberol@cumin1003: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-eqiad * 15:34 brennen@deploy1003: Finished deploy [phabricator/deployment@73e57ce]: deploy phab1004 for [[phab:T410849|T410849]] (followup for robots.txt) (duration: 00m 40s) * 15:33 brennen@deploy1003: Started deploy [phabricator/deployment@73e57ce]: deploy phab1004 for [[phab:T410849|T410849]] (followup for robots.txt) * 15:33 brennen@deploy1003: Finished deploy [phabricator/deployment@73e57ce]: deploy phab2002 for [[phab:T410849|T410849]] (followup for robots.txt) (duration: 00m 45s) * 15:32 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299468{{!}}ProductionServices.php: switch filebackend.php to rdb2015:6381 #2 (T418918 T291916)]] (duration: 07m 21s) * 15:32 brennen@deploy1003: Started deploy [phabricator/deployment@73e57ce]: deploy phab2002 for [[phab:T410849|T410849]] (followup for robots.txt) * 15:28 jiji@deploy1003: Rolling back deployment * 15:27 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf2002.codfw.wmnet with OS trixie * 15:27 jiji@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 15:26 jiji@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 15:25 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1299468{{!}}ProductionServices.php: switch filebackend.php to rdb2015:6381 #2 (T418918 T291916)]] * 15:22 urbanecm: Remove `migrateMentorStatusAwayToCommunityConfiguration` from updatelog on all wikis ([[phab:T409170|T409170]]; the script was only ever run as a dry-run) * 15:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 15:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 15:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf2001.codfw.wmnet with OS trixie * 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@d244a3e]: deploy phab1004 for [[phab:T410849|T410849]] (duration: 00m 42s) * 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@d244a3e]: deploy phab1004 for [[phab:T410849|T410849]] * 15:02 brennen@deploy1003: Finished deploy [phabricator/deployment@d244a3e]: deploy phab2002 for [[phab:T410849|T410849]] (duration: 00m 45s) * 15:01 brennen@deploy1003: Started deploy [phabricator/deployment@d244a3e]: deploy phab2002 for [[phab:T410849|T410849]] * 14:58 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf2001.codfw.wmnet with reason: host reimage * 14:52 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf2001.codfw.wmnet with reason: host reimage * 14:52 arnaudb@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phab[2002-2003].codfw.wmnet,phab[1004-1006].eqiad.wmnet with reason: [[phab:T410849|T410849]] * 14:47 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 14:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 14:40 moritzm: upgrade routinator in codfw to 0.15.2 [[phab:T428456|T428456]] * 14:35 brouberol@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad * 14:33 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf2001.codfw.wmnet with OS trixie * 14:26 brouberol@cumin1003: END (ERROR) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=97) rolling reboot on A:cephosd-eqiad * 14:26 brouberol@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad * 14:20 btullis@cumin1003: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-codfw * 14:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host parsoidtest1001.eqiad.wmnet * 14:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2153: Migration of db2153.codfw.wmnet completed * 14:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of rpki2003.codfw.wmnet to drbd * 14:14 moritzm: imported routinator 0.15.2-1bookworm to thirdparty/routinator for bookworm-wikimedia [[phab:T428456|T428456]] * 14:12 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:12 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1184: Migration of db1184.eqiad.wmnet completed * 14:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host parsoidtest1001.eqiad.wmnet * 14:07 Dreamy_Jazz: Afternoon UTC backport window done * 14:07 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 14:06 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299495{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502{{!}}SecurePollLogPager: Cast user IDs to ints before use (T428599)]] (duration: 06m 53s) * 14:06 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 14:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: rack depool * 14:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of rpki2003.codfw.wmnet to drbd * 14:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow2004.codfw.wmnet to drbd * 14:02 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:02 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1299495{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502{{!}}SecurePollLogPager: Cast user IDs to ints before use (T428599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:59 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1299495{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502{{!}}SecurePollLogPager: Cast user IDs to ints before use (T428599)]] * 13:58 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:58 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:56 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 13:56 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 13:55 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:55 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * {{safesubst:SAL entry|1=13:55 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298929{{!}}Simplify fragment processing (T423700)]], [[gerrit:1298926{{!}}Move ::getFragmentsToTransform() to Content<nowiki>{</nowiki>Text,DOM<nowiki>}</nowiki>TransformStage]], [[gerrit:1298927{{!}}OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925{{!}}Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497}} * 13:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:51 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow2004.codfw.wmnet to drbd * 13:50 cscott@deploy1003: cscott: Continuing with deployment * 13:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2045.codfw.wmnet to cluster codfw and group A * 13:48 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2045.codfw.wmnet to cluster codfw and group A * 13:48 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2027.codfw.wmnet to cluster codfw and group A * 13:47 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2027.codfw.wmnet to cluster codfw and group A * 13:46 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 13:45 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 13:44 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * {{safesubst:SAL entry|1=13:42 cscott@deploy1003: cscott: Backport for [[gerrit:1298929{{!}}Simplify fragment processing (T423700)]], [[gerrit:1298926{{!}}Move ::getFragmentsToTransform() to Content<nowiki>{</nowiki>Text,DOM<nowiki>}</nowiki>TransformStage]], [[gerrit:1298927{{!}}OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925{{!}}Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497{{!}}Store indicators}} * 13:41 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * {{safesubst:SAL entry|1=13:40 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1298929{{!}}Simplify fragment processing (T423700)]], [[gerrit:1298926{{!}}Move ::getFragmentsToTransform() to Content<nowiki>{</nowiki>Text,DOM<nowiki>}</nowiki>TransformStage]], [[gerrit:1298927{{!}}OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925{{!}}Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497{{!}}}} * 13:40 btullis@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-codfw * 13:39 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 13:37 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 13:35 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 13:33 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 13:32 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 13:32 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298834{{!}}config: Disable EmailConfirmationBanner on all wikis (T428291)]] (duration: 07m 01s) * 13:30 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2153: Migration of db2153.codfw.wmnet completed * 13:28 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 lucaswerkmeister-wmde@deploy1003: mmartorana, lucaswerkmeister-wmde: Continuing with deployment * 13:27 lucaswerkmeister-wmde@deploy1003: mmartorana, lucaswerkmeister-wmde: Backport for [[gerrit:1298834{{!}}config: Disable EmailConfirmationBanner on all wikis (T428291)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:26 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1184: Migration of db1184.eqiad.wmnet completed * 13:25 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298834{{!}}config: Disable EmailConfirmationBanner on all wikis (T428291)]] * 13:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 13:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 13:23 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:21 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 13:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2153.codfw.wmnet with OS trixie * 13:20 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2241: rack depool * 13:20 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1237: repool after maintenance db1237 * 13:19 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298654{{!}}Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] (duration: 09m 40s) * 13:17 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2006.codfw.wmnet * 13:17 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2006.codfw.wmnet * 13:16 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2251-2253].codfw.wmnet * 13:16 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2251-2253].codfw.wmnet * 13:16 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2005.codfw.wmnet * 13:16 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2005.codfw.wmnet * 13:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1184.eqiad.wmnet with OS trixie * 13:14 lucaswerkmeister-wmde@deploy1003: neriah, lucaswerkmeister-wmde: Continuing with deployment * 13:11 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 * 13:11 lucaswerkmeister-wmde@deploy1003: neriah, lucaswerkmeister-wmde: Backport for [[gerrit:1298654{{!}}Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:09 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298654{{!}}Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] * 13:04 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:04 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:04 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2153.codfw.wmnet with reason: host reimage * 13:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:03 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1015.eqiad.wmnet with OS trixie * 12:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage * 12:58 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2153.codfw.wmnet with reason: host reimage * 12:57 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1016.eqiad.wmnet with OS trixie * 12:57 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 * 12:56 XioNoX: lsw1-a4-codfw> request system reboot - [[phab:T427357|T427357]] * 12:55 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:53 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage * 12:50 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299477{{!}}hCaptcha: Roll out to all wikis for api account creation. (T426050)]] (duration: 07m 21s) * 12:46 kharlan@deploy1003: kharlan, dbrant: Continuing with deployment * 12:46 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 * 12:45 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1015.eqiad.wmnet with reason: host reimage * 12:45 kharlan@deploy1003: kharlan, dbrant: Backport for [[gerrit:1299477{{!}}hCaptcha: Roll out to all wikis for api account creation. (T426050)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:45 topranks: shut sub-interfaces for row A/B legacy vlans on cr1-codfw [[phab:T427357|T427357]] * 12:45 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 * 12:43 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1299477{{!}}hCaptcha: Roll out to all wikis for api account creation. (T426050)]] * 12:42 topranks: increase OSPF cost on ssw1-a1-codfw link to lsw1-a4-codfw to force traffic via alternate spine [[phab:T427357|T427357]] * 12:41 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299478{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]] (duration: 07m 02s) * 12:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1016.eqiad.wmnet with reason: host reimage * 12:40 moritzm: installing wireshark security updates * 12:40 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2153.codfw.wmnet with OS trixie * 12:38 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1184.eqiad.wmnet with OS trixie * 12:37 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 12:36 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1299478{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:34 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2153: Upgrading db2153.codfw.wmnet * 12:34 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1237: repool after maintenance db1237 * 12:34 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1299478{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]] * 12:34 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2153: Upgrading db2153.codfw.wmnet * 12:34 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1184: Upgrading db1184.eqiad.wmnet * 12:33 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1184: Upgrading db1184.eqiad.wmnet * 12:33 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:32 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1237.eqiad.wmnet with OS trixie * 12:32 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1015.eqiad.wmnet with reason: host reimage * 12:32 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1016.eqiad.wmnet with reason: host reimage * 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 12:27 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2005.codfw.wmnet * 12:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2046: repool after maintenance * 12:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2006.codfw.wmnet * 12:23 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298829{{!}}wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] (duration: 16m 04s) * 12:23 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2006.codfw.wmnet * 12:22 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2251-2253].codfw.wmnet * 12:22 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2005.codfw.wmnet * 12:20 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2251-2253].codfw.wmnet * 12:20 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 12:20 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: rack depool * 12:20 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 12:20 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2241: rack depool * 12:19 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1016 * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1016 * 12:19 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1015 * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1015 * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1016.eqiad.wmnet with OS trixie * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1015.eqiad.wmnet with OS trixie * 12:17 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 * 12:17 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 24 hosts with reason: Rack A4 depool * 12:16 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Continuing with deployment * 12:15 topranks: drain traffic on ssw1-a1-codfw - add gshut community in evpn underlay - [[phab:T427357|T427357]] * 12:14 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 * 12:13 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Backport for [[gerrit:1298829{{!}}wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:10 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1237.eqiad.wmnet with reason: host reimage * 12:07 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1298829{{!}}wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] * 12:05 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1237.eqiad.wmnet with reason: host reimage * 12:00 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Dmaza out of all services on: 2435 hosts * 11:51 atsuko@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 11:51 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie * 11:49 atsuko@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 11:48 atsuko@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 11:47 atsuko@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 11:45 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 11:44 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 11:43 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:43 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2046: repool after maintenance * 11:38 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 11:36 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:36 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2046.codfw.wmnet with OS trixie * 11:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2185.codfw.wmnet with reason: Reimage * 11:31 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging HMonroy out of all services on: 2435 hosts * 11:28 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging KSiebert out of all services on: 2435 hosts * 11:26 slyngs: CAS-SSO upgrade to version 7.3.7.2 * 11:26 slyngshede@dns1004: END - running authdns-update * 11:24 slyngshede@dns1004: START - running authdns-update * 11:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2046.codfw.wmnet with reason: host reimage * 11:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1043: repool after upgrade * 11:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2046.codfw.wmnet with reason: host reimage * 10:55 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2046.codfw.wmnet with OS trixie * 10:53 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2046: Upgrading es2046.codfw.wmnet * 10:53 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2046: Upgrading es2046.codfw.wmnet * 10:52 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply * 10:52 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:52 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply * 10:52 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply * 10:52 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:52 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply * 10:51 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:32 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1043: repool after upgrade * 10:31 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:28 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1160: Repooling * 10:26 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1043.eqiad.wmnet with OS trixie * 10:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:17 elukey: complete rollout of apache2 upgrades * 10:16 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:15 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:13 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:12 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:12 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:08 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1043.eqiad.wmnet with reason: host reimage * 10:04 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:04 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1043.eqiad.wmnet with reason: host reimage * 10:04 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:04 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:04 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:04 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:04 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:57 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1160: Repooling * 09:51 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 09:51 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 09:50 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 09:50 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 09:49 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1043.eqiad.wmnet with OS trixie * 09:48 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool es1043: Upgrading es1043.eqiad.wmnet * 09:48 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 09:47 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:45 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 09:41 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 09:36 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=5 --verbose --last-checked="20260603"` (after stopping previous scan run) * 09:34 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=5 --verbose` (after stopping previous scan run) * 09:27 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 09:26 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 09:17 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 09:17 fceratto@cumin1003: MariaDB change: Setting sections s5 as read-write * 09:17 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 09:14 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1043: Upgrading es1043.eqiad.wmnet * 09:14 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:12 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1042 to es4 eqiad primary [[phab:T428386|T428386]]', diff saved to https://phabricator.wikimedia.org/P93943 and previous config saved to /var/cache/conftool/dbconfig/20260609-091215-marostegui.json * 09:11 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1043 to es4 eqiad primary [[phab:T428386|T428386]]', diff saved to https://phabricator.wikimedia.org/P93942 and previous config saved to /var/cache/conftool/dbconfig/20260609-091147-marostegui.json * 09:03 jiji@cumin1003: conftool action : set/pooled=yes; selector: service=docker-registry,name=registry2005.codfw.wmnet * 08:59 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:59 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:57 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1237.eqiad.wmnet with OS trixie * 08:55 jiji@cumin1003: conftool action : set/pooled=no; selector: service=docker-registry,name=registry2005.codfw.wmnet * 08:55 jiji@cumin1003: conftool action : set/pooled=yes; selector: service=docker-registry,name=registry2004.codfw.wmnet * 08:50 jiji@cumin1003: conftool action : set/pooled=no; selector: service=docker-registry,name=registry2004.codfw.wmnet * 08:22 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=docker-registry,name=codfw * 08:22 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=docker-registry,name=eqiad * 08:08 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=docker-registry,name=eqiad * 08:08 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=docker-registry,name=codfw * 07:59 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:59 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix typoes - ayounsi@cumin1003" * 07:59 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix typoes - ayounsi@cumin1003" * 07:52 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 07:47 brouberol@dns1004: END - running authdns-update * 07:46 brouberol@dns1004: START - running authdns-update * 07:44 brouberol@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/kafka-ui: apply * 07:43 brouberol@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/kafka-ui: apply * 07:43 brouberol@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/kafka-ui: apply * 07:42 brouberol@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/kafka-ui: apply * 07:41 brouberol@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/kafka-ui: apply * 07:39 brouberol@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/kafka-ui: apply * 07:38 brouberol@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:37 brouberol@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 07:37 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie * 07:36 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.major-upgrade (exit_code=97) * 07:36 brouberol@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:36 brouberol@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:26 fceratto@dns1004: END - running authdns-update * 07:24 fceratto@dns1004: START - running authdns-update * 07:22 marostegui@dns1004: END - running authdns-update * 07:21 marostegui@dns1004: START - running authdns-update * 07:19 elukey@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:19 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Fix dse-k8s-wdqs2002 duplicate ipv6 address - elukey@cumin1003" * 07:19 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Fix dse-k8s-wdqs2002 duplicate ipv6 address - elukey@cumin1003" * 07:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1160.eqiad.wmnet with reason: Maintenance * 07:12 elukey@cumin1003: START - Cookbook sre.dns.netbox * 07:11 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1160: Repooling * 07:11 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1160: Repooling * 07:11 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1160: Repooling * 07:11 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1160: Repooling * 07:00 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:00 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1237.eqiad.wmnet with OS trixie * 06:24 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1160 [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93940 and previous config saved to /var/cache/conftool/dbconfig/20260609-062412-fceratto.json * 06:17 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 06:16 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 06:16 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 06:16 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 06:15 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 06:15 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 06:15 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 06:14 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 06:12 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1244 to s4 primary and set section read-write [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93939 and previous config saved to /var/cache/conftool/dbconfig/20260609-061222-fceratto.json * 06:11 fceratto@cumin1003: dbctl commit (dc=all): 'Set s4 eqiad as read-only for maintenance - [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93938 and previous config saved to /var/cache/conftool/dbconfig/20260609-061131-fceratto.json * 06:10 federico3: Starting s4 eqiad failover from db1160 to db1244 - [[phab:T426086|T426086]] * 06:01 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1244 with weight 0 [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93937 and previous config saved to /var/cache/conftool/dbconfig/20260609-060121-fceratto.json * 06:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 40 hosts with reason: Primary switchover s4 [[phab:T426086|T426086]] * 05:40 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie * 05:37 marostegui@dns1004: START - running authdns-update * 05:27 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1237: Upgrading db1237.eqiad.wmnet * 05:27 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1237: Upgrading db1237.eqiad.wmnet * 05:27 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:24 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1237 [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93935 and previous config saved to /var/cache/conftool/dbconfig/20260609-052420-marostegui.json * 05:23 marostegui@dns1004: START - running authdns-update * 05:23 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93934 and previous config saved to /var/cache/conftool/dbconfig/20260609-052311-marostegui.json * 05:22 marostegui@cumin1003: dbctl commit (dc=all): 'Set x1 eqiad as read-only for maintenance - [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93933 and previous config saved to /var/cache/conftool/dbconfig/20260609-052253-marostegui.json * 05:22 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T428158|T428158]] * 05:19 marostegui@cumin1003: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93932 and previous config saved to /var/cache/conftool/dbconfig/20260609-051859-marostegui.json * 05:18 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: Primary switchover x1 [[phab:T428158|T428158]] * 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.3 (duration: 02m 43s) * 03:40 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] (duration: 37m 16s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 38s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-06-08 == * 22:00 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298915{{!}}CommonSettings: Set $wgScoreSafeMode = false (T428484)]] (duration: 07m 42s) * 21:56 reedy@deploy1003: reedy: Continuing with deployment * 21:54 reedy@deploy1003: reedy: Backport for [[gerrit:1298915{{!}}CommonSettings: Set $wgScoreSafeMode = false (T428484)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:53 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1298915{{!}}CommonSettings: Set $wgScoreSafeMode = false (T428484)]] * 21:12 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298891{{!}}OOUIHTMLForm: Avoid treating form header as a clickable label (T428359)]] (duration: 08m 10s) * 21:07 mlitn@deploy1003: mlitn, neriah: Continuing with deployment * 21:05 mlitn@deploy1003: mlitn, neriah: Backport for [[gerrit:1298891{{!}}OOUIHTMLForm: Avoid treating form header as a clickable label (T428359)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:03 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1298891{{!}}OOUIHTMLForm: Avoid treating form header as a clickable label (T428359)]] * 20:43 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297162{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias]], [[gerrit:1298841{{!}}Squashed diff to master]] (duration: 07m 05s) * 20:39 mlitn@deploy1003: mlitn: Continuing with deployment * 20:38 mlitn@deploy1003: mlitn: Backport for [[gerrit:1297162{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias]], [[gerrit:1298841{{!}}Squashed diff to master]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:36 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1297162{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias]], [[gerrit:1298841{{!}}Squashed diff to master]] * 20:29 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298390{{!}}English Wikibooks: update FlaggedRevs configuration (T428329)]], [[gerrit:1298328{{!}}English Wikiversity: Add new user group "autopatrolled" (T428269)]] (duration: 08m 58s) * 20:25 mlitn@deploy1003: mlitn, vadymts1: Continuing with deployment * 20:22 mlitn@deploy1003: mlitn, vadymts1: Backport for [[gerrit:1298390{{!}}English Wikibooks: update FlaggedRevs configuration (T428329)]], [[gerrit:1298328{{!}}English Wikiversity: Add new user group "autopatrolled" (T428269)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1298390{{!}}English Wikibooks: update FlaggedRevs configuration (T428329)]], [[gerrit:1298328{{!}}English Wikiversity: Add new user group "autopatrolled" (T428269)]] * 20:03 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298879{{!}}SimpleCaptcha: Re-render captcha when edit form is redisplayed (T428437)]] (duration: 37m 43s) * 19:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:31 kharlan@deploy1003: kharlan: Continuing with deployment * 19:30 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:30 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:29 kharlan@deploy1003: kharlan: Backport for [[gerrit:1298879{{!}}SimpleCaptcha: Re-render captcha when edit form is redisplayed (T428437)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 19:28 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:27 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:25 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1298879{{!}}SimpleCaptcha: Re-render captcha when edit form is redisplayed (T428437)]] * 19:24 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab (duration: 01m 32s) * 19:23 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:22 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab * 19:20 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab (duration: 01m 40s) * 19:19 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab * 19:16 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:14 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:06 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:59 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2004 * 18:52 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2004 * 18:52 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2003 * 18:52 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2003 * 18:51 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:51 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2004 to codfw - jhancock@cumin2002" * 18:51 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2004 to codfw - jhancock@cumin2002" * 18:44 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:42 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:42 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2030 to codfw - jhancock@cumin2002" * 18:42 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2030 to codfw - jhancock@cumin2002" * 18:37 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:33 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2002 * 18:32 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2002 * 18:31 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:31 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2002 to codfw - jhancock@cumin2002" * 18:31 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2002 to codfw - jhancock@cumin2002" * 18:25 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:22 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2001 * 18:22 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2001 * 18:21 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:21 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: updating dse-k8s-wdqs2001 to codfw - jhancock@cumin2002" * 18:21 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: updating dse-k8s-wdqs2001 to codfw - jhancock@cumin2002" * 18:17 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:02 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T427286|T427286]] (duration: 00m 12s) * 18:02 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T427286|T427286]] * 17:37 jnuche@deploy1003: Installation of scap version "4.268.0" completed for 2 hosts * 17:35 jnuche@deploy1003: Installing scap version "4.268.0" for 2 host(s) * 17:21 claime: restarting varnish-frontend service on cp6012 * 17:21 claime: restarting varnish-frontend service on cp6011 * 17:21 claime: restarted varnish-frontend service on cp6009 * 17:13 taavi: bounce sirenbot to get it to re-join a channel * 17:05 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 17:05 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:58 urbanecm@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 16:57 urbanecm@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 16:55 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 16:53 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 16:53 urbanecm@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 16:52 urbanecm@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 16:30 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 16:29 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 16:29 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 16:28 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 16:28 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 16:28 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 16:28 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 16:27 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 16:27 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 16:26 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 16:26 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 16:25 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 16:18 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 16:17 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 16:17 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 16:16 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 16:16 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 16:16 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 16:16 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 16:15 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 16:14 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 16:14 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 16:14 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 16:14 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 16:13 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 16:13 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 16:13 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 16:12 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 16:12 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 16:10 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 16:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 16:10 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 16:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 16:10 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 16:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 16:09 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 16:08 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 16:08 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 16:07 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 16:06 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 15:57 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2042: repool after upgrade * 15:45 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db[2183-2184].codfw.wmnet * 15:45 jynus@cumin2002: START - Cookbook sre.hosts.remove-downtime for db[2183-2184].codfw.wmnet * 15:18 jynus: dbmaint on backup1-codfw@codfw ([[phab:T428467|T428467]]) * 15:12 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2042: repool after upgrade * 15:12 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 15:09 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 15:09 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 15:09 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 15:07 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2042.codfw.wmnet with OS trixie * 15:04 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 15:04 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 15:03 jynus@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db[2183-2184].codfw.wmnet with reason: Switchover db * 15:03 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 15:03 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 15:02 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 15:01 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply * 15:00 eevans@deploy1003: helmfile [staging] START helmfile.d/services/data-gateway: apply * 14:59 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:55 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:55 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:54 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:50 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:50 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:50 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:49 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:49 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2042.codfw.wmnet with reason: host reimage * 14:42 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2042.codfw.wmnet with reason: host reimage * 14:32 Lucas_WMDE: UTC afternoon backport+config window done * 14:32 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298709{{!}}Add translatable messages for WikiProject names (T427804)]], [[gerrit:1298710{{!}}Use translatable messages for WikiProject links (T427804)]], [[gerrit:1297644{{!}}WikiProject links - remove 'text' config (T427804)]] (duration: 31m 57s) * 14:27 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 14:26 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2042.codfw.wmnet with OS trixie * 14:26 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 14:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2042: Upgrading es2042.codfw.wmnet * 14:25 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2042: Upgrading es2042.codfw.wmnet * 14:25 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:24 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2043 to es4 codfw primary [[phab:T428386|T428386]]', diff saved to https://phabricator.wikimedia.org/P93926 and previous config saved to /var/cache/conftool/dbconfig/20260608-142423-marostegui.json * 14:23 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 14:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1041: repool after maintenance * 14:19 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Continuing with deployment * 14:18 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Backport for [[gerrit:1298709{{!}}Add translatable messages for WikiProject names (T427804)]], [[gerrit:1298710{{!}}Use translatable messages for WikiProject links (T427804)]], [[gerrit:1297644{{!}}WikiProject links - remove 'text' config (T427804)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:11 cgoubert@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=liftwing-openapi-server.* * 14:10 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp6013.* * 14:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:05 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 14:05 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 13:54 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 13:52 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 13:50 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 13:50 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 13:50 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296550{{!}}hCaptcha: Don't show AbuseFilter CAPTCHA for wbsetclaim API (T427608)]] (duration: 08m 31s) * 13:48 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 13:46 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 13:43 cgoubert@dns1004: END - running authdns-update * 13:43 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296550{{!}}hCaptcha: Don't show AbuseFilter CAPTCHA for wbsetclaim API (T427608)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:41 cgoubert@dns1004: START - running authdns-update * 13:41 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296550{{!}}hCaptcha: Don't show AbuseFilter CAPTCHA for wbsetclaim API (T427608)]] * 13:39 urbanecm@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * {{safesubst:SAL entry|1=13:38 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298758{{!}}feat(V2): toggle experiment features based on custom url override (T424646)]], [[gerrit:1298762{{!}}specialCreateAccount: use GECreateAccountExperimentV2 instead of hook (T424646)]], [[gerrit:1298764{{!}}fix: correctly read experiments param on Special:UserLogin]], [[gerrit:1298765{{!}}signup.js: use JS var instead of TestKitchen to show exp}} * 13:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1041: repool after maintenance * 13:38 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 13:38 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:37 urbanecm@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 13:36 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 13:35 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1041.eqiad.wmnet with OS trixie * 13:34 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 13:34 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2041: repool after upgrade * 13:34 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Continuing with deployment * 13:34 urbanecm@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 13:32 urbanecm@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * {{safesubst:SAL entry|1=13:30 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Backport for [[gerrit:1298758{{!}}feat(V2): toggle experiment features based on custom url override (T424646)]], [[gerrit:1298762{{!}}specialCreateAccount: use GECreateAccountExperimentV2 instead of hook (T424646)]], [[gerrit:1298764{{!}}fix: correctly read experiments param on Special:UserLogin]], [[gerrit:1298765{{!}}signup.js: use JS var instead of TestKitchen to show}} * {{safesubst:SAL entry|1=13:29 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298758{{!}}feat(V2): toggle experiment features based on custom url override (T424646)]], [[gerrit:1298762{{!}}specialCreateAccount: use GECreateAccountExperimentV2 instead of hook (T424646)]], [[gerrit:1298764{{!}}fix: correctly read experiments param on Special:UserLogin]], [[gerrit:1298765{{!}}signup.js: use JS var instead of TestKitchen to show expe}} * 13:21 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298418{{!}}NewUserMessage: Add $wgNewUserMessageOnAutoCreateFirstEdit (T426206)]], [[gerrit:1298717{{!}}Replace NewUserMessageOnAutoCreateFirstEdit with wgNewUserMessageOnFirstEdit (T426206)]], [[gerrit:1298734{{!}}Enable wgNewUserMessageOnFirstEdit on incubatorwiki (T426206)]] (duration: 11m 06s) * 13:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1041.eqiad.wmnet with reason: host reimage * 13:17 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Continuing with deployment * 13:12 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 13:12 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Backport for [[gerrit:1298418{{!}}NewUserMessage: Add $wgNewUserMessageOnAutoCreateFirstEdit (T426206)]], [[gerrit:1298717{{!}}Replace NewUserMessageOnAutoCreateFirstEdit with wgNewUserMessageOnFirstEdit (T426206)]], [[gerrit:1298734{{!}}Enable wgNewUserMessageOnFirstEdit on incubatorwiki (T426206)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki * 13:12 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 13:12 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1041.eqiad.wmnet with reason: host reimage * 13:11 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 13:11 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 13:10 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298418{{!}}NewUserMessage: Add $wgNewUserMessageOnAutoCreateFirstEdit (T426206)]], [[gerrit:1298717{{!}}Replace NewUserMessageOnAutoCreateFirstEdit with wgNewUserMessageOnFirstEdit (T426206)]], [[gerrit:1298734{{!}}Enable wgNewUserMessageOnFirstEdit on incubatorwiki (T426206)]] * 12:57 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298767{{!}}Follow-up: Allow CaptchaConsequence to be skipped via hook (T427608)]] (duration: 06m 20s) * 12:57 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1041.eqiad.wmnet with OS trixie * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:56 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1041: Upgrading es1041.eqiad.wmnet * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:55 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1041: Upgrading es1041.eqiad.wmnet * 12:55 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:54 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:53 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 12:53 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1298767{{!}}Follow-up: Allow CaptchaConsequence to be skipped via hook (T427608)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:51 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:51 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1298767{{!}}Follow-up: Allow CaptchaConsequence to be skipped via hook (T427608)]] * 12:49 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:49 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2041: repool after upgrade * 12:49 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:47 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:46 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 12:44 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 12:43 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 12:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 12:41 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 12:40 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2063.codfw.wmnet with OS bullseye * 12:32 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2062.codfw.wmnet with OS bullseye * 12:27 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2041.codfw.wmnet with OS trixie * 12:21 joal@deploy1003: Finished deploy [analytics/refinery@d67c584] (thin): Regular analytics weekly train THIN [analytics/refinery@d67c584f] (duration: 02m 00s) * 12:19 joal@deploy1003: Started deploy [analytics/refinery@d67c584] (thin): Regular analytics weekly train THIN [analytics/refinery@d67c584f] * 12:19 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2063.codfw.wmnet with reason: host reimage * 12:18 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 12:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 12:16 joal@deploy1003: Finished deploy [analytics/refinery@d67c584]: Regular analytics weekly train [analytics/refinery@d67c584f] (duration: 07m 52s) * 12:15 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2063.codfw.wmnet with reason: host reimage * 12:13 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2062.codfw.wmnet with reason: host reimage * 12:09 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2041.codfw.wmnet with reason: host reimage * 12:08 joal@deploy1003: Started deploy [analytics/refinery@d67c584]: Regular analytics weekly train [analytics/refinery@d67c584f] * 12:08 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2062.codfw.wmnet with reason: host reimage * 12:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add eqiad e8 public vlans - ayounsi@cumin1003" * 12:06 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add eqiad e8 public vlans - ayounsi@cumin1003" * 12:03 joal@deploy1003: Finished deploy [analytics/refinery@d67c584] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@d67c584f] (duration: 02m 00s) * 12:03 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2041.codfw.wmnet with reason: host reimage * 12:01 joal@deploy1003: Started deploy [analytics/refinery@d67c584] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@d67c584f] * 12:01 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 12:00 ayounsi@cumin1003: END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) * 12:00 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 12:00 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 12:00 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:57 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2063 * 11:57 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2063 * 11:57 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2063 * 11:57 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2063.codfw.wmnet 52.16.192.10.in-addr.arpa 2.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:56 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2063.codfw.wmnet 52.16.192.10.in-addr.arpa 2.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:56 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:56 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2063 - mvernon@cumin2002" * 11:56 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2063 - mvernon@cumin2002" * 11:51 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:51 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2063 * 11:50 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2063.codfw.wmnet with OS bullseye * 11:50 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2062 * 11:50 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2062 * 11:49 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2062 * 11:49 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2062.codfw.wmnet 123.0.192.10.in-addr.arpa 3.2.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:49 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2062.codfw.wmnet 123.0.192.10.in-addr.arpa 3.2.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:49 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:49 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2062 - mvernon@cumin2002" * 11:49 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2062 - mvernon@cumin2002" * 11:47 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2041.codfw.wmnet with OS trixie * 11:45 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2041: Upgrading es2041.codfw.wmnet * 11:45 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2041: Upgrading es2041.codfw.wmnet * 11:44 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:44 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.major-upgrade (exit_code=97) * 11:44 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:44 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1042: repool after maintenance * 11:43 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:43 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2062 * 11:42 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2062.codfw.wmnet with OS bullseye * 11:30 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298728{{!}}SpecialMediaSearch: Prefer thumb steps over thumb limits (T424032)]] (duration: 17m 39s) * 11:25 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 11:18 Raine: progressively switching shellbox to bookworm (start) * 11:15 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 11:14 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 11:14 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1298728{{!}}SpecialMediaSearch: Prefer thumb steps over thumb limits (T424032)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:13 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 11:12 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 11:12 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1298728{{!}}SpecialMediaSearch: Prefer thumb steps over thumb limits (T424032)]] * 11:02 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be2062 * 11:02 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be2063 * 10:58 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1042: repool after maintenance * 10:58 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:56 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1042.eqiad.wmnet with OS trixie * 10:47 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298721{{!}}GuessedThumbnailInfo: Also allow showing webp originals (T428202)]] (duration: 16m 41s) * 10:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1042.eqiad.wmnet with reason: host reimage * 10:39 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 10:39 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 10:38 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 10:36 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2160.codfw.wmnet * 10:36 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2160.codfw.wmnet * 10:35 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2043: repool after upgrade * 10:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2160.codfw.wmnet with reason: Reboot * 10:34 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1298721{{!}}GuessedThumbnailInfo: Also allow showing webp originals (T428202)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:34 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1042.eqiad.wmnet with reason: host reimage * 10:30 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1298721{{!}}GuessedThumbnailInfo: Also allow showing webp originals (T428202)]] * 10:18 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1042.eqiad.wmnet with OS trixie * 10:18 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:18 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:18 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:18 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:15 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1042: Upgrading es1042.eqiad.wmnet * 10:14 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:14 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:14 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:14 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:13 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1042: Upgrading es1042.eqiad.wmnet * 10:13 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:12 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be2063 * 10:09 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be2062 * 10:07 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:07 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:07 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:06 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 09:52 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply * 09:52 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply * 09:50 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply * 09:49 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply * 09:49 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2043: repool after upgrade * 09:49 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2043.codfw.wmnet with OS trixie * 09:44 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 09:44 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 09:42 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: sync * 09:42 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: sync * 09:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2043.codfw.wmnet with reason: host reimage * 09:27 jelto@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org * 09:23 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2043.codfw.wmnet with reason: host reimage * 09:17 jelto@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org * 09:15 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: sync * 09:15 ozge@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: sync * 09:07 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2043.codfw.wmnet with OS trixie * 09:06 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2043: Upgrading es2043.codfw.wmnet * 09:06 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2043: Upgrading es2043.codfw.wmnet * 09:05 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:41 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1217.eqiad.wmnet with OS trixie * 08:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1217.eqiad.wmnet with reason: host reimage * 08:15 taavi@cumin1003: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database urwikisource ([[phab:T415977|T415977]]) * 08:14 taavi@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database urwikisource ([[phab:T415977|T415977]]) * 08:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1217.eqiad.wmnet with reason: host reimage * 08:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2052: repool after upgrade * 08:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1051: repool after maintenance * 08:03 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Managing sanitization for wikis urwikisource in section s5 * 07:55 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1217.eqiad.wmnet with OS trixie * 07:53 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1217.eqiad.wmnet with reason: reimage * 07:53 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis urwikisource in section s5 * 07:52 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Checking sanitization for wikis urwikisource in section s5 * 07:50 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis urwikisource in section s5 * 07:50 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.sanitize-wiki (exit_code=97) Managing sanitization for wikis urwikisource in section s5 * 07:50 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis urwikisource in section s5 * 07:44 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297681{{!}}Global rollout - Sub-ref deployments to Group 0, Group 1 and frwiki (T425662)]] (duration: 32m 51s) * 07:32 wmde-fisch@deploy1003: wmde-fisch, lilients: Continuing with deployment * 07:29 wmde-fisch@deploy1003: wmde-fisch, lilients: Backport for [[gerrit:1297681{{!}}Global rollout - Sub-ref deployments to Group 0, Group 1 and frwiki (T425662)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:21 elukey: upgrade sudo package on an-* hosts for [[phab:T428384|T428384]] * 07:18 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2052: repool after upgrade * 07:18 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1051: repool after maintenance * 07:17 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:17 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:12 taavi@cumin1003: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database urwikisource ([[phab:T415977|T415977]]) * 07:12 elukey: upgrade exim4 packages on seaborgium for security upgrades * 07:11 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1297681{{!}}Global rollout - Sub-ref deployments to Group 0, Group 1 and frwiki (T425662)]] * 06:36 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1051.eqiad.wmnet with OS trixie * 06:20 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1051.eqiad.wmnet with reason: host reimage * 06:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1051.eqiad.wmnet with reason: host reimage * 06:15 taavi@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database urwikisource ([[phab:T415977|T415977]]) * 05:58 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1051.eqiad.wmnet with OS trixie * 05:54 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2052.codfw.wmnet with OS trixie * 05:44 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool es1051: Upgrading es1051.eqiad.wmnet * 05:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2052.codfw.wmnet with reason: host reimage * 05:35 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2052.codfw.wmnet with reason: host reimage * 05:35 marostegui@dns1004: END - running authdns-update * 05:34 marostegui@dns1004: START - running authdns-update * 05:33 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1051: Upgrading es1051.eqiad.wmnet * 05:33 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:31 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1054 to es3 eqiad primary [[phab:T428050|T428050]]', diff saved to https://phabricator.wikimedia.org/P93895 and previous config saved to /var/cache/conftool/dbconfig/20260608-053156-marostegui.json * 05:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2052.codfw.wmnet with OS trixie * 05:18 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2052: Upgrading es2052.codfw.wmnet * 05:18 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2052: Upgrading es2052.codfw.wmnet * 05:18 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade == 2026-06-07 == * 16:32 elukey: `elukey@cumin1003:~$ sudo cumin 'cp6* and not cp6014* and not cp6010*' "varnish-frontend-restart" -b 1` * 16:29 elukey: restart varnish-frontend on cp6014 == 2026-06-06 == * 09:07 ammarpad@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=hewiki --logwiki=metawiki W.Mechelke Tungsten_Mechelke # [[phab:T428182|T428182]] == 2026-06-05 == * 22:16 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 21:01 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=10 --verbose` (after stopping the other commons scan) * 20:56 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=30 --verbose` (after stopping the other commons scan) * 20:20 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290093{{!}}Enable wmgUseUrlShortenerLegacy on test2wiki (T107188)]] (duration: 10m 02s) * 20:16 krinkle@deploy1003: krinkle: Continuing with deployment * 20:12 krinkle@deploy1003: krinkle: Backport for [[gerrit:1290093{{!}}Enable wmgUseUrlShortenerLegacy on test2wiki (T107188)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:10 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1290093{{!}}Enable wmgUseUrlShortenerLegacy on test2wiki (T107188)]] * 16:45 jgreen@dns1004: END - running authdns-update * 16:44 jgreen@dns1004: START - running authdns-update * 16:17 dzahn@dns1005: END - running authdns-update * 16:17 mutante: DNS - adding new project language "mag" - Magahi - a language spoken in India and Nepal by about 12 million native speakers ([[phab:T428266|T428266]]) * 16:16 dzahn@dns1005: START - running authdns-update * 14:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:38 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:37 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 12:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:30 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:30 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 12:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2202.codfw.wmnet with reason: Reboot * 12:28 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:28 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 12:08 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:07 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 12:07 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:06 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 11:29 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 11:28 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:55 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:54 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:31 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1054: repool after upgrade * 08:08 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 08:07 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 08:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 08:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/kafka-ui: apply * 07:39 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1054: repool after upgrade * 07:38 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 07:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 07:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/kafka-ui: apply * 07:07 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 06:01 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1054.eqiad.wmnet with OS trixie * 05:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1054.eqiad.wmnet with reason: host reimage * 05:37 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1054.eqiad.wmnet with reason: host reimage * 05:22 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1054.eqiad.wmnet with OS trixie * 05:21 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1054: Upgrading es1054.eqiad.wmnet * 05:21 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1054: Upgrading es1054.eqiad.wmnet * 05:20 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 01:55 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1010.eqiad.wmnet with OS trixie * 01:39 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: host reimage * 01:32 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: host reimage * 01:16 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1010.eqiad.wmnet with OS trixie * 00:56 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1007.eqiad.wmnet with OS trixie * 00:40 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1007.eqiad.wmnet with reason: host reimage * 00:33 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1007.eqiad.wmnet with reason: host reimage * 00:17 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1007.eqiad.wmnet with OS trixie * 00:02 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297268{{!}}Redirect unknown wikinews languages to portal (T427126)]] (duration: 07m 02s) == 2026-06-04 == * 23:57 ladsgroup@deploy1003: ladsgroup, pppery: Continuing with deployment * 23:57 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1006.eqiad.wmnet with OS trixie * 23:57 ladsgroup@deploy1003: ladsgroup, pppery: Backport for [[gerrit:1297268{{!}}Redirect unknown wikinews languages to portal (T427126)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:55 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1297268{{!}}Redirect unknown wikinews languages to portal (T427126)]] * 23:40 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1006.eqiad.wmnet with reason: host reimage * 23:36 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1006.eqiad.wmnet with reason: host reimage * 23:20 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1006.eqiad.wmnet with OS trixie * 21:28 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host releases1003.eqiad.wmnet with OS trixie * 21:04 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases1003.eqiad.wmnet with reason: host reimage * 20:58 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on releases1003.eqiad.wmnet with reason: host reimage * 20:50 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5030.* * 20:42 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host releases1003.eqiad.wmnet with OS trixie * 20:27 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1100.eqiad.wmnet,service=(cdn{{!}}ats-be) * 20:26 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6013.drmrs.wmnet,service=(cdn{{!}}ats-be) * 20:20 brett@dns1006: END - running authdns-update * 20:19 brett@dns1006: START - running authdns-update * 20:18 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5030.eqsin.wmnet with OS trixie * 20:10 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296015{{!}}Deploy PRV to 6 wikis (T427851)]] (duration: 07m 39s) * 20:08 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist group2.dblist extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` * 20:06 arlolra@deploy1003: arlolra: Continuing with deployment * 20:04 arlolra@deploy1003: arlolra: Backport for [[gerrit:1296015{{!}}Deploy PRV to 6 wikis (T427851)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:02 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1296015{{!}}Deploy PRV to 6 wikis (T427851)]] * 19:49 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5030.eqsin.wmnet with reason: host reimage * 19:43 cmooney@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5030.eqsin.wmnet with reason: host reimage * 19:15 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5030 * 19:15 cmooney@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5030 * 19:14 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cp5030 * 19:14 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5030.eqsin.wmnet 27.0.132.10.in-addr.arpa 7.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:14 cmooney@cumin1003: START - Cookbook sre.dns.wipe-cache cp5030.eqsin.wmnet 27.0.132.10.in-addr.arpa 7.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:14 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:14 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5030 - cmooney@cumin1003" * 19:13 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5030 - cmooney@cumin1003" * 19:09 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 19:08 cmooney@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5030 * 19:08 cmooney@cumin1003: START - Cookbook sre.hosts.reimage for host cp5030.eqsin.wmnet with OS trixie * 18:51 cmooney@dns2005: END - running authdns-update * 18:50 cmooney@dns2005: START - running authdns-update * 18:43 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:42 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove IPs that had been used for eqsin cr links - cmooney@cumin1003" * 18:40 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove IPs that had been used for eqsin cr links - cmooney@cumin1003" * 18:37 sukhe: sukhe@cp6013:~$ sudo traffic_server -C clear_cache * 18:36 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 18:08 dancy@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.5 refs [[phab:T423914|T423914]] * 17:17 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297751{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297752{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] (duration: 06m 40s) * 17:13 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 17:13 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297751{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297752{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:11 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297751{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297752{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] * 16:55 topranks: shift traffic off cr1-esams et-1/0/1 link to asw1-by27-esams [[phab:T427056|T427056]] * 16:45 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297741{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297742{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] (duration: 13m 58s) * 16:41 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 16:33 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297741{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297742{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:31 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297741{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297742{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] * 16:17 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 16:03 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297740{{!}}hCaptcha: Move ConfirmEditCaptchaClass hook inside hCaptcha block (T428183)]] (duration: 10m 21s) * 16:03 elukey: uploaded spicerack_12.7.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia * 15:59 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 15:55 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297740{{!}}hCaptcha: Move ConfirmEditCaptchaClass hook inside hCaptcha block (T428183)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:53 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297740{{!}}hCaptcha: Move ConfirmEditCaptchaClass hook inside hCaptcha block (T428183)]] * 15:44 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5030.* * 15:41 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2007.codfw.wmnet with OS trixie * 15:39 ladsgroup@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0) * 15:28 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views * 15:24 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297730{{!}}ptwiki: Disable Article Guidance experiment (T426871)]] (duration: 07m 26s) * 15:24 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2007.codfw.wmnet with reason: host reimage * 15:20 sbisson@deploy1003: sbisson: Continuing with deployment * 15:19 sbisson@deploy1003: sbisson: Backport for [[gerrit:1297730{{!}}ptwiki: Disable Article Guidance experiment (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:19 jayme@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2007.codfw.wmnet with reason: host reimage * 15:17 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1297730{{!}}ptwiki: Disable Article Guidance experiment (T426871)]] * 15:13 ladsgroup@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0) * 15:06 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297724{{!}}Revert "Start reading from new file tables on commons"]] (duration: 07m 00s) * 15:05 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views * 15:02 zabe@deploy1003: zabe: Continuing with deployment * 15:01 zabe@deploy1003: zabe: Backport for [[gerrit:1297724{{!}}Revert "Start reading from new file tables on commons"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:59 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1297724{{!}}Revert "Start reading from new file tables on commons"]] * 14:57 zabe@deploy1003: Finished scap sync-world: [[phab:T416548|T416548]] (duration: 05m 10s) * 14:56 jayme@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-main2007.codfw.wmnet with OS trixie * 14:52 zabe@deploy1003: Started scap sync-world: [[phab:T416548|T416548]] * 14:50 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:49 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:43 zabe@deploy1003: sync-world aborted: Backport for [[gerrit:1270513{{!}}Start reading from new file tables on commons (T416548)]] (duration: 03m 58s) * 14:43 zabe@deploy1003: zabe: Continuing with deployment * 14:41 zabe@deploy1003: zabe: Backport for [[gerrit:1270513{{!}}Start reading from new file tables on commons (T416548)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:40 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f1-codfw * 14:40 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device lsw1-f1-codfw * 14:39 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1270513{{!}}Start reading from new file tables on commons (T416548)]] * 14:36 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297711{{!}}hCaptcha: Enable for MobileFrontend in some Group 2 wikis (T425940)]] (duration: 08m 20s) * 14:32 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:30 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297711{{!}}hCaptcha: Enable for MobileFrontend in some Group 2 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:29 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1057: repool after upgrade * 14:28 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297711{{!}}hCaptcha: Enable for MobileFrontend in some Group 2 wikis (T425940)]] * 14:20 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 14:16 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:15 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply * 14:15 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:15 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply * 14:13 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297704{{!}}Use the globalblock-local-status right over globalblock-whitelist (T277942)]], [[gerrit:1296620{{!}}core-Permissions: Stop assigning unused globalblock-whitelist right (T277942)]] (duration: 06m 46s) * 14:10 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 14:08 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:08 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297704{{!}}Use the globalblock-local-status right over globalblock-whitelist (T277942)]], [[gerrit:1296620{{!}}core-Permissions: Stop assigning unused globalblock-whitelist right (T277942)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:07 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:06 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply * 14:06 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297704{{!}}Use the globalblock-local-status right over globalblock-whitelist (T277942)]], [[gerrit:1296620{{!}}core-Permissions: Stop assigning unused globalblock-whitelist right (T277942)]] * 14:06 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:06 tappof: bump space for prometheus k8s-aux in eqiad * 14:05 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply * 14:05 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:04 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply * 13:56 _joe_: transferred requestctl api tokens for all ops to the db ([[phab:T428119|T428119]]) * 13:56 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2050 to es3 codfw primary [[phab:T428050|T428050]]', diff saved to https://phabricator.wikimedia.org/P93878 and previous config saved to /var/cache/conftool/dbconfig/20260604-135631-marostegui.json * 13:56 Dreamy_Jazz: Afternoon UTC backport window done * 13:54 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297700{{!}}Revert "hCaptcha: Provide always challenge sitekey for account creation"]] (duration: 13m 38s) * 13:51 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 13:50 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 13:47 sukhe: sukhe@cp6011:~$ sudo -i varnish-frontend-restart * 13:44 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1057: repool after upgrade * 13:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:43 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297700{{!}}Revert "hCaptcha: Provide always challenge sitekey for account creation"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:41 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1057.eqiad.wmnet with OS trixie * 13:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297700{{!}}Revert "hCaptcha: Provide always challenge sitekey for account creation"]] * 13:38 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297692{{!}}hCaptcha: Provide always challenge sitekey for account creation (T421041)]] (duration: 05m 27s) * 13:38 dreamyjazz@deploy1003: dreamyjazz: Rolling back deployment * 13:36 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: down * 13:35 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297692{{!}}hCaptcha: Provide always challenge sitekey for account creation (T421041)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:33 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297692{{!}}hCaptcha: Provide always challenge sitekey for account creation (T421041)]] * 13:31 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295978{{!}}Update config for WikiProjects linking prototype (T427804)]] (duration: 17m 13s) * 13:26 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Continuing with deployment * 13:25 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1057.eqiad.wmnet with reason: host reimage * 13:17 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1057.eqiad.wmnet with reason: host reimage * 13:16 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Backport for [[gerrit:1295978{{!}}Update config for WikiProjects linking prototype (T427804)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:14 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1295978{{!}}Update config for WikiProjects linking prototype (T427804)]] * 13:13 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:13 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1220: Migration of db1220.eqiad.wmnet completed * 13:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: down * 13:12 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1224', diff saved to https://phabricator.wikimedia.org/P93875 and previous config saved to /var/cache/conftool/dbconfig/20260604-131219-marostegui.json * 13:00 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1057.eqiad.wmnet with OS trixie * 13:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1057: Upgrading es1057.eqiad.wmnet * 12:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1057: Upgrading es1057.eqiad.wmnet * 12:59 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:56 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296557{{!}}wmf-config: Skip CAPTCHA for action=mcrundo (T427612)]] (duration: 08m 30s) * 12:52 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Continuing with deployment * 12:50 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Backport for [[gerrit:1296557{{!}}wmf-config: Skip CAPTCHA for action=mcrundo (T427612)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2050: repool after upgrade * 12:48 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296557{{!}}wmf-config: Skip CAPTCHA for action=mcrundo (T427612)]] * 12:37 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 12:37 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 12:28 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1220: Migration of db1220.eqiad.wmnet completed * 12:20 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1220.eqiad.wmnet with OS trixie * 12:04 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2050: repool after upgrade * 12:04 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 12:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1220.eqiad.wmnet with reason: host reimage * 11:59 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1220.eqiad.wmnet with reason: host reimage * 11:42 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1220.eqiad.wmnet with OS trixie * 11:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2050.codfw.wmnet with OS trixie * 11:40 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1220: Upgrading db1220.eqiad.wmnet * 11:37 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1220: Upgrading db1220.eqiad.wmnet * 11:36 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1179: Migration of db1179.eqiad.wmnet completed * 11:23 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2050.codfw.wmnet with reason: host reimage * 11:16 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2050.codfw.wmnet with reason: host reimage * 11:00 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2050.codfw.wmnet with OS trixie * 11:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2050: Upgrading es2050.codfw.wmnet * 10:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2050: Upgrading es2050.codfw.wmnet * 10:59 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:59 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2057: repool after upgrade * 10:58 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:55 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 10:46 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1179: Migration of db1179.eqiad.wmnet completed * 10:38 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1179.eqiad.wmnet with OS trixie * 10:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1179.eqiad.wmnet with reason: host reimage * 10:16 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply * 10:15 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply * 10:15 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply * 10:15 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/kartotherian: apply * 10:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1179.eqiad.wmnet with reason: host reimage * 10:13 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2057: repool after upgrade * 10:13 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2057.codfw.wmnet with OS trixie * 09:59 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1179.eqiad.wmnet with OS trixie * 09:58 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1179: Upgrading db1179.eqiad.wmnet * 09:58 jynus: redoing m2 backups after grant change [[phab:T411111|T411111]] * 09:57 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1179: Upgrading db1179.eqiad.wmnet * 09:56 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:54 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2057.codfw.wmnet with reason: host reimage * 09:53 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 09:49 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2057.codfw.wmnet with reason: host reimage * 09:39 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:39 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Migration of db1224.eqiad.wmnet completed * 09:38 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 09:37 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 09:36 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 09:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/kafka-ui: apply * 09:33 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2057.codfw.wmnet with OS trixie * 09:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2057: Upgrading es2057.codfw.wmnet * 09:32 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2057: Upgrading es2057.codfw.wmnet * 09:31 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:26 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=30 --sleep=60 --verbose` * 09:25 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist "group0.dblist + group1.dblist - mediamoderation-continuous-scan.dblist" extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` * 08:54 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Introduce pluggable authentication - oblivian@cumin1003" * 08:54 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Introduce pluggable authentication - oblivian@cumin1003 * 08:53 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Migration of db1224.eqiad.wmnet completed * 08:53 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Introduce pluggable authentication - oblivian@cumin1003 * 08:53 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Introduce pluggable authentication - oblivian@cumin1003" * 08:29 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 08:29 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 08:24 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 08:24 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 08:21 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 08:21 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1224.eqiad.wmnet with OS trixie * 08:21 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 08:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1224.eqiad.wmnet with reason: host reimage * 08:02 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2249.codfw.wmnet with reason: upgrade * 08:00 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1224.eqiad.wmnet with reason: host reimage * 07:53 marostegui: Install mariadb 10.11.17 on db2249 [[phab:T427345|T427345]] * 07:43 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1224.eqiad.wmnet with OS trixie * 07:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1224: Upgrading db1224.eqiad.wmnet * 07:41 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1224: Upgrading db1224.eqiad.wmnet * 07:41 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:39 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:39 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1255: Migration of db1255.eqiad.wmnet completed * 07:34 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297536{{!}}hCaptcha risk scores: VE plugin to collect risk scores for block notices (T426943)]], [[gerrit:1297200{{!}}hCaptcha: Render a fresh mobile widget for each captcha attempt (T425929)]], [[gerrit:1297173{{!}}hCaptcha: Enable risk-score collection for users blocked by IP blocks (T424629)]] (duration: 08m 56s) * 07:29 kharlan@deploy1003: kharlan, harroyo-wmf: Continuing with deployment * 07:27 kharlan@deploy1003: kharlan, harroyo-wmf: Backport for [[gerrit:1297536{{!}}hCaptcha risk scores: VE plugin to collect risk scores for block notices (T426943)]], [[gerrit:1297200{{!}}hCaptcha: Render a fresh mobile widget for each captcha attempt (T425929)]], [[gerrit:1297173{{!}}hCaptcha: Enable risk-score collection for users blocked by IP blocks (T424629)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwd * 07:25 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297536{{!}}hCaptcha risk scores: VE plugin to collect risk scores for block notices (T426943)]], [[gerrit:1297200{{!}}hCaptcha: Render a fresh mobile widget for each captcha attempt (T425929)]], [[gerrit:1297173{{!}}hCaptcha: Enable risk-score collection for users blocked by IP blocks (T424629)]] * 07:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2191: Migration of db2191.codfw.wmnet completed * 07:12 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297550{{!}}Revert "EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion"]] (duration: 06m 45s) * 07:08 kharlan@deploy1003: kharlan: Continuing with deployment * 07:08 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297550{{!}}Revert "EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:06 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297550{{!}}Revert "EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion"]] * 07:04 otto@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297260{{!}}EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] (duration: 399m 30s) * 07:03 otto@deploy1003: otto: Rolling back deployment * 06:53 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1255: Migration of db1255.eqiad.wmnet completed * 06:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1255.eqiad.wmnet with OS trixie * 06:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2191: Migration of db2191.codfw.wmnet completed * 06:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1255.eqiad.wmnet with reason: host reimage * 06:32 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2191.codfw.wmnet with OS trixie * 06:31 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1255.eqiad.wmnet with reason: host reimage * 06:16 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1255.eqiad.wmnet with OS trixie * 06:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2191.codfw.wmnet with reason: host reimage * 06:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1255: Upgrading db1255.eqiad.wmnet * 06:12 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1255: Upgrading db1255.eqiad.wmnet * 06:12 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2191.codfw.wmnet with reason: host reimage * 06:04 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db1255 [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93836 and previous config saved to /var/cache/conftool/dbconfig/20260604-060428-cwilliams.json * 06:03 cwilliams@dns1004: END - running authdns-update * 06:02 cwilliams@dns1004: START - running authdns-update * 05:54 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db1258 to x3 primary and set section read-write [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93835 and previous config saved to /var/cache/conftool/dbconfig/20260604-055429-cwilliams.json * 05:53 cwilliams@cumin1003: dbctl commit (dc=all): 'Set x3 eqiad as read-only for maintenance - [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93834 and previous config saved to /var/cache/conftool/dbconfig/20260604-055346-cwilliams.json * 05:53 cezmunsta: Starting x3 eqiad failover from db1255 to db1258 - [[phab:T427895|T427895]] * 05:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2191.codfw.wmnet with OS trixie * 05:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2191: Upgrading db2191.codfw.wmnet * 05:50 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2191: Upgrading db2191.codfw.wmnet * 05:50 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db1258 with weight 0 [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93833 and previous config saved to /var/cache/conftool/dbconfig/20260604-055021-cwilliams.json * 05:50 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:50 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Primary switchover x3 [[phab:T427895|T427895]] * 05:48 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 05:46 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2191 [[phab:T428120|T428120]]', diff saved to https://phabricator.wikimedia.org/P93832 and previous config saved to /var/cache/conftool/dbconfig/20260604-054614-marostegui.json * 05:45 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db2215 to x1 primary [[phab:T428120|T428120]]', diff saved to https://phabricator.wikimedia.org/P93831 and previous config saved to /var/cache/conftool/dbconfig/20260604-054528-marostegui.json * 05:44 marostegui: Starting x1 codfw failover from db2191 to db2215 - [[phab:T428120|T428120]] * 05:27 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: Primary switchover x1 [[phab:T428120|T428120]] * 05:27 marostegui@cumin1003: dbctl commit (dc=all): 'Set db2215 with weight 0 [[phab:T428120|T428120]]', diff saved to https://phabricator.wikimedia.org/P93830 and previous config saved to /var/cache/conftool/dbconfig/20260604-052722-marostegui.json * 05:19 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 03:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93829 and previous config saved to /var/cache/conftool/dbconfig/20260604-034546-fceratto.json * 03:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P93828 and previous config saved to /var/cache/conftool/dbconfig/20260604-033538-fceratto.json * 03:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P93827 and previous config saved to /var/cache/conftool/dbconfig/20260604-032531-fceratto.json * 03:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93826 and previous config saved to /var/cache/conftool/dbconfig/20260604-031523-fceratto.json * 03:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1263 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93825 and previous config saved to /var/cache/conftool/dbconfig/20260604-030710-fceratto.json * 03:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1263.eqiad.wmnet with reason: Maintenance * 03:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93824 and previous config saved to /var/cache/conftool/dbconfig/20260604-030642-fceratto.json * 02:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P93823 and previous config saved to /var/cache/conftool/dbconfig/20260604-025634-fceratto.json * 02:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P93822 and previous config saved to /var/cache/conftool/dbconfig/20260604-024627-fceratto.json * 02:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93821 and previous config saved to /var/cache/conftool/dbconfig/20260604-023619-fceratto.json * 02:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1262 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93820 and previous config saved to /var/cache/conftool/dbconfig/20260604-022809-fceratto.json * 02:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Maintenance * 02:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93819 and previous config saved to /var/cache/conftool/dbconfig/20260604-022742-fceratto.json * 02:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P93818 and previous config saved to /var/cache/conftool/dbconfig/20260604-021734-fceratto.json * 02:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P93817 and previous config saved to /var/cache/conftool/dbconfig/20260604-020726-fceratto.json * 01:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93816 and previous config saved to /var/cache/conftool/dbconfig/20260604-015718-fceratto.json * 01:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1261 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93815 and previous config saved to /var/cache/conftool/dbconfig/20260604-014909-fceratto.json * 01:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1261.eqiad.wmnet with reason: Maintenance * 01:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93814 and previous config saved to /var/cache/conftool/dbconfig/20260604-014841-fceratto.json * 01:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P93813 and previous config saved to /var/cache/conftool/dbconfig/20260604-013833-fceratto.json * 01:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P93812 and previous config saved to /var/cache/conftool/dbconfig/20260604-012826-fceratto.json * 01:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93811 and previous config saved to /var/cache/conftool/dbconfig/20260604-011818-fceratto.json * 01:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1260 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93810 and previous config saved to /var/cache/conftool/dbconfig/20260604-011005-fceratto.json * 01:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1260.eqiad.wmnet with reason: Maintenance * 01:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93809 and previous config saved to /var/cache/conftool/dbconfig/20260604-010937-fceratto.json * 00:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P93808 and previous config saved to /var/cache/conftool/dbconfig/20260604-005929-fceratto.json * 00:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P93807 and previous config saved to /var/cache/conftool/dbconfig/20260604-004922-fceratto.json * 00:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93806 and previous config saved to /var/cache/conftool/dbconfig/20260604-003914-fceratto.json * 00:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1252 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93805 and previous config saved to /var/cache/conftool/dbconfig/20260604-002851-fceratto.json * 00:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1252.eqiad.wmnet with reason: Maintenance * 00:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93804 and previous config saved to /var/cache/conftool/dbconfig/20260604-002821-fceratto.json * 00:26 otto@deploy1003: otto: Backport for [[gerrit:1297260{{!}}EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 00:24 otto@deploy1003: Started scap sync-world: Backport for [[gerrit:1297260{{!}}EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] * 00:18 Amir1: mwscript-k8s --follow --dblist=all -- extensions/timeline/maintenance/DeleteOldTimelineFiles.php --date {{Gerrit|20210101000000}} * 00:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P93803 and previous config saved to /var/cache/conftool/dbconfig/20260604-001813-fceratto.json * 00:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P93802 and previous config saved to /var/cache/conftool/dbconfig/20260604-000805-fceratto.json == 2026-06-03 == * 23:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93801 and previous config saved to /var/cache/conftool/dbconfig/20260603-235758-fceratto.json * 23:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93800 and previous config saved to /var/cache/conftool/dbconfig/20260603-234935-fceratto.json * 23:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1248.eqiad.wmnet with reason: Maintenance * 23:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93799 and previous config saved to /var/cache/conftool/dbconfig/20260603-234907-fceratto.json * 23:42 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296561{{!}}Add a maintenance script to delete old files]], [[gerrit:1296560{{!}}Add a maintenance script to delete old files]] (duration: 07m 09s) * 23:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P93798 and previous config saved to /var/cache/conftool/dbconfig/20260603-233859-fceratto.json * 23:37 ladsgroup@deploy1003: ladsgroup, reedy: Continuing with deployment * 23:36 ladsgroup@deploy1003: ladsgroup, reedy: Backport for [[gerrit:1296561{{!}}Add a maintenance script to delete old files]], [[gerrit:1296560{{!}}Add a maintenance script to delete old files]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:34 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1296561{{!}}Add a maintenance script to delete old files]], [[gerrit:1296560{{!}}Add a maintenance script to delete old files]] * 23:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P93797 and previous config saved to /var/cache/conftool/dbconfig/20260603-232852-fceratto.json * 23:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 23:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 23:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93796 and previous config saved to /var/cache/conftool/dbconfig/20260603-231844-fceratto.json * 23:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93795 and previous config saved to /var/cache/conftool/dbconfig/20260603-231031-fceratto.json * 23:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1247.eqiad.wmnet with reason: Maintenance * 23:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93794 and previous config saved to /var/cache/conftool/dbconfig/20260603-231001-fceratto.json * 22:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P93793 and previous config saved to /var/cache/conftool/dbconfig/20260603-225953-fceratto.json * 22:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P93792 and previous config saved to /var/cache/conftool/dbconfig/20260603-224945-fceratto.json * 22:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93791 and previous config saved to /var/cache/conftool/dbconfig/20260603-223937-fceratto.json * 22:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1244 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93790 and previous config saved to /var/cache/conftool/dbconfig/20260603-223116-fceratto.json * 22:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1244.eqiad.wmnet with reason: Maintenance * 22:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93789 and previous config saved to /var/cache/conftool/dbconfig/20260603-223048-fceratto.json * 22:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P93788 and previous config saved to /var/cache/conftool/dbconfig/20260603-222041-fceratto.json * 22:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P93787 and previous config saved to /var/cache/conftool/dbconfig/20260603-221034-fceratto.json * 22:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93786 and previous config saved to /var/cache/conftool/dbconfig/20260603-220026-fceratto.json * 21:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1243 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93785 and previous config saved to /var/cache/conftool/dbconfig/20260603-215110-fceratto.json * 21:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1243.eqiad.wmnet with reason: Maintenance * 21:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93784 and previous config saved to /var/cache/conftool/dbconfig/20260603-215053-fceratto.json * 21:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P93783 and previous config saved to /var/cache/conftool/dbconfig/20260603-214046-fceratto.json * 21:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P93782 and previous config saved to /var/cache/conftool/dbconfig/20260603-213038-fceratto.json * 21:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93781 and previous config saved to /var/cache/conftool/dbconfig/20260603-212030-fceratto.json * 21:12 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1242 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93779 and previous config saved to /var/cache/conftool/dbconfig/20260603-211206-fceratto.json * 21:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1242.eqiad.wmnet with reason: Maintenance * 21:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93778 and previous config saved to /var/cache/conftool/dbconfig/20260603-211138-fceratto.json * 21:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P93774 and previous config saved to /var/cache/conftool/dbconfig/20260603-210130-fceratto.json * 20:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P93773 and previous config saved to /var/cache/conftool/dbconfig/20260603-205122-fceratto.json * 20:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93772 and previous config saved to /var/cache/conftool/dbconfig/20260603-204115-fceratto.json * 20:33 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297228{{!}}Attribution research don't use testKitchen compatibility layer (T417050)]] (duration: 06m 41s) * 20:32 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1241 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93771 and previous config saved to /var/cache/conftool/dbconfig/20260603-203254-fceratto.json * 20:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1241.eqiad.wmnet with reason: Maintenance * 20:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93770 and previous config saved to /var/cache/conftool/dbconfig/20260603-203227-fceratto.json * 20:29 cjming@deploy1003: cjming: Continuing with deployment * 20:29 cjming@deploy1003: cjming: Backport for [[gerrit:1297228{{!}}Attribution research don't use testKitchen compatibility layer (T417050)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:26 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1297228{{!}}Attribution research don't use testKitchen compatibility layer (T417050)]] * 20:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P93769 and previous config saved to /var/cache/conftool/dbconfig/20260603-202219-fceratto.json * 20:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P93766 and previous config saved to /var/cache/conftool/dbconfig/20260603-201211-fceratto.json * 20:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93765 and previous config saved to /var/cache/conftool/dbconfig/20260603-200203-fceratto.json * 19:59 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 19:59 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 19:59 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 19:59 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 19:53 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93764 and previous config saved to /var/cache/conftool/dbconfig/20260603-195341-fceratto.json * 19:53 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1238.eqiad.wmnet with reason: Maintenance * 19:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93763 and previous config saved to /var/cache/conftool/dbconfig/20260603-195313-fceratto.json * 19:47 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5032.* * 19:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P93762 and previous config saved to /var/cache/conftool/dbconfig/20260603-194306-fceratto.json * 19:39 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5032.* * 19:37 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5032.* * 19:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P93761 and previous config saved to /var/cache/conftool/dbconfig/20260603-193258-fceratto.json * 19:26 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 19:25 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 19:25 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 19:25 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 19:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93760 and previous config saved to /var/cache/conftool/dbconfig/20260603-192250-fceratto.json * 19:22 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 19:22 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 19:14 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93759 and previous config saved to /var/cache/conftool/dbconfig/20260603-191437-fceratto.json * 19:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1024-1025].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 19:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1221.eqiad.wmnet with reason: Maintenance * 19:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93758 and previous config saved to /var/cache/conftool/dbconfig/20260603-191348-fceratto.json * 19:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P93757 and previous config saved to /var/cache/conftool/dbconfig/20260603-190340-fceratto.json * 18:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P93756 and previous config saved to /var/cache/conftool/dbconfig/20260603-185331-fceratto.json * 18:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93755 and previous config saved to /var/cache/conftool/dbconfig/20260603-184324-fceratto.json * 18:34 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1199 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93754 and previous config saved to /var/cache/conftool/dbconfig/20260603-183455-fceratto.json * 18:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance * 18:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93753 and previous config saved to /var/cache/conftool/dbconfig/20260603-183427-fceratto.json * 18:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P93752 and previous config saved to /var/cache/conftool/dbconfig/20260603-182420-fceratto.json * 18:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P93751 and previous config saved to /var/cache/conftool/dbconfig/20260603-181412-fceratto.json * 18:10 dancy@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.5 refs [[phab:T423914|T423914]] * 18:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93750 and previous config saved to /var/cache/conftool/dbconfig/20260603-180404-fceratto.json * 17:57 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5032.* * 17:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93749 and previous config saved to /var/cache/conftool/dbconfig/20260603-175544-fceratto.json * 17:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance * 17:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93748 and previous config saved to /var/cache/conftool/dbconfig/20260603-175342-fceratto.json * 17:52 hashar: contint1003: sudo puppet agent --disable "Prevent Jenkins from coming back" * 17:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P93747 and previous config saved to /var/cache/conftool/dbconfig/20260603-174334-fceratto.json * 17:38 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:37 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 17:37 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:36 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:36 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:35 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:35 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:35 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:34 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:34 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:33 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P93746 and previous config saved to /var/cache/conftool/dbconfig/20260603-173327-fceratto.json * 17:33 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:32 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:29 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5032.* * 17:26 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host sretest2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 17:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93745 and previous config saved to /var/cache/conftool/dbconfig/20260603-172319-fceratto.json * 17:18 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: Stopping before sync operations * 17:17 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: Started scap sync-world: No-deploy scap run to verify scap config change * 17:17 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:15 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:15 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1253 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93744 and previous config saved to /var/cache/conftool/dbconfig/20260603-171521-fceratto.json * 17:15 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:15 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance * 17:14 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93743 and previous config saved to /var/cache/conftool/dbconfig/20260603-171452-fceratto.json * 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:13 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:12 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:10 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:09 ayounsi@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2012.wikimedia.org with OS trixie * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P93742 and previous config saved to /var/cache/conftool/dbconfig/20260603-170444-fceratto.json * 17:04 swfrench@deploy1003: Stopping before sync operations * 17:03 swfrench@deploy1003: Started scap sync-world: No-deploy scap run to verify clean state before config change * 16:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P93741 and previous config saved to /var/cache/conftool/dbconfig/20260603-165436-fceratto.json * 16:53 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:53 hashar: Restarting CI Jenkins one last time # [[phab:T418521|T418521]] * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:44 btullis@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295922{{!}}Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] (duration: 07m 16s) * 16:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93740 and previous config saved to /var/cache/conftool/dbconfig/20260603-164428-fceratto.json * 16:43 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:43 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:42 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:41 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:40 btullis@deploy1003: btullis: Continuing with deployment * 16:39 btullis@deploy1003: btullis: Backport for [[gerrit:1295922{{!}}Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93739 and previous config saved to /var/cache/conftool/dbconfig/20260603-163726-fceratto.json * 16:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance * 16:37 btullis@deploy1003: Started scap sync-world: Backport for [[gerrit:1295922{{!}}Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] * 16:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93738 and previous config saved to /var/cache/conftool/dbconfig/20260603-163658-fceratto.json * 16:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P93737 and previous config saved to /var/cache/conftool/dbconfig/20260603-162650-fceratto.json * 16:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P93736 and previous config saved to /var/cache/conftool/dbconfig/20260603-161643-fceratto.json * 16:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93735 and previous config saved to /var/cache/conftool/dbconfig/20260603-160635-fceratto.json * 16:04 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:59 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93734 and previous config saved to /var/cache/conftool/dbconfig/20260603-155928-fceratto.json * 15:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance * 15:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93733 and previous config saved to /var/cache/conftool/dbconfig/20260603-155859-fceratto.json * 15:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 15:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P93732 and previous config saved to /var/cache/conftool/dbconfig/20260603-154852-fceratto.json * 15:46 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:46 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2012.wikimedia.org with OS trixie * 15:40 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:40 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 15:40 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 15:40 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 15:39 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 15:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P93731 and previous config saved to /var/cache/conftool/dbconfig/20260603-153844-fceratto.json * 15:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93729 and previous config saved to /var/cache/conftool/dbconfig/20260603-152836-fceratto.json * 15:25 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host sretest2012 * 15:25 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2012 * 15:25 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host sretest2012 * 15:25 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2012 * 15:24 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:23 mutante: disabling jenkins on CI servers for maintenance * 15:23 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host sretest2012 * 15:23 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2012 * 15:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 15:21 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1202 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93728 and previous config saved to /var/cache/conftool/dbconfig/20260603-152129-fceratto.json * 15:21 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance * 15:21 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:21 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding sretest2012 to codfw - jhancock@cumin2002" * 15:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 15:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93727 and previous config saved to /var/cache/conftool/dbconfig/20260603-152102-fceratto.json * 15:20 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding sretest2012 to codfw - jhancock@cumin2002" * 15:18 brouberol@dns1004: END - running authdns-update * 15:18 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:16 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 15:16 brouberol@dns1004: START - running authdns-update * 15:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P93726 and previous config saved to /var/cache/conftool/dbconfig/20260603-151055-fceratto.json * 15:01 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P93725 and previous config saved to /var/cache/conftool/dbconfig/20260603-150047-fceratto.json * 14:57 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 14:52 cmooney@cumin1003: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox * 14:51 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93723 and previous config saved to /var/cache/conftool/dbconfig/20260603-145039-fceratto.json * 14:48 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297137{{!}}Revert "MultimediaViewer: enable image carousel as a beta feature on Wikipedias"]] (duration: 06m 46s) * 14:47 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 14:46 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:46 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:43 mlitn@deploy1003: mlitn: Continuing with deployment * 14:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93722 and previous config saved to /var/cache/conftool/dbconfig/20260603-144334-fceratto.json * 14:43 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:43 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance * 14:43 mlitn@deploy1003: mlitn: Backport for [[gerrit:1297137{{!}}Revert "MultimediaViewer: enable image carousel as a beta feature on Wikipedias"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93721 and previous config saved to /var/cache/conftool/dbconfig/20260603-144306-fceratto.json * 14:41 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:41 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:41 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1297137{{!}}Revert "MultimediaViewer: enable image carousel as a beta feature on Wikipedias"]] * 14:39 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:39 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:39 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:39 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:38 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:35 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 14:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 14:34 sgimeno@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297130{{!}}editor: make redesigned anon warning the default experience (T424595)]] (duration: 10m 45s) * 14:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P93719 and previous config saved to /var/cache/conftool/dbconfig/20260603-143259-fceratto.json * 14:30 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:28 sgimeno@deploy1003: sgimeno: Continuing with deployment * 14:25 sgimeno@deploy1003: sgimeno: Backport for [[gerrit:1297130{{!}}editor: make redesigned anon warning the default experience (T424595)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:24 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:24 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:23 sgimeno@deploy1003: Started scap sync-world: Backport for [[gerrit:1297130{{!}}editor: make redesigned anon warning the default experience (T424595)]] * 14:23 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P93717 and previous config saved to /var/cache/conftool/dbconfig/20260603-142251-fceratto.json * 14:22 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:22 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:21 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:21 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:21 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:20 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:20 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:20 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:20 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:19 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:19 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:16 vriley@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:16 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:16 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:13 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:12 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93716 and previous config saved to /var/cache/conftool/dbconfig/20260603-141242-fceratto.json * 14:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:11 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:11 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:10 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mc2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:10 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host mc2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:10 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:09 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:08 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:07 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:05 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296631{{!}}translate: adding separate read/write endpoints (T425377)]] (duration: 13m 06s) * 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1191 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93715 and previous config saved to /var/cache/conftool/dbconfig/20260603-140537-fceratto.json * 14:05 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance * 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93714 and previous config saved to /var/cache/conftool/dbconfig/20260603-140507-fceratto.json * 14:01 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:58 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:58 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:58 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 13:56 dcausse@deploy1003: atsuko, dcausse: Rolling back deployment * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T426633|T426633]])', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20260603-133440-fceratto.json * 13:29 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:29 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2186: Migration of db2186.codfw.wmnet completed * 13:28 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295910{{!}}hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]] (duration: 07m 36s) * 13:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1174 ([[phab:T426633|T426633]])', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20260603-132638-fceratto.json * 13:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance * 13:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93710 and previous config saved to /var/cache/conftool/dbconfig/20260603-132605-fceratto.json * 13:25 sukhe: sudo cumin 'A:lvs or A:liberica' 'disable-puppet "merging CR 1282764"' * 13:23 kharlan@deploy1003: kharlan: Continuing with deployment * 13:22 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295910{{!}}hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:20 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295910{{!}}hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]] * 13:18 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296649{{!}}hCaptcha: Roll out to all except enwiki for mobile apps. (T426048)]] (duration: 07m 46s) * 13:16 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20260603-131556-fceratto.json * 13:15 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:13 kharlan@deploy1003: dbrant, kharlan: Continuing with deployment * 13:12 kharlan@deploy1003: dbrant, kharlan: Backport for [[gerrit:1296649{{!}}hCaptcha: Roll out to all except enwiki for mobile apps. (T426048)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296649{{!}}hCaptcha: Roll out to all except enwiki for mobile apps. (T426048)]] * 13:09 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:09 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add codfw d3 and e5 public vlans - ayounsi@cumin1003" * 13:09 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add codfw d3 and e5 public vlans - ayounsi@cumin1003" * 13:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P93708 and previous config saved to /var/cache/conftool/dbconfig/20260603-130548-fceratto.json * 13:05 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 12:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93706 and previous config saved to /var/cache/conftool/dbconfig/20260603-125540-fceratto.json * 12:51 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297110{{!}}ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] (duration: 07m 44s) * 12:49 jgreen@dns1004: END - running authdns-update * 12:47 jgreen@dns1004: START - running authdns-update * 12:46 jiji@deploy1003: jiji: Continuing with deployment * 12:46 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93705 and previous config saved to /var/cache/conftool/dbconfig/20260603-124624-fceratto.json * 12:46 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance * 12:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93704 and previous config saved to /var/cache/conftool/dbconfig/20260603-124556-fceratto.json * 12:45 jiji@deploy1003: jiji: Backport for [[gerrit:1297110{{!}}ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:43 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2186: Migration of db2186.codfw.wmnet completed * 12:43 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1297110{{!}}ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] * 12:41 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1067.eqiad.wmnet with OS bullseye * 12:38 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1292364{{!}}Update hCaptcha checks to retrieve API parameters from $_REQUEST (T427105)]] (duration: 11m 15s) * 12:36 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2186.codfw.wmnet with OS trixie * 12:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P93702 and previous config saved to /var/cache/conftool/dbconfig/20260603-123548-fceratto.json * 12:34 dreamyjazz@deploy1003: somerandomdeveloper, dreamyjazz: Continuing with deployment * 12:31 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1066.eqiad.wmnet with OS bullseye * 12:29 dreamyjazz@deploy1003: somerandomdeveloper, dreamyjazz: Backport for [[gerrit:1292364{{!}}Update hCaptcha checks to retrieve API parameters from $_REQUEST (T427105)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:27 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1292364{{!}}Update hCaptcha checks to retrieve API parameters from $_REQUEST (T427105)]] * 12:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P93701 and previous config saved to /var/cache/conftool/dbconfig/20260603-122541-fceratto.json * 12:22 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1067.eqiad.wmnet with reason: host reimage * 12:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2186.codfw.wmnet with reason: host reimage * 12:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93700 and previous config saved to /var/cache/conftool/dbconfig/20260603-121533-fceratto.json * 12:13 mvernon@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ms-be1066.eqiad.wmnet with reason: host reimage * 12:13 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2186.codfw.wmnet with reason: host reimage * 12:11 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1067.eqiad.wmnet with reason: host reimage * 12:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93699 and previous config saved to /var/cache/conftool/dbconfig/20260603-120732-fceratto.json * 12:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 12:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance * 12:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93698 and previous config saved to /var/cache/conftool/dbconfig/20260603-120634-fceratto.json * 12:03 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1066.eqiad.wmnet with reason: host reimage * 11:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P93697 and previous config saved to /var/cache/conftool/dbconfig/20260603-115626-fceratto.json * 11:54 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2186.codfw.wmnet with OS trixie * 11:54 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be1067 * 11:54 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1067 * 11:52 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1067 * 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be1067.eqiad.wmnet 96.48.64.10.in-addr.arpa 6.9.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:52 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be1067.eqiad.wmnet 96.48.64.10.in-addr.arpa 6.9.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1067 - mvernon@cumin2002" * 11:52 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1067 - mvernon@cumin2002" * 11:48 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2186: Upgrading db2186.codfw.wmnet * 11:48 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2186: Upgrading db2186.codfw.wmnet * 11:48 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:47 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:46 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be1067 * 11:46 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be1067.eqiad.wmnet with OS bullseye * 11:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P93695 and previous config saved to /var/cache/conftool/dbconfig/20260603-114618-fceratto.json * 11:46 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be1066 * 11:46 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1066 * 11:45 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1066 * 11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be1066.eqiad.wmnet 117.32.64.10.in-addr.arpa 7.1.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:45 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be1066.eqiad.wmnet 117.32.64.10.in-addr.arpa 7.1.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1066 - mvernon@cumin2002" * 11:45 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1066 - mvernon@cumin2002" * 11:43 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply * 11:41 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:40 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be1066 * 11:40 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be1066.eqiad.wmnet with OS bullseye * 11:39 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be1067 * 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93693 and previous config saved to /var/cache/conftool/dbconfig/20260603-113611-fceratto.json * 11:33 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:33 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2196: Migration of db2196.codfw.wmnet completed * 11:29 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93691 and previous config saved to /var/cache/conftool/dbconfig/20260603-112909-fceratto.json * 11:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance * 11:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance * 11:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93690 and previous config saved to /var/cache/conftool/dbconfig/20260603-112838-fceratto.json * 11:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:20 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:20 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:20 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P93689 and previous config saved to /var/cache/conftool/dbconfig/20260603-111831-fceratto.json * 11:14 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:09 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply * 11:09 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply * 11:08 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P93687 and previous config saved to /var/cache/conftool/dbconfig/20260603-110823-fceratto.json * 11:07 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be1066 * 11:07 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply * 11:06 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply * 11:05 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply * 11:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:01 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:01 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:00 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289895{{!}}Update UserInfoCard to be enabled by default for certain user groups (T426021)]] (duration: 07m 37s) * 11:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:59 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply * 10:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:59 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply * 10:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:58 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply * 10:58 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93685 and previous config saved to /var/cache/conftool/dbconfig/20260603-105815-fceratto.json * 10:58 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:56 mszwarc@deploy1003: mszwarc: Continuing with deployment * 10:55 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1289895{{!}}Update UserInfoCard to be enabled by default for certain user groups (T426021)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:54 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply * 10:54 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply * 10:53 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply * 10:53 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1289895{{!}}Update UserInfoCard to be enabled by default for certain user groups (T426021)]] * 10:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:50 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1198 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93684 and previous config saved to /var/cache/conftool/dbconfig/20260603-105006-fceratto.json * 10:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance * 10:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93683 and previous config saved to /var/cache/conftool/dbconfig/20260603-104939-fceratto.json * 10:45 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:45 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:44 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2196: Migration of db2196.codfw.wmnet completed * 10:44 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:41 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 10:40 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 10:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P93681 and previous config saved to /var/cache/conftool/dbconfig/20260603-103931-fceratto.json * 10:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1053: repool after upgrade * 10:37 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2196.codfw.wmnet with OS trixie * 10:36 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297090{{!}}hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] (duration: 12m 03s) * 10:32 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P93679 and previous config saved to /var/cache/conftool/dbconfig/20260603-102924-fceratto.json * 10:26 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297090{{!}}hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:24 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297090{{!}}hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] * 10:22 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be1067 * 10:21 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be1066 * 10:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2196.codfw.wmnet with reason: host reimage * 10:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93677 and previous config saved to /var/cache/conftool/dbconfig/20260603-101916-fceratto.json * 10:15 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2013.codfw.wmnet * 10:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2196.codfw.wmnet with reason: host reimage * 10:11 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93676 and previous config saved to /var/cache/conftool/dbconfig/20260603-101105-fceratto.json * 10:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1189.eqiad.wmnet with reason: Maintenance * 10:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93675 and previous config saved to /var/cache/conftool/dbconfig/20260603-101037-fceratto.json * 10:10 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2013.codfw.wmnet * 10:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P93673 and previous config saved to /var/cache/conftool/dbconfig/20260603-100029-fceratto.json * 09:59 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2196.codfw.wmnet with OS trixie * 09:57 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2196: Upgrading db2196.codfw.wmnet * 09:57 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2196: Upgrading db2196.codfw.wmnet * 09:57 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:52 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1053: repool after upgrade * 09:52 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:52 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:52 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:52 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:51 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:51 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:51 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P93670 and previous config saved to /var/cache/conftool/dbconfig/20260603-095022-fceratto.json * 09:49 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:49 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:48 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1053.eqiad.wmnet with OS trixie * 09:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:43 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2013.codfw.wmnet * 09:41 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on es1053.eqiad.wmnet with reason: host reimage * 09:41 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1053.eqiad.wmnet with reason: host reimage * 09:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93669 and previous config saved to /var/cache/conftool/dbconfig/20260603-094014-fceratto.json * 09:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2215: Migration of db2215.codfw.wmnet completed * 09:38 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2013.codfw.wmnet * 09:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93667 and previous config saved to /var/cache/conftool/dbconfig/20260603-093146-fceratto.json * 09:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance * 09:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93666 and previous config saved to /var/cache/conftool/dbconfig/20260603-093119-fceratto.json * 09:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1211: Migration of db1211.eqiad.wmnet completed * 09:27 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297069{{!}}hCaptcha: Collect risk score for blocked account creations (T427784)]] (duration: 07m 26s) * 09:25 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1053.eqiad.wmnet with OS trixie * 09:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add public1-b3-codfw gateway IPs - ayounsi@cumin1003" * 09:24 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add public1-b3-codfw gateway IPs - ayounsi@cumin1003" * 09:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1053: Upgrading es1053.eqiad.wmnet * 09:23 kharlan@deploy1003: kharlan: Continuing with deployment * 09:22 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1053: Upgrading es1053.eqiad.wmnet * 09:22 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:21 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297069{{!}}hCaptcha: Collect risk score for blocked account creations (T427784)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:21 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply * 09:21 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2054: repool after upgrade * 09:21 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply * 09:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P93661 and previous config saved to /var/cache/conftool/dbconfig/20260603-092111-fceratto.json * 09:20 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 09:20 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297069{{!}}hCaptcha: Collect risk score for blocked account creations (T427784)]] * 09:14 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297065{{!}}Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] (duration: 07m 06s) * 09:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P93659 and previous config saved to /var/cache/conftool/dbconfig/20260603-091104-fceratto.json * 09:10 kharlan@deploy1003: kharlan: Continuing with deployment * 09:09 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297065{{!}}Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:07 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297065{{!}}Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] * 09:06 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 09:06 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297064{{!}}Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 10m 54s) * 09:05 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 09:04 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 09:01 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003 - [[phab:T422043|T422043]]" * 09:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93656 and previous config saved to /var/cache/conftool/dbconfig/20260603-090056-fceratto.json * 09:00 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003 - [[phab:T422043|T422043]]" * 09:00 ayounsi@cumin1003: END (ERROR) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=97) generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003" * 09:00 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003" * 08:59 kharlan@deploy1003: kharlan: Continuing with deployment * 08:59 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297064{{!}}Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:55 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297064{{!}}Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] * 08:53 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296635{{!}}Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 11m 43s) * 08:52 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2215: Migration of db2215.codfw.wmnet completed * 08:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet * 08:52 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet * 08:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb[1022-1023].eqiad.wmnet * 08:51 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb[1022-1023].eqiad.wmnet * 08:50 kharlan@deploy1003: kharlan: Rolling back deployment * 08:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93652 and previous config saved to /var/cache/conftool/dbconfig/20260603-084846-fceratto.json * 08:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance * 08:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93651 and previous config saved to /var/cache/conftool/dbconfig/20260603-084819-fceratto.json * 08:47 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296635{{!}}Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2215.codfw.wmnet with OS trixie * 08:45 jiji@cumin1003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check docker-registry: maintenance * 08:45 jiji@cumin1003: START - Cookbook sre.discovery.service-route check docker-registry: maintenance * 08:43 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1211: Migration of db1211.eqiad.wmnet completed * 08:41 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296635{{!}}Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] * 08:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1211.eqiad.wmnet with OS trixie * 08:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93649 and previous config saved to /var/cache/conftool/dbconfig/20260603-083811-fceratto.json * 08:37 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296632{{!}}Image Browsing: add accessible labels to carousel elements (T407793)]] (duration: 32m 11s) * 08:36 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2054: repool after upgrade * 08:35 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool es2054.codfw.wmnet: After reimage * 08:35 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2054.codfw.wmnet: After reimage * 08:35 jiji@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:34 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 08:34 jiji@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:33 jiji@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:33 jiji@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:31 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:31 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:31 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2054.codfw.wmnet with OS trixie * 08:30 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:29 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 08:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2215.codfw.wmnet with reason: host reimage * 08:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93647 and previous config saved to /var/cache/conftool/dbconfig/20260603-082804-fceratto.json * 08:25 mszwarc@deploy1003: mlitn, mszwarc: Continuing with deployment * 08:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1211.eqiad.wmnet with reason: host reimage * 08:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1049: repool after upgrade * 08:22 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2215.codfw.wmnet with reason: host reimage * 08:22 mszwarc@deploy1003: mlitn, mszwarc: Backport for [[gerrit:1296632{{!}}Image Browsing: add accessible labels to carousel elements (T407793)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:18 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1211.eqiad.wmnet with reason: host reimage * 08:18 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 08:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93645 and previous config saved to /var/cache/conftool/dbconfig/20260603-081756-fceratto.json * 08:17 jiji@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 08:17 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 08:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 08:14 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2054.codfw.wmnet with reason: host reimage * 08:08 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2054.codfw.wmnet with reason: host reimage * 08:05 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296632{{!}}Image Browsing: add accessible labels to carousel elements (T407793)]] * {{safesubst:SAL entry|1=08:04 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296580{{!}}Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703{{!}}jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713{{!}}conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627{{!}}Add missing lazy img to carousel (T427821)]], [[gerrit:1295968{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T426799)]}} * 08:03 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93643 and previous config saved to /var/cache/conftool/dbconfig/20260603-080346-fceratto.json * 08:03 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1211.eqiad.wmnet with OS trixie * 08:03 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance * 08:03 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2215.codfw.wmnet with OS trixie * 08:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1211: Upgrading db1211.eqiad.wmnet * 08:02 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2215: Upgrading db2215.codfw.wmnet * 08:01 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:01 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1211: Upgrading db1211.eqiad.wmnet * 08:01 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2215: Upgrading db2215.codfw.wmnet * 08:01 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:01 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:01 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1157: Repooling * 08:01 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1157: Repooling * 08:00 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 07:57 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1022-1023].eqiad.wmnet with reason: Reimaging upstream server * 07:57 mszwarc@deploy1003: anzx, mlitn, mfossati, mszwarc: Continuing with deployment * 07:56 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Reimaging upstream server * {{safesubst:SAL entry|1=07:54 mszwarc@deploy1003: anzx, mlitn, mfossati, mszwarc: Backport for [[gerrit:1296580{{!}}Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703{{!}}jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713{{!}}conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627{{!}}Add missing lazy img to carousel (T427821)]], [[gerrit:1295968{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T42}} * 07:52 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2231: repool after maintenance * 07:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2054.codfw.wmnet with OS trixie * 07:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2054: Upgrading es2054.codfw.wmnet * 07:50 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2054: Upgrading es2054.codfw.wmnet * 07:50 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:50 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296580{{!}}Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703{{!}}jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713{{!}}conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627{{!}}Add missing lazy img to carousel (T427821)]], [[gerrit:1295968{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T426799)]] * 07:48 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296516{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] (duration: 32m 13s) * 07:44 marostegui@dns1004: END - running authdns-update * 07:43 marostegui@dns1004: START - running authdns-update * 07:42 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1056 to es2 eqiad primary [[phab:T427875|T427875]]', diff saved to https://phabricator.wikimedia.org/P93637 and previous config saved to /var/cache/conftool/dbconfig/20260603-074250-marostegui.json * 07:37 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1049: repool after upgrade * 07:37 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:35 mszwarc@deploy1003: mszwarc, stran: Continuing with deployment * 07:35 mszwarc@deploy1003: mszwarc, stran: Backport for [[gerrit:1296516{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:32 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1049.eqiad.wmnet with OS trixie * 07:16 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296516{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] * 07:14 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1049.eqiad.wmnet with reason: host reimage * 07:07 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1049.eqiad.wmnet with reason: host reimage * 07:07 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2231: repool after maintenance * 07:04 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 06:57 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2231.codfw.wmnet with OS trixie * 06:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1049.eqiad.wmnet with OS trixie * 06:46 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1049: Upgrading es1049.eqiad.wmnet * 06:46 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2056 to es2 codfw primary [[phab:T427875|T427875]]', diff saved to https://phabricator.wikimedia.org/P93632 and previous config saved to /var/cache/conftool/dbconfig/20260603-064623-marostegui.json * 06:45 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1049: Upgrading es1049.eqiad.wmnet * 06:45 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:44 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1056: repool after upgrade * 06:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2231.codfw.wmnet with reason: host reimage * 06:36 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2231.codfw.wmnet with reason: host reimage * 06:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2231.codfw.wmnet with OS trixie * 06:09 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2231: Upgrading db2231.codfw.wmnet * 06:09 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2231: Upgrading db2231.codfw.wmnet * 06:09 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:59 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1056: repool after upgrade * 05:59 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 05:55 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1056.eqiad.wmnet with OS trixie * 05:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1056.eqiad.wmnet with reason: host reimage * 05:33 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1056.eqiad.wmnet with reason: host reimage * 05:18 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1056.eqiad.wmnet with OS trixie * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1056: Upgrading es1056.eqiad.wmnet * 05:17 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1056: Upgrading es1056.eqiad.wmnet * 05:16 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade == 2026-06-02 == * 22:21 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296689{{!}}hCaptcha: Correct inaccurate comment]] (duration: 06m 27s) * 22:18 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 22:18 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 22:17 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 22:17 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296689{{!}}hCaptcha: Correct inaccurate comment]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:15 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296689{{!}}hCaptcha: Correct inaccurate comment]] * 22:13 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296551{{!}}hCaptcha: Enable for badlogin on group0 wikis (T426875)]] (duration: 08m 31s) * 22:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 22:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 22:09 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 22:07 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296551{{!}}hCaptcha: Enable for badlogin on group0 wikis (T426875)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:05 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296551{{!}}hCaptcha: Enable for badlogin on group0 wikis (T426875)]] * 20:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93621 and previous config saved to /var/cache/conftool/dbconfig/20260602-203945-fceratto.json * 20:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93620 and previous config saved to /var/cache/conftool/dbconfig/20260602-202937-fceratto.json * 20:27 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1054.eqiad.wmnet * 20:27 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:27 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1054.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:26 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1054.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:20 jiji@cumin1003: START - Cookbook sre.dns.netbox * 20:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93619 and previous config saved to /var/cache/conftool/dbconfig/20260602-201929-fceratto.json * 20:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93618 and previous config saved to /var/cache/conftool/dbconfig/20260602-200922-fceratto.json * 20:03 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1054.eqiad.wmnet * 19:48 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1053.eqiad.wmnet * 19:48 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:48 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1053.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 19:37 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1053.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 19:09 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93617 and previous config saved to /var/cache/conftool/dbconfig/20260602-190907-fceratto.json * 19:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance * 19:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93616 and previous config saved to /var/cache/conftool/dbconfig/20260602-190811-fceratto.json * 19:05 dancy@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.5 refs [[phab:T423914|T423914]] * 18:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P93615 and previous config saved to /var/cache/conftool/dbconfig/20260602-185804-fceratto.json * 18:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P93614 and previous config saved to /var/cache/conftool/dbconfig/20260602-184757-fceratto.json * 18:38 jiji@cumin1003: START - Cookbook sre.dns.netbox * 18:38 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:38 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93612 and previous config saved to /var/cache/conftool/dbconfig/20260602-183749-fceratto.json * 18:37 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:37 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:33 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1053.eqiad.wmnet * 18:30 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1259 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93611 and previous config saved to /var/cache/conftool/dbconfig/20260602-183023-fceratto.json * 18:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance * 18:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93610 and previous config saved to /var/cache/conftool/dbconfig/20260602-182956-fceratto.json * 18:27 mutante: gerrit delete unused plugin projects: barricade, WikimediaBlocks and WikimediaWebSessions * 18:26 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1052.eqiad.wmnet * 18:26 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:26 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1052.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 18:25 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1052.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 18:25 dancy: Train is blocked at testwikis on https://phabricator.wikimedia.org/T427935 * 18:21 Daimona: Running query from [[phab:T427962|T427962]]#11978299 in x1.wikishared * 18:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P93609 and previous config saved to /var/cache/conftool/dbconfig/20260602-181949-fceratto.json * 18:16 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296615{{!}}feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614{{!}}feat(cleanMentorList): Add a feature flag (T427386)]] (duration: 34m 09s) * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 18:12 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 18:12 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 18:12 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 18:10 jiji@cumin1003: START - Cookbook sre.dns.netbox * 18:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P93608 and previous config saved to /var/cache/conftool/dbconfig/20260602-180941-fceratto.json * 18:08 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 18:07 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 18:06 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 18:06 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 18:05 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:05 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:05 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 18:05 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 18:04 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 18:02 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 18:02 swfrench-wmf: reverting shellbox to 2026-05-20-192555 due to errors in shellbox-syntaxhighlight * 18:02 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 18:01 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 18:01 urbanecm@deploy1003: urbanecm: Continuing with deployment * 18:01 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1296615{{!}}feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614{{!}}feat(cleanMentorList): Add a feature flag (T427386)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:00 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1052.eqiad.wmnet * 17:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93607 and previous config saved to /var/cache/conftool/dbconfig/20260602-175933-fceratto.json * 17:58 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:57 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:56 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1051.eqiad.wmnet * 17:56 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:56 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1051.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:55 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1051.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:53 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93605 and previous config saved to /var/cache/conftool/dbconfig/20260602-175227-fceratto.json * 17:52 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance * 17:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93604 and previous config saved to /var/cache/conftool/dbconfig/20260602-175157-fceratto.json * 17:51 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:51 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:50 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:50 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:50 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:49 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:49 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:48 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:48 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:47 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:44 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:42 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:42 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:42 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P93603 and previous config saved to /var/cache/conftool/dbconfig/20260602-174150-fceratto.json * 17:41 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1296615{{!}}feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614{{!}}feat(cleanMentorList): Add a feature flag (T427386)]] * 17:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P93602 and previous config saved to /var/cache/conftool/dbconfig/20260602-173143-fceratto.json * 17:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93601 and previous config saved to /var/cache/conftool/dbconfig/20260602-172135-fceratto.json * 17:14 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93600 and previous config saved to /var/cache/conftool/dbconfig/20260602-171422-fceratto.json * 17:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance * 17:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93599 and previous config saved to /var/cache/conftool/dbconfig/20260602-171354-fceratto.json * 17:04 jiji@cumin1003: START - Cookbook sre.dns.netbox * 17:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P93598 and previous config saved to /var/cache/conftool/dbconfig/20260602-170344-fceratto.json * 16:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P93597 and previous config saved to /var/cache/conftool/dbconfig/20260602-165336-fceratto.json * 16:49 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1051.eqiad.wmnet * 16:48 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1050.eqiad.wmnet * 16:48 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:48 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1050.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:47 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1050.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93596 and previous config saved to /var/cache/conftool/dbconfig/20260602-164328-fceratto.json * 16:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93595 and previous config saved to /var/cache/conftool/dbconfig/20260602-163622-fceratto.json * 16:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance * 16:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93594 and previous config saved to /var/cache/conftool/dbconfig/20260602-163550-fceratto.json * 16:34 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:34 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1072.eqiad.wmnet with OS trixie * 16:30 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:29 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:27 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2006.codfw.wmnet with OS trixie * 16:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P93593 and previous config saved to /var/cache/conftool/dbconfig/20260602-162542-fceratto.json * 16:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P93591 and previous config saved to /var/cache/conftool/dbconfig/20260602-161534-fceratto.json * 16:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1072.eqiad.wmnet with reason: host reimage * 16:10 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1071.eqiad.wmnet with OS trixie * 16:10 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296624{{!}}Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 06m 40s) * 16:09 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2006.codfw.wmnet with reason: host reimage * 16:05 kharlan@deploy1003: kharlan: Continuing with deployment * 16:05 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1072.eqiad.wmnet with reason: host reimage * 16:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93590 and previous config saved to /var/cache/conftool/dbconfig/20260602-160527-fceratto.json * 16:05 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2006.codfw.wmnet with reason: host reimage * 16:05 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296624{{!}}Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:03 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296624{{!}}Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] * 15:59 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295909{{!}}hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] (duration: 09m 48s) * 15:59 kharlan@deploy1003: kharlan: Rolling back deployment * 15:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93589 and previous config saved to /var/cache/conftool/dbconfig/20260602-155817-fceratto.json * 15:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance * 15:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93588 and previous config saved to /var/cache/conftool/dbconfig/20260602-155749-fceratto.json * 15:54 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1071.eqiad.wmnet with reason: host reimage * 15:53 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1072.eqiad.wmnet with OS trixie * 15:51 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1070.eqiad.wmnet with OS trixie * 15:51 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295909{{!}}hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:50 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1071.eqiad.wmnet with reason: host reimage * 15:49 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295909{{!}}hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] * 15:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P93587 and previous config saved to /var/cache/conftool/dbconfig/20260602-154742-fceratto.json * 15:47 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296558{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] (duration: 07m 24s) * 15:43 kharlan@deploy1003: kharlan: Continuing with deployment * 15:42 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296558{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:40 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296558{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] * 15:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P93586 and previous config saved to /var/cache/conftool/dbconfig/20260602-153734-fceratto.json * 15:37 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1071.eqiad.wmnet with OS trixie * 15:36 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1069.eqiad.wmnet with OS trixie * 15:35 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1070.eqiad.wmnet with reason: host reimage * 15:32 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:32 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:31 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1070.eqiad.wmnet with reason: host reimage * 15:30 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:29 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93585 and previous config saved to /var/cache/conftool/dbconfig/20260602-152726-fceratto.json * 15:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2158: Repooling * {{safesubst:SAL entry|1=15:22 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295502{{!}}Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106{{!}}Remove unused 'writeapi' right]], [[gerrit:1296566{{!}}Clean up bot password configuration]], [[gerrit:1296563{{!}}Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574{{!}}cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582{{!}}U}} * 15:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1069.eqiad.wmnet with reason: host reimage * 15:20 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93583 and previous config saved to /var/cache/conftool/dbconfig/20260602-152026-fceratto.json * 15:20 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance * 15:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93582 and previous config saved to /var/cache/conftool/dbconfig/20260602-151958-fceratto.json * 15:19 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:19 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:18 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1070.eqiad.wmnet with OS trixie * 15:18 dreamyjazz@deploy1003: matmarex, anzx, dreamyjazz: Continuing with deployment * 15:18 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 15:17 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:17 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:15 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1069.eqiad.wmnet with reason: host reimage * {{safesubst:SAL entry|1=15:15 dreamyjazz@deploy1003: matmarex, anzx, dreamyjazz: Backport for [[gerrit:1295502{{!}}Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106{{!}}Remove unused 'writeapi' right]], [[gerrit:1296566{{!}}Clean up bot password configuration]], [[gerrit:1296563{{!}}Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574{{!}}cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582}} * 15:14 jiji@cumin1003: START - Cookbook sre.dns.netbox * {{safesubst:SAL entry|1=15:13 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1295502{{!}}Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106{{!}}Remove unused 'writeapi' right]], [[gerrit:1296566{{!}}Clean up bot password configuration]], [[gerrit:1296563{{!}}Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574{{!}}cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582{{!}}Us}} * 15:12 jayme@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-main2006.codfw.wmnet with OS trixie * 15:12 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1068.eqiad.wmnet with OS trixie * 15:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P93580 and previous config saved to /var/cache/conftool/dbconfig/20260602-150951-fceratto.json * 15:09 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296514{{!}}[Growth] Set wgGEMentorshipCleanupEnabled to false on all wikis (T427386)]] (duration: 06m 22s) * 15:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1167: Repooling after Icing wait-for-green timeout * 15:06 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1050.eqiad.wmnet * 15:06 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1049.eqiad.wmnet * 15:06 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:06 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1049.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:05 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1049.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:02 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1296514{{!}}[Growth] Set wgGEMentorshipCleanupEnabled to false on all wikis (T427386)]] * 15:02 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1069.eqiad.wmnet with OS trixie * 15:01 jiji@cumin1003: START - Cookbook sre.dns.netbox * 14:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P93578 and previous config saved to /var/cache/conftool/dbconfig/20260602-145943-fceratto.json * 14:54 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage * 14:52 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:52 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:52 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1049.eqiad.wmnet * 14:51 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1067.eqiad.wmnet with OS trixie * 14:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:50 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage * 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93575 and previous config saved to /var/cache/conftool/dbconfig/20260602-144935-fceratto.json * 14:42 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for pc2021.codfw.wmnet * 14:42 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for pc2021.codfw.wmnet * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2250.codfw.wmnet * 14:41 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2250.codfw.wmnet * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2158.codfw.wmnet * 14:41 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2158.codfw.wmnet * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc2021: Repooling * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 14:41 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache * 14:41 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc2021: Repooling * 14:41 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93573 and previous config saved to /var/cache/conftool/dbconfig/20260602-144110-fceratto.json * 14:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance * 14:41 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2158: Repooling * 14:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93571 and previous config saved to /var/cache/conftool/dbconfig/20260602-144043-fceratto.json * 14:38 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:38 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:38 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:37 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1048.eqiad.wmnet * 14:37 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:37 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1048.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 14:37 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1068.eqiad.wmnet with OS trixie * 14:36 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1066.eqiad.wmnet with OS trixie * 14:34 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage * 14:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P93569 and previous config saved to /var/cache/conftool/dbconfig/20260602-143035-fceratto.json * 14:30 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage * 14:25 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1048.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 14:21 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1167: Repooling after Icing wait-for-green timeout * 14:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage * 14:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P93566 and previous config saved to /var/cache/conftool/dbconfig/20260602-142027-fceratto.json * 14:17 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1067.eqiad.wmnet with OS trixie * 14:17 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 14:17 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1167.eqiad.wmnet * 14:17 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1167.eqiad.wmnet * 14:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1065.eqiad.wmnet with OS trixie * 14:15 jayme@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2006.codfw.wmnet with OS trixie * 14:14 jiji@cumin1003: START - Cookbook sre.dns.netbox * 14:13 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage * 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93564 and previous config saved to /var/cache/conftool/dbconfig/20260602-141019-fceratto.json * 14:09 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments userOptions.php --delete --nowarn growthexperiments-homepage-variant # [[phab:T417621|T417621]] * 14:09 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1048.eqiad.wmnet * 14:08 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments userOptions.php --delete growthexperiments-homepage-variant # [[phab:T417621|T417621]] * 14:05 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 14:01 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93563 and previous config saved to /var/cache/conftool/dbconfig/20260602-140140-fceratto.json * 14:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 14:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1156.eqiad.wmnet with reason: Maintenance * 14:01 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1066.eqiad.wmnet with OS trixie * 14:00 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage * 14:00 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 14:00 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 14:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93562 and previous config saved to /var/cache/conftool/dbconfig/20260602-140022-fceratto.json * 14:00 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1064.eqiad.wmnet with OS trixie * 13:56 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage * 13:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1167.eqiad.wmnet with OS trixie * 13:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P93561 and previous config saved to /var/cache/conftool/dbconfig/20260602-135015-fceratto.json * 13:47 topranks: revert all config to normal on cr1-codfw and ssw1-a1-codfw * 13:43 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1065.eqiad.wmnet with OS trixie * 13:42 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1064.eqiad.wmnet with reason: host reimage * 13:40 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1063.eqiad.wmnet with OS trixie * 13:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P93560 and previous config saved to /var/cache/conftool/dbconfig/20260602-134007-fceratto.json * 13:38 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1167.eqiad.wmnet with reason: host reimage * 13:35 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs1002.eqiad.wmnet with OS trixie * 13:35 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs1003.eqiad.wmnet with OS trixie * 13:34 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:34 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:32 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1064.eqiad.wmnet with reason: host reimage * 13:31 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1167.eqiad.wmnet with reason: host reimage * 13:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93559 and previous config saved to /var/cache/conftool/dbconfig/20260602-132959-fceratto.json * 13:27 slyngshede@dns1004: END - running authdns-update * 13:25 slyngshede@dns1004: START - running authdns-update * 13:24 topranks: increase OSPF cost on ssw1-a1-codfw et-0/0/4 towards lsw1-a5-codfw [[phab:T427301|T427301]] * 13:23 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1063.eqiad.wmnet with reason: host reimage * 13:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93558 and previous config saved to /var/cache/conftool/dbconfig/20260602-132314-fceratto.json * 13:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1210.eqiad.wmnet with reason: Maintenance * 13:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93557 and previous config saved to /var/cache/conftool/dbconfig/20260602-132246-fceratto.json * 13:20 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1064.eqiad.wmnet with OS trixie * 13:19 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 13:19 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1062.eqiad.wmnet with OS trixie * 13:18 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1063.eqiad.wmnet with reason: host reimage * 13:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2049: repool after upgrade * 13:17 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 13:16 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1167.eqiad.wmnet with OS trixie * 13:15 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 13:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1167: Upgrading db1167.eqiad.wmnet * 13:13 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1167: Upgrading db1167.eqiad.wmnet * 13:13 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:12 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 13:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P93554 and previous config saved to /var/cache/conftool/dbconfig/20260602-131238-fceratto.json * 13:12 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 13:12 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 13:11 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 13:07 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1003.eqiad.wmnet with OS trixie * 13:07 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1002.eqiad.wmnet with OS trixie * 13:06 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1063.eqiad.wmnet with OS trixie * 13:04 jayme@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-main2006.codfw.wmnet with OS trixie * 13:04 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:04 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:03 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1022-1023].eqiad.wmnet with reason: Reimaging upstream servers * 13:03 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1001.eqiad.wmnet with OS trixie * 13:03 topranks: increase OSPF cost on ssw1-a1-codfw et-0/0/2 towards lsw1-a3-codfw [[phab:T427301|T427301]] * 13:03 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1062.eqiad.wmnet with reason: host reimage * 13:02 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Reimaging upstream servers * 13:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P93553 and previous config saved to /var/cache/conftool/dbconfig/20260602-130230-fceratto.json * 12:59 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1062.eqiad.wmnet with reason: host reimage * 12:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:57 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:57 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2161: Migration of db2161.codfw.wmnet completed * 12:54 topranks: shutdown sub-interfaces on cr1-codfw et-1/1/5 for row A/B vlans [[phab:T427301|T427301]] * 12:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 12:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93550 and previous config saved to /var/cache/conftool/dbconfig/20260602-125223-fceratto.json * 12:50 topranks: enable bgp graceful-shutdown in overlay on ssw1-a1-codfw [[phab:T427301|T427301]] * 12:49 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc1061.eqiad.wmnet with OS trixie * 12:48 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt * 12:48 ayounsi@cumin1003: START - Cookbook sre.hosts.remove-downtime for lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt * 12:47 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1062.eqiad.wmnet with OS trixie * 12:45 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93548 and previous config saved to /var/cache/conftool/dbconfig/20260602-124541-fceratto.json * 12:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance * 12:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93547 and previous config saved to /var/cache/conftool/dbconfig/20260602-124512-fceratto.json * 12:43 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc1060.eqiad.wmnet with OS trixie * 12:42 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 12:42 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mc1061.eqiad.wmnet with reason: host reimage * 12:42 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1061.eqiad.wmnet with reason: host reimage * 12:41 topranks: enable bgp graceful-shutdown in underlay on ssw1-a1-codfw [[phab:T427301|T427301]] * 12:35 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mc1060.eqiad.wmnet with reason: host reimage * 12:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P93545 and previous config saved to /var/cache/conftool/dbconfig/20260602-123505-fceratto.json * 12:33 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 12:33 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1060.eqiad.wmnet with reason: host reimage * 12:31 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2049: repool after upgrade * 12:31 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 12:29 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1061.eqiad.wmnet with OS trixie * 12:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2049.codfw.wmnet with OS trixie * 12:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P93542 and previous config saved to /var/cache/conftool/dbconfig/20260602-122459-fceratto.json * 12:24 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1059.eqiad.wmnet with OS trixie * 12:21 XioNoX: reboot lsw1-a3-codfw for software upgrade - [[phab:T427301|T427301]] * 12:20 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1060.eqiad.wmnet with OS trixie * 12:20 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 12:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1058.eqiad.wmnet with OS trixie * 12:17 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 12:16 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296532{{!}}hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533{{!}}hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] (duration: 09m 02s) * 12:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93539 and previous config saved to /var/cache/conftool/dbconfig/20260602-121451-fceratto.json * 12:11 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 12:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2049.codfw.wmnet with reason: host reimage * 12:11 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt with reason: Switch maintenance * 12:10 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2161: Migration of db2161.codfw.wmnet completed * 12:09 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Switch maintenance * 12:09 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296532{{!}}hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533{{!}}hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:08 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1200 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93537 and previous config saved to /var/cache/conftool/dbconfig/20260602-120755-fceratto.json * 12:07 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1059.eqiad.wmnet with reason: host reimage * 12:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance * 12:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93536 and previous config saved to /var/cache/conftool/dbconfig/20260602-120728-fceratto.json * 12:07 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 12:07 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296532{{!}}hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533{{!}}hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] * 12:05 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2049.codfw.wmnet with reason: host reimage * 12:04 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1058.eqiad.wmnet with reason: host reimage * 12:02 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1059.eqiad.wmnet with reason: host reimage * 12:01 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2161.codfw.wmnet with OS trixie * 12:00 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1058.eqiad.wmnet with reason: host reimage * 11:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P93535 and previous config saved to /var/cache/conftool/dbconfig/20260602-115721-fceratto.json * 11:55 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 11:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:55 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 11:53 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 11:53 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 11:53 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:50 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1059.eqiad.wmnet with OS trixie * 11:49 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1057.eqiad.wmnet with OS trixie * 11:49 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2049.codfw.wmnet with OS trixie * 11:48 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2049: Upgrading es2049.codfw.wmnet * 11:48 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2049: Upgrading es2049.codfw.wmnet * 11:47 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:47 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1058.eqiad.wmnet with OS trixie * 11:47 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2056: repool after upgrade * 11:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P93532 and previous config saved to /var/cache/conftool/dbconfig/20260602-114713-fceratto.json * 11:45 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1056.eqiad.wmnet with OS trixie * 11:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2161.codfw.wmnet with reason: host reimage * 11:40 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2161.codfw.wmnet with reason: host reimage * 11:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93531 and previous config saved to /var/cache/conftool/dbconfig/20260602-113705-fceratto.json * 11:33 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1057.eqiad.wmnet with reason: host reimage * 11:30 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1185 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93529 and previous config saved to /var/cache/conftool/dbconfig/20260602-113019-fceratto.json * 11:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance * 11:29 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1056.eqiad.wmnet with reason: host reimage * 11:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1161: Repooling * 11:26 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1161: Repooling * 11:23 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2161.codfw.wmnet with OS trixie * 11:22 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1057.eqiad.wmnet with reason: host reimage * 11:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2161: Upgrading db2161.codfw.wmnet * 11:21 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2161: Upgrading db2161.codfw.wmnet * 11:21 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1056.eqiad.wmnet with reason: host reimage * 11:21 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P93527 and previous config saved to /var/cache/conftool/dbconfig/20260602-111954-fceratto.json * 11:15 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2161 [[phab:T427892|T427892]]', diff saved to https://phabricator.wikimedia.org/P93525 and previous config saved to /var/cache/conftool/dbconfig/20260602-111511-cwilliams.json * 11:12 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2165 to s8 primary [[phab:T427892|T427892]]', diff saved to https://phabricator.wikimedia.org/P93524 and previous config saved to /var/cache/conftool/dbconfig/20260602-111200-cwilliams.json * 11:10 cezmunsta: Starting s8 codfw failover from db2161 to db2165 - [[phab:T427892|T427892]] * 11:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P93523 and previous config saved to /var/cache/conftool/dbconfig/20260602-110947-fceratto.json * 11:09 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1057.eqiad.wmnet with OS trixie * 11:09 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1056.eqiad.wmnet with OS trixie * 11:04 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2165 with weight 0 [[phab:T427892|T427892]]', diff saved to https://phabricator.wikimedia.org/P93522 and previous config saved to /var/cache/conftool/dbconfig/20260602-110420-cwilliams.json * 11:03 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s8 [[phab:T427892|T427892]] * 11:02 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2056: repool after upgrade * 11:01 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93520 and previous config saved to /var/cache/conftool/dbconfig/20260602-105939-fceratto.json * 10:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93519 and previous config saved to /var/cache/conftool/dbconfig/20260602-105239-fceratto.json * 10:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 10:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance * 10:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93518 and previous config saved to /var/cache/conftool/dbconfig/20260602-105202-fceratto.json * 10:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2056.codfw.wmnet with OS trixie * 10:42 moritzm: installing busybox security updates * 10:42 claime: Enabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 10:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P93517 and previous config saved to /var/cache/conftool/dbconfig/20260602-104154-fceratto.json * 10:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P93516 and previous config saved to /var/cache/conftool/dbconfig/20260602-103146-fceratto.json * 10:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2056.codfw.wmnet with reason: host reimage * 10:27 claime: Disabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 10:25 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2056.codfw.wmnet with reason: host reimage * 10:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93515 and previous config saved to /var/cache/conftool/dbconfig/20260602-102139-fceratto.json * 10:09 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2056.codfw.wmnet with OS trixie * 10:08 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2056: Upgrading es2056.codfw.wmnet * 10:08 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2056: Upgrading es2056.codfw.wmnet * 10:08 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:06 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply * 10:06 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply * 09:56 claime: Enabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 09:46 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on cumin2003.codfw.wmnet with reason: in setup * 09:45 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1187: Pooling * 09:37 claime: Running puppet on cp6010 and cp6011 - [[phab:T422937|T422937]] * 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow2004.codfw.wmnet to plain * 09:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93511 and previous config saved to /var/cache/conftool/dbconfig/20260602-093716-fceratto.json * 09:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance * 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow2004.codfw.wmnet to plain * 09:34 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of rpki2003.codfw.wmnet to plain * 09:34 claime: Disabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 09:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of rpki2003.codfw.wmnet to plain * 09:32 moritzm: temporarily remove ganeti2045 from the codfw cluster [[phab:T427357|T427357]] * 09:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1055.eqiad.wmnet with OS trixie * 09:15 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1187: Pooling * 09:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1055.eqiad.wmnet with reason: host reimage * 09:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1187 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93508 and previous config saved to /var/cache/conftool/dbconfig/20260602-091126-fceratto.json * 09:09 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1055.eqiad.wmnet with reason: host reimage * 09:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1187 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93506 and previous config saved to /var/cache/conftool/dbconfig/20260602-090432-fceratto.json * 09:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1187.eqiad.wmnet with reason: Maintenance * 08:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2250.codfw.wmnet with reason: rack A3 maintenance * 08:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:56 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1055.eqiad.wmnet with OS trixie * 08:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:54 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:54 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:53 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 08:52 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 08:51 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 08:50 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 08:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 08:41 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:39 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:37 urbanecm: Reset user email of Barras@votewiki to the one of Barras@SUL * 08:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance * 08:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93505 and previous config saved to /var/cache/conftool/dbconfig/20260602-083033-fceratto.json * 08:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:29 slyngs: IDP, new configuration in preparation for webauthn * 08:20 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P93504 and previous config saved to /var/cache/conftool/dbconfig/20260602-082026-fceratto.json * 08:19 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 08:18 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 08:18 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:17 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296488{{!}}Revert "translate: adding separate read/write endpoints" (T425377)]] (duration: 03m 33s) * 08:16 atsuko@deploy1003: atsuko: Rolling back deployment * 08:16 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2053: repool after upgrade * 08:15 atsuko@deploy1003: atsuko: Backport for [[gerrit:1296488{{!}}Revert "translate: adding separate read/write endpoints" (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:13 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1296488{{!}}Revert "translate: adding separate read/write endpoints" (T425377)]] * 08:11 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:10 marostegui: Install mariadb 10.11.17 on es2053 [[phab:T427345|T427345]] * 08:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P93502 and previous config saved to /var/cache/conftool/dbconfig/20260602-081018-fceratto.json * 08:09 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:09 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: Depool for rack maintenance * 08:03 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296262{{!}}translate: fixing missed variable in credentials formatting closure (T425377)]] (duration: 14m 47s) * 08:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93499 and previous config saved to /var/cache/conftool/dbconfig/20260602-080011-fceratto.json * 07:59 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 07:59 atsuko@deploy1003: atsuko: Rolling back deployment * 07:58 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 07:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1181 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93498 and previous config saved to /var/cache/conftool/dbconfig/20260602-075759-fceratto.json * 07:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1181.eqiad.wmnet with reason: Maintenance * 07:57 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling * 07:50 atsuko@deploy1003: atsuko: Backport for [[gerrit:1296262{{!}}translate: fixing missed variable in credentials formatting closure (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:49 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1296262{{!}}translate: fixing missed variable in credentials formatting closure (T425377)]] * 07:48 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1181: Pooling * 07:47 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1181: Pooling * 07:44 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1181: Reboot * 07:43 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1181: Reboot * 07:42 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1181.eqiad.wmnet with reason: Reboot * 07:41 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 07:41 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:41 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1181: Migration of db1181.eqiad.wmnet completed * 07:40 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294949{{!}}translate: adding separate read/write endpoints (T425377)]] (duration: 21m 01s) * 07:39 atsuko@deploy1003: atsuko: Rolling back deployment * 07:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93490 and previous config saved to /var/cache/conftool/dbconfig/20260602-073904-fceratto.json * 07:32 XioNoX: pfw1-eqiad# delete protocols bgp group Production family inet6 - [[phab:T423384|T423384]] * 07:30 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2053: repool after upgrade * 07:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2158.codfw.wmnet with reason: rack A3 maintenance * 07:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93487 and previous config saved to /var/cache/conftool/dbconfig/20260602-072856-fceratto.json * 07:28 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2158: rack A3 maintenance * 07:28 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2158: rack A3 maintenance * 07:27 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on pc2021.codfw.wmnet with reason: rack A3 maintenance * 07:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc2021: rack A3 maintenance * 07:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 07:25 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache * 07:25 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool pc2021: rack A3 maintenance * 07:23 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2241: Depool for rack maintenance * 07:23 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2241.codfw.wmnet * 07:23 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2241.codfw.wmnet * 07:21 atsuko@deploy1003: atsuko: Backport for [[gerrit:1294949{{!}}translate: adding separate read/write endpoints (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:20 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2053.codfw.wmnet with OS trixie * 07:19 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1294949{{!}}translate: adding separate read/write endpoints (T425377)]] * 07:15 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2241.codfw.wmnet with reason: Depool for rack maintenance * 07:14 marostegui: Install mariadb 10.11.17 on db2186 [[phab:T427345|T427345]] * 07:12 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: Depool for rack maintenance * 07:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2186.codfw.wmnet with reason: upgrade * 07:12 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2241: Depool for rack maintenance * 07:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2053.codfw.wmnet with reason: host reimage * 06:59 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2053.codfw.wmnet with reason: host reimage * 06:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93478 and previous config saved to /var/cache/conftool/dbconfig/20260602-065533-fceratto.json * 06:55 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1181: Migration of db1181.eqiad.wmnet completed * 06:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 06:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1181.eqiad.wmnet with OS trixie * 06:43 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2053.codfw.wmnet with OS trixie * 06:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2053: Upgrading es2053.codfw.wmnet * 06:41 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2053: Upgrading es2053.codfw.wmnet * 06:41 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:37 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 06:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 06:36 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 06:36 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1052: repool after upgrade * 06:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1181.eqiad.wmnet with reason: host reimage * 06:24 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1181.eqiad.wmnet with reason: host reimage * 06:22 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 06:21 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 06:16 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 06:15 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 06:08 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1181.eqiad.wmnet with OS trixie * 06:05 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1181: Upgrading db1181.eqiad.wmnet * 06:05 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1181: Upgrading db1181.eqiad.wmnet * 06:04 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:02 marostegui@dns1004: END - running authdns-update * 06:01 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1181 [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93473 and previous config saved to /var/cache/conftool/dbconfig/20260602-060157-marostegui.json * 06:01 marostegui@dns1004: START - running authdns-update * 06:00 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db1236 to s7 primary and set section read-write [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93472 and previous config saved to /var/cache/conftool/dbconfig/20260602-060041-marostegui.json * 06:00 marostegui@cumin1003: dbctl commit (dc=all): 'Set s7 eqiad as read-only for maintenance - [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93471 and previous config saved to /var/cache/conftool/dbconfig/20260602-060018-marostegui.json * 06:00 marostegui: Starting s7 eqiad failover from db1181 to db1236 - [[phab:T426088|T426088]] * 05:51 marostegui@cumin1003: dbctl commit (dc=all): 'Set db1236 with weight 0 [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93470 and previous config saved to /var/cache/conftool/dbconfig/20260602-055153-marostegui.json * 05:51 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 [[phab:T426088|T426088]] * 05:50 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1052: repool after upgrade * 05:50 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 05:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1052.eqiad.wmnet with OS trixie * 05:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1052.eqiad.wmnet with reason: host reimage * 05:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:22 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1052.eqiad.wmnet with reason: host reimage * 05:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:07 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1052.eqiad.wmnet with OS trixie * 05:06 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1052: Upgrading es1052.eqiad.wmnet * 05:06 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1052: Upgrading es1052.eqiad.wmnet * 05:05 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 04:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 04:49 ryankemper: [[phab:T425007|T425007]] (k8s) created 4 wdqs namespaces on `dse-k8s-codfw`'s `admin_ng` ns: `wdqs-[internal,external]` & `wdqs-[internal,external]-next`; certs issued * 04:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 04:40 ryankemper@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 04:36 ryankemper@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 04:05 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.2 (duration: 05m 33s) == 2026-06-01 == * 23:27 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295963{{!}}Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962{{!}}Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] (duration: 07m 17s) * 23:23 jdlrobson@deploy1003: mfossati, jdlrobson: Continuing with deployment * 23:22 jdlrobson@deploy1003: mfossati, jdlrobson: Backport for [[gerrit:1295963{{!}}Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962{{!}}Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:20 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1295963{{!}}Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962{{!}}Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] * 23:15 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296022{{!}}Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028{{!}}styles: Limit selector to badge client pref (T427407)]] (duration: 09m 33s) * 23:11 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 23:07 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1296022{{!}}Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028{{!}}styles: Limit selector to badge client pref (T427407)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:06 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1296022{{!}}Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028{{!}}styles: Limit selector to badge client pref (T427407)]] * 23:04 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6015.* * 22:36 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296024{{!}}Add maintenance script to scrape SVG render files]] (duration: 06m 22s) * 22:32 reedy@deploy1003: reedy: Continuing with deployment * 22:31 reedy@deploy1003: reedy: Backport for [[gerrit:1296024{{!}}Add maintenance script to scrape SVG render files]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:30 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1296024{{!}}Add maintenance script to scrape SVG render files]] * 22:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 22:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 22:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 21:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 21:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 21:51 sbassett: Deployed updated mitigation for [[phab:T326691|T326691]] * 21:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 21:35 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 21:35 maryum: Deployed security fix for [[phab:T427611|T427611]] * 21:35 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 21:33 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 21:32 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 21:27 maryum: Deployed security fix for [[phab:T427235|T427235]] * 21:13 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296002{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009{{!}}Redirect Special:AccountRecovery to the shared domain (T427692)]] (duration: 09m 20s) * 21:09 catrope@deploy1003: catrope, arlolra: Continuing with deployment * 21:09 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 21:09 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 21:08 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 21:07 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 21:07 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 21:06 catrope@deploy1003: catrope, arlolra: Backport for [[gerrit:1296002{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009{{!}}Redirect Special:AccountRecovery to the shared domain (T427692)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:04 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1296002{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009{{!}}Redirect Special:AccountRecovery to the shared domain (T427692)]] * 20:53 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 20:37 ryankemper@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wdqs1015.eqiad.wmnet with reason: [[phab:T427852|T427852]] hw failure * 20:26 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1285412{{!}}Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531{{!}}Enable AbuseFilter block action on nlwiki (T427384)]] (duration: 07m 48s) * 20:22 catrope@deploy1003: sfaci, xxblackburnxx, catrope: Continuing with deployment * 20:20 catrope@deploy1003: sfaci, xxblackburnxx, catrope: Backport for [[gerrit:1285412{{!}}Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531{{!}}Enable AbuseFilter block action on nlwiki (T427384)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:18 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1285412{{!}}Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531{{!}}Enable AbuseFilter block action on nlwiki (T427384)]] * 20:12 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295504{{!}}passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] (duration: 07m 37s) * 20:08 catrope@deploy1003: catrope: Continuing with deployment * 20:07 catrope@deploy1003: catrope: Backport for [[gerrit:1295504{{!}}passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:05 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1295504{{!}}passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] * 19:48 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 19:47 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 19:47 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 19:46 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 19:46 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 19:45 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 19:01 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: sync * 19:00 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: sync * 18:24 otto@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295950{{!}}mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] (duration: 06m 42s) * 18:20 otto@deploy1003: otto: Continuing with deployment * 18:19 otto@deploy1003: otto: Backport for [[gerrit:1295950{{!}}mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:17 otto@deploy1003: Started scap sync-world: Backport for [[gerrit:1295950{{!}}mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] * 18:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 18:05 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 18:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd2001.codfw.wmnet to plain * 18:02 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 18:02 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd2001.codfw.wmnet to plain * 18:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2003.codfw.wmnet to plain * 18:01 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply * 18:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2003.codfw.wmnet to plain * 17:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 17:58 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 17:53 jasmine@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2006.codfw.wmnet with OS trixie * 17:42 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295976{{!}}nlwiki: change to Wikipedia 25 logo (T424519)]] (duration: 07m 29s) * 17:37 samtar@deploy1003: chlod, samtar: Continuing with deployment * 17:36 samtar@deploy1003: chlod, samtar: Backport for [[gerrit:1295976{{!}}nlwiki: change to Wikipedia 25 logo (T424519)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:34 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1295976{{!}}nlwiki: change to Wikipedia 25 logo (T424519)]] * 17:20 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1236: Update * 17:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd2001.codfw.wmnet to drbd * 17:04 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling * 17:04 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 17:04 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1180: Pooling * 17:03 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 17:03 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling * 17:03 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 16:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd2001.codfw.wmnet to drbd * 16:58 Amir1: drop flaggedrevs tables on wikinews wikis ([[phab:T423577|T423577]]) * 16:57 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 16:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93462 and previous config saved to /var/cache/conftool/dbconfig/20260601-165717-fceratto.json * 16:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93460 and previous config saved to /var/cache/conftool/dbconfig/20260601-164709-fceratto.json * 16:42 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling * 16:37 ryankemper@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,cluster=wdqs-main,service=wdqs-main,name=wdqs1015.eqiad.wmnet * 16:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93458 and previous config saved to /var/cache/conftool/dbconfig/20260601-163701-fceratto.json * 16:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:35 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1236.eqiad.wmnet * 16:35 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1236.eqiad.wmnet * 16:35 ryankemper@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,cluster=wdqs,service=wdqs-main,name=wdqs1015.eqiad.wmnet * 16:34 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update * 16:34 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1236: Update * 16:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1236.eqiad.wmnet with reason: Kernel update [[phab:T426633|T426633]] * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:30 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1236.eqiad.wmnet * 16:30 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1236.eqiad.wmnet * 16:30 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update * 16:29 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1236: Update * 16:29 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update * 16:29 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2003.codfw.wmnet to drbd * 16:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93455 and previous config saved to /var/cache/conftool/dbconfig/20260601-162653-fceratto.json * 16:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 16:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1209: Migration of db1209.eqiad.wmnet completed * 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1236.eqiad.wmnet with reason: Kernel update [[phab:T426633|T426633]] * 16:09 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1236: Update * 16:09 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1236: Update * 16:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:06 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 16:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2003.codfw.wmnet to drbd * 16:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 16:03 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 16:02 moritzm: temporarily remove ganeti2027 from the codfw cluster [[phab:T427357|T427357]] * 15:56 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:56 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.depool (exit_code=97) depool db1224: Pooling * 15:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2005.codfw.wmnet with OS bullseye * 15:53 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1224: Pooling * 15:51 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 15:49 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling * 15:49 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:48 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet * 15:45 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling * 15:44 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2005.codfw.wmnet with reason: host reimage * 15:40 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:40 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling * 15:40 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet * 15:40 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet * 15:40 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet * 15:40 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet * 15:39 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 15:39 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 15:39 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1209: Migration of db1209.eqiad.wmnet completed * 15:39 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:38 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:38 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling * 15:37 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2005.codfw.wmnet with reason: host reimage * 15:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 15:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1209.eqiad.wmnet with OS trixie * 15:28 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295802{{!}}hCaptcha: Raise SiteVerify error threshold to 100]] (duration: 06m 15s) * 15:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93446 and previous config saved to /var/cache/conftool/dbconfig/20260601-152638-fceratto.json * 15:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 15:26 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:25 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet * 15:25 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet * 15:25 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling * 15:25 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:24 kharlan@deploy1003: kharlan: Continuing with deployment * 15:24 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295802{{!}}hCaptcha: Raise SiteVerify error threshold to 100]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:22 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host testvm2005.codfw.wmnet with OS bullseye * 15:22 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295802{{!}}hCaptcha: Raise SiteVerify error threshold to 100]] * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:20 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295946{{!}}hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] (duration: 08m 24s) * 15:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:16 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 15:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1209.eqiad.wmnet with reason: host reimage * 15:14 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1295946{{!}}hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:13 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:12 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1295946{{!}}hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] * 15:10 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1209.eqiad.wmnet with reason: host reimage * 15:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93445 and previous config saved to /var/cache/conftool/dbconfig/20260601-151024-fceratto.json * 15:08 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-reboot (exit_code=0) rolling reboot on A:sessionstore * 15:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93443 and previous config saved to /var/cache/conftool/dbconfig/20260601-150017-fceratto.json * 14:55 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1209.eqiad.wmnet with OS trixie * 14:52 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 14:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1209: Upgrading db1209.eqiad.wmnet * 14:52 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 14:52 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1209: Upgrading db1209.eqiad.wmnet * 14:52 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 14:51 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:51 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 14:50 atsuko@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 14:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93441 and previous config saved to /var/cache/conftool/dbconfig/20260601-145010-fceratto.json * 14:49 atsuko@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 14:49 atsuko@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 14:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:42 atsuko@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 14:41 atsuko@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93440 and previous config saved to /var/cache/conftool/dbconfig/20260601-144002-fceratto.json * 14:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:30 ladsgroup@deploy1003: Synchronized portals: Deploy portals ([[phab:T421797|T421797]]) (duration: 02m 43s) * 14:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:27 ladsgroup@deploy1003: Synchronized portals/wikipedia.org/assets: Deploy portals ([[phab:T421797|T421797]]) (duration: 06m 10s) * 14:25 sukhe@dns1004: END - running authdns-update * 14:23 sukhe@dns1004: START - running authdns-update * 14:22 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 14:21 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:16 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:12 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:12 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:11 Lucas_WMDE: UTC afternoon backport+config window done * 14:10 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295918{{!}}Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] (duration: 11m 06s) * 14:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:05 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, codenamenoreste: Continuing with deployment * 14:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, codenamenoreste: Backport for [[gerrit:1295918{{!}}Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:01 eevans@cumin1003: START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:sessionstore * 13:58 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1295918{{!}}Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] * 13:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:52 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1265.eqiad.wmnet with OS trixie * 13:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93439 and previous config saved to /var/cache/conftool/dbconfig/20260601-133947-fceratto.json * 13:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 13:37 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage * 13:35 atsukoito: restarted pybal.service on lvs2013 * 13:31 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage * 13:31 atsukoito: restarted pybal.service on lvs2014 * 13:24 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs-test2001.codfw.wmnet * 13:24 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:22 atsukoito: restarted pybal.service on lvs1019 * 13:22 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in eqiad/ml-serve-eqiad: maintenance * 13:21 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in eqiad/ml-serve-eqiad: maintenance * 13:20 atsukoito: restarted pybal.service on lvs1020 * 13:20 Msz2001: UTC afternoon backpot+config window done * 13:20 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295875{{!}}Add SetGlobalPreference maintenance script (T427476)]] (duration: 06m 22s) * 13:19 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs-test2001.codfw.wmnet * 13:18 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1265.eqiad.wmnet with OS trixie * 13:18 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:16 mszwarc@deploy1003: mszwarc: Continuing with deployment * 13:15 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1295875{{!}}Add SetGlobalPreference maintenance script (T427476)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:14 atsukoito: sudo cumin 'A:lvs-low-traffic-eqiad' 'systemctl restart pybal.service' * 13:14 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1295875{{!}}Add SetGlobalPreference maintenance script (T427476)]] * 13:12 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295536{{!}}swwiki: Enable the Visual Editor on the project namespace (T427117)]] (duration: 10m 06s) * 13:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93438 and previous config saved to /var/cache/conftool/dbconfig/20260601-130949-fceratto.json * 13:08 mszwarc@deploy1003: codenamenoreste, mszwarc: Continuing with deployment * 13:07 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 13:06 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . * 13:05 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 13:04 mszwarc@deploy1003: codenamenoreste, mszwarc: Backport for [[gerrit:1295536{{!}}swwiki: Enable the Visual Editor on the project namespace (T427117)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:04 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 13:04 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 13:03 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 13:02 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1295536{{!}}swwiki: Enable the Visual Editor on the project namespace (T427117)]] * 12:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93437 and previous config saved to /var/cache/conftool/dbconfig/20260601-125941-fceratto.json * 12:56 dpogorzelski@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=inference,name=eqiad * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'readability' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'logo-detection' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'edit-check' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . * 12:52 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:50 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:49 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93436 and previous config saved to /var/cache/conftool/dbconfig/20260601-124934-fceratto.json * 12:48 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:47 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:46 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:44 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:43 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:42 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:41 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93435 and previous config saved to /var/cache/conftool/dbconfig/20260601-123926-fceratto.json * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:29 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:28 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 12:28 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:27 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:27 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2005.codfw.wmnet to plain * 12:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2005.codfw.wmnet to plain * 12:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet * 12:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 12:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2005.codfw.wmnet to drbd * 12:20 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:17 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:15 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in eqiad/ml-serve-eqiad: maintenance * 12:15 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in eqiad/ml-serve-eqiad: maintenance * 12:11 dpogorzelski@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=inference,name=eqiad * 12:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2005.codfw.wmnet to drbd * 12:05 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet * 12:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 12:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2027.codfw.wmnet * 12:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 11:59 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in eqiad/ml-serve-eqiad: maintenance * 11:59 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in eqiad/ml-serve-eqiad: maintenance * 11:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93434 and previous config saved to /var/cache/conftool/dbconfig/20260601-113911-fceratto.json * 11:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 11:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93433 and previous config saved to /var/cache/conftool/dbconfig/20260601-113843-fceratto.json * 11:37 moritzm: installing Exim security updates * 11:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:32 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:32 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:32 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93432 and previous config saved to /var/cache/conftool/dbconfig/20260601-112835-fceratto.json * 11:25 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 11:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:22 moritzm: installing imagemagick security updates * 11:22 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:22 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:22 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 11:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93430 and previous config saved to /var/cache/conftool/dbconfig/20260601-111827-fceratto.json * 11:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:14 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 11:12 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 11:10 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93429 and previous config saved to /var/cache/conftool/dbconfig/20260601-110820-fceratto.json * 11:04 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 11:01 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1055: repool after upgrade * 11:01 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93427 and previous config saved to /var/cache/conftool/dbconfig/20260601-110121-fceratto.json * 11:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance * 10:54 marostegui@dns1004: END - running authdns-update * 10:52 marostegui@dns1004: START - running authdns-update * 10:48 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1050 to es1 eqiad primary [[phab:T427032|T427032]]', diff saved to https://phabricator.wikimedia.org/P93425 and previous config saved to /var/cache/conftool/dbconfig/20260601-104837-marostegui.json * 10:47 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2055 to es1 codfw primary [[phab:T427032|T427032]]', diff saved to https://phabricator.wikimedia.org/P93424 and previous config saved to /var/cache/conftool/dbconfig/20260601-104739-marostegui.json * 10:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1177: Migration of db1177.eqiad.wmnet completed * 10:40 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet * 10:34 kamila@cumin1003: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet * 10:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93421 and previous config saved to /var/cache/conftool/dbconfig/20260601-103316-fceratto.json * 10:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93418 and previous config saved to /var/cache/conftool/dbconfig/20260601-102308-fceratto.json * 10:16 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1055: repool after upgrade * 10:15 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1055.eqiad.wmnet with OS trixie * 10:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93415 and previous config saved to /var/cache/conftool/dbconfig/20260601-101300-fceratto.json * 10:09 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 10:07 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 10:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93414 and previous config saved to /var/cache/conftool/dbconfig/20260601-100252-fceratto.json * 10:00 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1177: Migration of db1177.eqiad.wmnet completed * 09:58 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1055.eqiad.wmnet with reason: host reimage * 09:56 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 09:54 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 09:53 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1055.eqiad.wmnet with reason: host reimage * 09:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1177.eqiad.wmnet with OS trixie * 09:51 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 09:50 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 09:39 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1055.eqiad.wmnet with OS trixie * 09:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1055: Upgrading es1055.eqiad.wmnet * 09:38 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1055: Upgrading es1055.eqiad.wmnet * 09:37 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1177.eqiad.wmnet with reason: host reimage * 09:31 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1177.eqiad.wmnet with reason: host reimage * 09:17 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1177.eqiad.wmnet with OS trixie * 09:15 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 09:14 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 09:13 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 09:12 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 09:12 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1177: Upgrading db1177.eqiad.wmnet * 09:11 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1177: Upgrading db1177.eqiad.wmnet * 09:11 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93410 and previous config saved to /var/cache/conftool/dbconfig/20260601-090237-fceratto.json * 09:02 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93409 and previous config saved to /var/cache/conftool/dbconfig/20260601-090209-fceratto.json * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P93408 and previous config saved to /var/cache/conftool/dbconfig/20260601-085202-fceratto.json * 08:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P93407 and previous config saved to /var/cache/conftool/dbconfig/20260601-084154-fceratto.json * 08:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93406 and previous config saved to /var/cache/conftool/dbconfig/20260601-083146-fceratto.json * 08:24 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93405 and previous config saved to /var/cache/conftool/dbconfig/20260601-082442-fceratto.json * 08:24 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1168.eqiad.wmnet with reason: Maintenance * 07:58 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295454{{!}}Disable the creation of synthetic main refs in production (T427484)]] (duration: 11m 26s) * 07:56 XioNoX: add no_p2p term to pfw1-codfw BGP_fundraising_export - [[phab:T423384|T423384]] * 07:52 wmde-fisch@deploy1003: lilients, wmde-fisch: Continuing with deployment * 07:51 wmde-fisch@deploy1003: lilients, wmde-fisch: Backport for [[gerrit:1295454{{!}}Disable the creation of synthetic main refs in production (T427484)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:47 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1295454{{!}}Disable the creation of synthetic main refs in production (T427484)]] * 07:45 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294826{{!}}Update VE core submodule to master (9cf5524e7) (T424232)]] (duration: 31m 34s) * 07:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 07:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 07:32 wmde-fisch@deploy1003: wmde-fisch: Continuing with deployment * 07:31 wmde-fisch@deploy1003: wmde-fisch: Backport for [[gerrit:1294826{{!}}Update VE core submodule to master (9cf5524e7) (T424232)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet * 07:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet * 07:13 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1294826{{!}}Update VE core submodule to master (9cf5524e7) (T424232)]] * 06:48 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 06:47 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. == 2026-05-31 == * 02:06 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 30s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-30 == * 16:21 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:21 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:21 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:21 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 06:39 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 06:39 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 06:39 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 06:38 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 27s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-29 == * 23:39 aokoth@cumin1003: END (PASS) - Cookbook sre.vrts.upgrade (exit_code=0) on VRTS host vrts1003.eqiad.wmnet * 23:37 aokoth@cumin1003: START - Cookbook sre.vrts.upgrade on VRTS host vrts1003.eqiad.wmnet * 21:42 catrope@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 21:41 catrope@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 17:40 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295487{{!}}Hide experiment if not active and no assigned group]] (duration: 06m 54s) * 17:35 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 17:34 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1295487{{!}}Hide experiment if not active and no assigned group]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:33 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1295487{{!}}Hide experiment if not active and no assigned group]] * 16:30 jgreen@dns1004: END - running authdns-update * 16:28 jgreen@dns1004: START - running authdns-update * 16:13 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:12 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 15:28 dancy@deploy1003: Installation of scap version "4.267.0" completed for 2 hosts * 15:26 dancy@deploy1003: Installing scap version "4.267.0" for 2 host(s) * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:15 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295466{{!}}GlobalPreferencesHandler: Cast auto-reveal expiry to int (T427625)]] (duration: 07m 58s) * 14:11 kharlan@deploy1003: kharlan: Continuing with deployment * 14:09 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295466{{!}}GlobalPreferencesHandler: Cast auto-reveal expiry to int (T427625)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:07 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295466{{!}}GlobalPreferencesHandler: Cast auto-reveal expiry to int (T427625)]] * 13:53 moritzm: imported OpenJDK 21 21.0.11+10-1~deb12u1 to component/jdk21 (backport of latest Java 21 security release for Bookworm) * 12:09 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader1006.wikimedia.org * 12:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1006.wikimedia.org with OS trixie * 11:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1006.wikimedia.org with reason: host reimage * 11:47 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1006.wikimedia.org with reason: host reimage * 11:36 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader1006.wikimedia.org with OS trixie * 11:15 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:15 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:13 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader1006.wikimedia.org on all recursors * 11:12 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader1006.wikimedia.org on all recursors * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:06 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:00 jmm@cumin2002: START - Cookbook sre.dns.netbox * 11:00 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader1006.wikimedia.org * 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader1005.wikimedia.org * 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1005.wikimedia.org with OS trixie * 10:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1005.wikimedia.org with reason: host reimage * 10:40 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2212: Pooling * 10:37 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1005.wikimedia.org with reason: host reimage * 10:27 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader1005.wikimedia.org with OS trixie * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 10:01 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:59 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:55 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 09:50 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 09:49 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:45 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:44 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup2014.codfw.wmnet with OS bookworm * 09:33 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:20 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup2014.codfw.wmnet with reason: host reimage * 09:12 jynus@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on backup2014.codfw.wmnet with reason: host reimage * 09:10 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 09:10 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 09:03 jelto@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM etherpad2002.codfw.wmnet * 08:59 jelto@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM etherpad2002.codfw.wmnet * 08:59 jelto: gnt-instance modify -B memory=4g,vcpus=1 etherpad2002.codfw.wmnet - [[phab:T427588|T427588]] * 08:54 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2014.codfw.wmnet with OS bookworm * 08:51 jelto@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM etherpad1004.eqiad.wmnet * 08:50 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams-internal: apply * 08:50 jynus@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host backup2014.codfw.wmnet with OS bookworm * 08:49 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams-internal: apply * 08:47 jelto@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM etherpad1004.eqiad.wmnet * 08:46 jelto: gnt-instance modify -B memory=4g,vcpus=1 etherpad1004.eqiad.wmnet - [[phab:T427588|T427588]] * 08:42 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2212: Pooling * 08:42 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 08:39 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2212: Pooling * 08:39 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 08:38 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams-internal: apply * 08:37 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams-internal: apply * 08:37 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams-internal: apply * 08:36 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams-internal: apply * 08:33 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2014.codfw.wmnet with OS bookworm * 08:31 jynus@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host backup2014.codfw.wmnet with OS bookworm * 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader1005.wikimedia.org on all recursors * 08:21 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader1005.wikimedia.org on all recursors * 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 08:21 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 08:18 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 08:17 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 08:16 jmm@cumin2002: START - Cookbook sre.dns.netbox * 08:16 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader1005.wikimedia.org * 08:05 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2212: Pooling * 07:59 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 07:59 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 07:54 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 07:54 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2212.codfw.wmnet * 07:54 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2212.codfw.wmnet * 07:22 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2014.codfw.wmnet with OS bookworm * 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader2006.wikimedia.org * 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader2006.wikimedia.org with OS trixie * 06:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader2006.wikimedia.org with reason: host reimage * 06:53 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader2006.wikimedia.org with reason: host reimage * 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader2006.wikimedia.org with OS trixie * 06:32 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:32 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader2006.wikimedia.org on all recursors * 06:31 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader2006.wikimedia.org on all recursors * 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:31 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:27 jmm@cumin2002: START - Cookbook sre.dns.netbox * 06:27 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader2006.wikimedia.org * 03:01 vriley@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts db1224.eqiad.wmnet * 03:00 vriley@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1224.eqiad.wmnet * 03:00 vriley@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1224.eqiad.wmnet * 02:56 vriley@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1224.eqiad.wmnet * 01:47 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5032.eqsin.wmnet with OS trixie * 01:18 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5032.eqsin.wmnet with reason: host reimage * 01:14 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5032.eqsin.wmnet with reason: host reimage * 00:31 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cp5032.eqsin.wmnet with OS trixie * 00:29 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host cp5032.eqsin.wmnet * 00:23 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 00:22 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply * 00:21 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 00:21 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply == 2026-05-28 == * 23:07 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 23:07 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new ae1.522 interface - pt1979@cumin2002" * 23:07 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new ae1.522 interface - pt1979@cumin2002" * 23:02 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 22:34 andrewbogott: reprepro includedeb trixie-wikimedia /home/andrew/magnum-cluster-api_0.36.6-1~wmf13u2_amd64.deb * 22:31 logmsgbot: dreamyjazz Deployed security patch for [[phab:T426388|T426388]] * 21:33 maryum: Deployed security fix for [[phab:T426867|T426867]] * 21:21 alexsanford: Deployed security fix for [[phab:T426889|T426889]] * 21:07 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host cp5032.eqsin.wmnet * 21:04 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "setup new eqsin vlan - pt1979@cumin2002 - [[phab:T427393|T427393]]" * 21:04 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "setup new eqsin vlan - pt1979@cumin2002 - [[phab:T427393|T427393]]" * 20:48 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295066{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T420336 T427098 T427354 T427082)]], [[gerrit:1295067{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T427082)]] (duration: 07m 34s) * 20:44 arlolra@deploy1003: arlolra: Continuing with deployment * 20:43 arlolra@deploy1003: arlolra: Backport for [[gerrit:1295066{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T420336 T427098 T427354 T427082)]], [[gerrit:1295067{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T427082)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:41 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1295066{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T420336 T427098 T427354 T427082)]], [[gerrit:1295067{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T427082)]] * 20:34 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293805{{!}}Deploy PRV to 7 wikis (T427331)]] (duration: 07m 20s) * 20:30 arlolra@deploy1003: arlolra: Continuing with deployment * 20:29 arlolra@deploy1003: arlolra: Backport for [[gerrit:1293805{{!}}Deploy PRV to 7 wikis (T427331)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:27 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1293805{{!}}Deploy PRV to 7 wikis (T427331)]] * 20:22 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291996{{!}}Replace deprecated Hooks::getInstance (T426981)]], [[gerrit:1294393{{!}}Permissions: Create wmf-officeit group on officewiki]], [[gerrit:1294229{{!}}Deploy IRS Direct Reporting feature to enwiki (T427369)]], [[gerrit:1295039{{!}}Add 2FA enforcement demotion config for phase 2 groups (T423119)]] (duration: 09m 07s) * 20:18 stran@deploy1003: alexsanford, stran, catrope, dreamyjazz: Continuing with deployment * 20:14 stran@deploy1003: alexsanford, stran, catrope, dreamyjazz: Backport for [[gerrit:1291996{{!}}Replace deprecated Hooks::getInstance (T426981)]], [[gerrit:1294393{{!}}Permissions: Create wmf-officeit group on officewiki]], [[gerrit:1294229{{!}}Deploy IRS Direct Reporting feature to enwiki (T427369)]], [[gerrit:1295039{{!}}Add 2FA enforcement demotion config for phase 2 groups (T423119)]] synced to the testservers (see https://wikitech. * 20:13 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5032.eqsin.wmnet with OS trixie * 20:13 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1291996{{!}}Replace deprecated Hooks::getInstance (T426981)]], [[gerrit:1294393{{!}}Permissions: Create wmf-officeit group on officewiki]], [[gerrit:1294229{{!}}Deploy IRS Direct Reporting feature to enwiki (T427369)]], [[gerrit:1295039{{!}}Add 2FA enforcement demotion config for phase 2 groups (T423119)]] * 19:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1018.eqiad.wmnet * 19:27 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1018.eqiad.wmnet * 19:09 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1018.eqiad.wmnet with reason: Kernel reboot * 19:09 brett: Stopping pybal/puppet/downtiming lvs1018.eqiad.wmnet for reboot * 19:05 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1019.eqiad.wmnet * 19:05 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1019.eqiad.wmnet * 18:52 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cp5032.eqsin.wmnet with OS trixie * 18:51 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:51 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change cp5032 IP - pt1979@cumin2002" * 18:51 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change cp5032 IP - pt1979@cumin2002" * 18:47 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 18:40 mutante: planet1003/planet2003 - apt-get upgrade - all pending package upgrades * 18:35 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: Kernel reboot * 18:34 brett: Stopping pybal/puppet/downtiming lvs1019.eqiad.wmnet for reboot and BIOS update/memory self-healing - [[phab:T426109|T426109]] * 18:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs2011.codfw.wmnet * 18:25 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs2011.codfw.wmnet * 18:19 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2011.codfw.wmnet with reason: Kernel reboot * 18:19 brett: Stopping pybal/puppet/downtiming lvs2011.codfw.wmnet for reboot * 18:09 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs2013.codfw.wmnet * 18:06 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs2013.codfw.wmnet * 18:00 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2013.codfw.wmnet with reason: Kernel reboot * 17:57 brett: Stopping pybal/puppet/downtiming lvs2013.codfw.wmnet for reboot * 17:19 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 16:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93393 and previous config saved to /var/cache/conftool/dbconfig/20260528-164514-fceratto.json * 16:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P93392 and previous config saved to /var/cache/conftool/dbconfig/20260528-163507-fceratto.json * 16:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P93391 and previous config saved to /var/cache/conftool/dbconfig/20260528-162459-fceratto.json * 16:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db1224.eqiad.wmnet with reason: unreachable [[phab:T427535|T427535]] * 16:17 swfrench-wmf: reprepro include xdebug_3.4.4-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:17 swfrench-wmf: reprepro include wikidiff2_1.14.1-2+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:17 swfrench-wmf: reprepro include php-yaml_2.2.4-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-xhprof_2.3.10-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-wmerrors_2.0.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-uuid_1.3.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-redis_6.2.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 swfrench-wmf: reprepro include php-pcov_1.0.12-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 swfrench-wmf: reprepro include php-memcached_3.3.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 16:15 swfrench-wmf: reprepro include php-luasandbox_4.1.2-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 16:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93390 and previous config saved to /var/cache/conftool/dbconfig/20260528-161452-fceratto.json * 16:14 swfrench-wmf: reprepro include php-imagick_3.7.0-13+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:14 swfrench-wmf: reprepro include php-excimer_1.2.5-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:09 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:09 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:06 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1251 ([[phab:T426633|T426633]])', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20260528-160646-fceratto.json * 16:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1251.eqiad.wmnet with reason: Maintenance * 16:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93388 and previous config saved to /var/cache/conftool/dbconfig/20260528-160613-fceratto.json * 15:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P93387 and previous config saved to /var/cache/conftool/dbconfig/20260528-155605-fceratto.json * 15:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P93386 and previous config saved to /var/cache/conftool/dbconfig/20260528-154557-fceratto.json * 15:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93385 and previous config saved to /var/cache/conftool/dbconfig/20260528-153550-fceratto.json * 15:27 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1235 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93384 and previous config saved to /var/cache/conftool/dbconfig/20260528-152736-fceratto.json * 15:27 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1235.eqiad.wmnet with reason: Maintenance * 15:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93383 and previous config saved to /var/cache/conftool/dbconfig/20260528-152708-fceratto.json * 15:20 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp5032.eqsin.wmnet with reason: Testing reimaging on new subnet * 15:18 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5032.* * 15:17 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P93382 and previous config saved to /var/cache/conftool/dbconfig/20260528-151701-fceratto.json * 15:17 jhathaway: dmarc ingress test on mx-in1001 * 15:14 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:14 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P93381 and previous config saved to /var/cache/conftool/dbconfig/20260528-150653-fceratto.json * 14:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93380 and previous config saved to /var/cache/conftool/dbconfig/20260528-145646-fceratto.json * 14:56 moritzm: installing nginx security updates * 14:49 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1234 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93379 and previous config saved to /var/cache/conftool/dbconfig/20260528-144936-fceratto.json * 14:49 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 14:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1234.eqiad.wmnet with reason: Maintenance * 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93378 and previous config saved to /var/cache/conftool/dbconfig/20260528-144909-fceratto.json * 14:48 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader2005.wikimedia.org * 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader2005.wikimedia.org with OS trixie * 14:47 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 14:39 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2189.codfw.wmnet * 14:39 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2189.codfw.wmnet * 14:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P93377 and previous config saved to /var/cache/conftool/dbconfig/20260528-143901-fceratto.json * 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader2005.wikimedia.org with reason: host reimage * 14:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P93376 and previous config saved to /var/cache/conftool/dbconfig/20260528-142854-fceratto.json * 14:28 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:28 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader2005.wikimedia.org with reason: host reimage * 14:27 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:19 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294998{{!}}ImageContentLookup: Fix issue created by strict types (T427505)]], [[gerrit:1295001{{!}}Enable hCaptcha for VisualEditor in group 1 (T425940)]] (duration: 11m 29s) * 14:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93375 and previous config saved to /var/cache/conftool/dbconfig/20260528-141846-fceratto.json * 14:15 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1232 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93374 and previous config saved to /var/cache/conftool/dbconfig/20260528-141029-fceratto.json * 14:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1232.eqiad.wmnet with reason: Maintenance * 14:10 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader2005.wikimedia.org with OS trixie * 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93373 and previous config saved to /var/cache/conftool/dbconfig/20260528-141001-fceratto.json * 14:09 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1294998{{!}}ImageContentLookup: Fix issue created by strict types (T427505)]], [[gerrit:1295001{{!}}Enable hCaptcha for VisualEditor in group 1 (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:08 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294998{{!}}ImageContentLookup: Fix issue created by strict types (T427505)]], [[gerrit:1295001{{!}}Enable hCaptcha for VisualEditor in group 1 (T425940)]] * 14:00 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down * 13:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P93371 and previous config saved to /var/cache/conftool/dbconfig/20260528-135951-fceratto.json * 13:58 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp6015.drmrs.wmnet,service=(cdn{{!}}ats-be) * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:55 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader2005.wikimedia.org on all recursors * 13:55 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader2005.wikimedia.org on all recursors * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:55 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P93370 and previous config saved to /var/cache/conftool/dbconfig/20260528-134944-fceratto.json * 13:40 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 13:40 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 13:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93369 and previous config saved to /var/cache/conftool/dbconfig/20260528-133936-fceratto.json * 13:39 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 13:38 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 13:36 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294986{{!}}Image Carousel: check candidate pages (T427336)]] (duration: 06m 40s) * 13:34 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 13:33 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 13:32 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93368 and previous config saved to /var/cache/conftool/dbconfig/20260528-133230-fceratto.json * 13:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance * 13:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93367 and previous config saved to /var/cache/conftool/dbconfig/20260528-133202-fceratto.json * 13:31 mlitn@deploy1003: mlitn: Continuing with deployment * 13:31 mlitn@deploy1003: mlitn: Backport for [[gerrit:1294986{{!}}Image Carousel: check candidate pages (T427336)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:29 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1294986{{!}}Image Carousel: check candidate pages (T427336)]] * 13:22 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 13:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P93366 and previous config saved to /var/cache/conftool/dbconfig/20260528-132155-fceratto.json * 13:21 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 13:17 elukey: clean up a lof ot stale Kafka ACLs on Kafka Jumbo - Details in [[phab:T425528|T425528]] * 13:14 jmm@cumin2002: START - Cookbook sre.dns.netbox * 13:14 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader2005.wikimedia.org * 13:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P93365 and previous config saved to /var/cache/conftool/dbconfig/20260528-131147-fceratto.json * 13:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93364 and previous config saved to /var/cache/conftool/dbconfig/20260528-130139-fceratto.json * 12:54 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1218 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93363 and previous config saved to /var/cache/conftool/dbconfig/20260528-125439-fceratto.json * 12:54 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1218.eqiad.wmnet with reason: Maintenance * 12:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93362 and previous config saved to /var/cache/conftool/dbconfig/20260528-125412-fceratto.json * 12:48 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 12:48 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 12:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P93361 and previous config saved to /var/cache/conftool/dbconfig/20260528-124404-fceratto.json * 12:44 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 12:43 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 12:39 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 12:38 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 12:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P93360 and previous config saved to /var/cache/conftool/dbconfig/20260528-123357-fceratto.json * 12:25 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest1006.eqiad.wmnet with OS trixie * 12:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93359 and previous config saved to /var/cache/conftool/dbconfig/20260528-122349-fceratto.json * 12:15 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93358 and previous config saved to /var/cache/conftool/dbconfig/20260528-121551-fceratto.json * 12:15 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance * 12:15 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie * 12:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93357 and previous config saved to /var/cache/conftool/dbconfig/20260528-121523-fceratto.json * 12:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P93356 and previous config saved to /var/cache/conftool/dbconfig/20260528-120515-fceratto.json * 12:02 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest1006.eqiad.wmnet with OS trixie * 12:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 12:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 12:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 12:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 11:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P93355 and previous config saved to /var/cache/conftool/dbconfig/20260528-115508-fceratto.json * 11:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93354 and previous config saved to /var/cache/conftool/dbconfig/20260528-114500-fceratto.json * 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93353 and previous config saved to /var/cache/conftool/dbconfig/20260528-113635-fceratto.json * 11:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 11:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance * 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93352 and previous config saved to /var/cache/conftool/dbconfig/20260528-113559-fceratto.json * 11:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P93351 and previous config saved to /var/cache/conftool/dbconfig/20260528-112551-fceratto.json * 11:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P93350 and previous config saved to /var/cache/conftool/dbconfig/20260528-111543-fceratto.json * 11:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93349 and previous config saved to /var/cache/conftool/dbconfig/20260528-110536-fceratto.json * 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1195 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93348 and previous config saved to /var/cache/conftool/dbconfig/20260528-105820-fceratto.json * 10:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie * 10:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1195.eqiad.wmnet with reason: Maintenance * 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93347 and previous config saved to /var/cache/conftool/dbconfig/20260528-105753-fceratto.json * 10:56 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 10:55 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 10:55 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 10:55 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 10:50 moritzm: update trixie netboot image for 13.5 point release [[phab:T427072|T427072]] * 10:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P93346 and previous config saved to /var/cache/conftool/dbconfig/20260528-104745-fceratto.json * 10:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P93345 and previous config saved to /var/cache/conftool/dbconfig/20260528-103738-fceratto.json * 10:29 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P13724 # [[phab:T406971|T406971]] * 10:28 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P14223 # [[phab:T422264|T422264]] * 10:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93344 and previous config saved to /var/cache/conftool/dbconfig/20260528-102730-fceratto.json * 10:26 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P1748 # [[phab:T422392|T422392]] * 10:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93343 and previous config saved to /var/cache/conftool/dbconfig/20260528-101900-fceratto.json * 10:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance * 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93342 and previous config saved to /var/cache/conftool/dbconfig/20260528-101829-fceratto.json * 10:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P93341 and previous config saved to /var/cache/conftool/dbconfig/20260528-100822-fceratto.json * 09:59 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290687{{!}}stream: webrequest.page_view (T426092 T426091)]] (duration: 06m 41s) * 09:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P93340 and previous config saved to /var/cache/conftool/dbconfig/20260528-095814-fceratto.json * 09:55 javiermonton@deploy1003: javiermonton: Continuing with deployment * 09:54 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1290687{{!}}stream: webrequest.page_view (T426092 T426091)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:52 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1290687{{!}}stream: webrequest.page_view (T426092 T426091)]] * 09:48 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294243{{!}}Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937{{!}}CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] (duration: 07m 37s) * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93339 and previous config saved to /var/cache/conftool/dbconfig/20260528-094807-fceratto.json * 09:44 dreamyjazz@deploy1003: dreamyjazz, stran: Continuing with deployment * 09:44 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:43 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:43 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:43 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:42 dreamyjazz@deploy1003: dreamyjazz, stran: Backport for [[gerrit:1294243{{!}}Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937{{!}}CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294243{{!}}Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937{{!}}CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] * 09:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93338 and previous config saved to /var/cache/conftool/dbconfig/20260528-093920-fceratto.json * 09:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: Maintenance * 09:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93337 and previous config saved to /var/cache/conftool/dbconfig/20260528-093849-fceratto.json * 09:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P93336 and previous config saved to /var/cache/conftool/dbconfig/20260528-092842-fceratto.json * 09:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance * 09:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93335 and previous config saved to /var/cache/conftool/dbconfig/20260528-092239-fceratto.json * 09:22 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pki-root1001.eqiad.wmnet * 09:22 elukey@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:22 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki-root1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1003" * 09:22 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki-root1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1003" * 09:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:18 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 09:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P93334 and previous config saved to /var/cache/conftool/dbconfig/20260528-091834-fceratto.json * 09:18 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 09:18 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 09:17 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1165: Reboot completed * 09:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 09:17 elukey@cumin1003: START - Cookbook sre.dns.netbox * 09:14 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 09:13 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:13 elukey@cumin1003: START - Cookbook sre.hosts.decommission for hosts pki-root1001.eqiad.wmnet * 09:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P93332 and previous config saved to /var/cache/conftool/dbconfig/20260528-091231-fceratto.json * 09:09 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:09 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93331 and previous config saved to /var/cache/conftool/dbconfig/20260528-090826-fceratto.json * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P93329 and previous config saved to /var/cache/conftool/dbconfig/20260528-090224-fceratto.json * 09:02 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Deploying to prod (duration: 02m 31s) * 09:01 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2216 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93328 and previous config saved to /var/cache/conftool/dbconfig/20260528-090114-fceratto.json * 09:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2216.codfw.wmnet with reason: Maintenance * 09:00 joal@deploy1003: Finished deploy [analytics/refinery@878cb24] (thin): Regular analytics weekly train THIN - 2[analytics/refinery@878cb24a] (duration: 02m 08s) * 08:59 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Deploying to prod * 08:58 joal@deploy1003: Started deploy [analytics/refinery@878cb24] (thin): Regular analytics weekly train THIN - 2[analytics/refinery@878cb24a] * 08:57 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Testing on backup host (duration: 00m 53s) * 08:56 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Testing on backup host * 08:56 joal@deploy1003: Finished deploy [analytics/refinery@878cb24]: Regular analytics weekly train - 2 [analytics/refinery@878cb24a] (duration: 06m 54s) * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93327 and previous config saved to /var/cache/conftool/dbconfig/20260528-085216-fceratto.json * 08:50 XioNoX: cr1-codfw# delete protocols bgp group fundraising family inet6 - [[phab:T423384|T423384]] * 08:49 joal@deploy1003: Started deploy [analytics/refinery@878cb24]: Regular analytics weekly train - 2 [analytics/refinery@878cb24a] * 08:49 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294925{{!}}hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] (duration: 09m 20s) * 08:49 joal@deploy1003: Finished deploy [analytics/refinery@878cb24] (hadoop-test): Regular analytics weekly train TEST -2 [analytics/refinery@878cb24a] (duration: 02m 00s) * 08:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93326 and previous config saved to /var/cache/conftool/dbconfig/20260528-084906-fceratto.json * 08:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1209.eqiad.wmnet with reason: Maintenance * 08:48 slyngshede@dns1004: END - running authdns-update * 08:47 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1165: Reboot completed * 08:47 joal@deploy1003: Started deploy [analytics/refinery@878cb24] (hadoop-test): Regular analytics weekly train TEST -2 [analytics/refinery@878cb24a] * 08:47 slyngs: Upgrade IDP to CAS 7.3.7.1 * 08:46 slyngshede@dns1004: START - running authdns-update * 08:45 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 08:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93324 and previous config saved to /var/cache/conftool/dbconfig/20260528-084149-fceratto.json * 08:41 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1294925{{!}}hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294925{{!}}hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] * 08:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet * 08:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet * 08:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93323 and previous config saved to /var/cache/conftool/dbconfig/20260528-083504-fceratto.json * 08:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1025].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 08:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1165.eqiad.wmnet with reason: Maintenance * 08:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93322 and previous config saved to /var/cache/conftool/dbconfig/20260528-083331-fceratto.json * 08:24 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1209: Test * 08:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P93320 and previous config saved to /var/cache/conftool/dbconfig/20260528-082324-fceratto.json * 08:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2189: repool after crash * 08:17 slyngshede@dns1004: END - running authdns-update * 08:16 slyngshede@dns1004: START - running authdns-update * 08:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P93318 and previous config saved to /var/cache/conftool/dbconfig/20260528-081316-fceratto.json * 08:10 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 08:09 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1209: Test * 08:05 hashar@deploy1003: Finished deploy [integration/docroot@2a51016]: build: update dependencies + eslint fix in comment. f021d3f..2a51016 (duration: 00m 13s) * 08:05 hashar@deploy1003: Started deploy [integration/docroot@2a51016]: build: update dependencies + eslint fix in comment. f021d3f..2a51016 * 08:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93315 and previous config saved to /var/cache/conftool/dbconfig/20260528-080309-fceratto.json * 07:56 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93314 and previous config saved to /var/cache/conftool/dbconfig/20260528-075631-fceratto.json * 07:56 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020,1022-1023].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 07:56 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance * 07:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93313 and previous config saved to /var/cache/conftool/dbconfig/20260528-075521-fceratto.json * 07:47 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab replica * 07:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93311 and previous config saved to /var/cache/conftool/dbconfig/20260528-074513-fceratto.json * 07:37 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2189: repool after crash * 07:36 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab replica * 07:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93309 and previous config saved to /var/cache/conftool/dbconfig/20260528-073506-fceratto.json * 07:34 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica * 07:29 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294808{{!}}Don't run the click intent experiment on mobile (T426743)]] (duration: 06m 29s) * 07:25 wmde-fisch@deploy1003: thiemowmde, wmde-fisch: Continuing with deployment * 07:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93308 and previous config saved to /var/cache/conftool/dbconfig/20260528-072458-fceratto.json * 07:24 wmde-fisch@deploy1003: thiemowmde, wmde-fisch: Backport for [[gerrit:1294808{{!}}Don't run the click intent experiment on mobile (T426743)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:24 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica * 07:23 tgr@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwikisource --logwiki=metawiki Ioed Renamed_user_4232d41570b9e8f46ef150e5e360e446 # [[phab:T427459|T427459]] * 07:22 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1294808{{!}}Don't run the click intent experiment on mobile (T426743)]] * 07:20 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1270986{{!}}Update wikimania wordmark for 2026 (T413331)]] (duration: 06m 54s) * 07:18 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93307 and previous config saved to /var/cache/conftool/dbconfig/20260528-071836-fceratto.json * 07:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1264.eqiad.wmnet with reason: Maintenance * 07:16 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1167: Reboot completed * 07:16 wmde-fisch@deploy1003: wmde-fisch, robertsky: Continuing with deployment * 07:15 wmde-fisch@deploy1003: wmde-fisch, robertsky: Backport for [[gerrit:1270986{{!}}Update wikimania wordmark for 2026 (T413331)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:13 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1270986{{!}}Update wikimania wordmark for 2026 (T413331)]] * 07:11 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289898{{!}}Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] (duration: 07m 15s) * 07:07 wmde-fisch@deploy1003: wmde-fisch, arthurtaylor: Continuing with deployment * 07:06 wmde-fisch@deploy1003: wmde-fisch, arthurtaylor: Backport for [[gerrit:1289898{{!}}Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:04 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1289898{{!}}Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] * 06:43 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1167: Reboot completed * 06:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93303 and previous config saved to /var/cache/conftool/dbconfig/20260528-064217-fceratto.json * 06:33 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93302 and previous config saved to /var/cache/conftool/dbconfig/20260528-063357-fceratto.json * 06:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 06:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance * 06:25 hashar: Restarting CI Jenkins for plugins upgrades * 06:16 fceratto@dns1005: END - running authdns-update * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1209 [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93301 and previous config saved to /var/cache/conftool/dbconfig/20260528-061609-fceratto.json * 06:14 fceratto@dns1005: START - running authdns-update * 06:11 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1193 to s8 primary and set section read-write [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93300 and previous config saved to /var/cache/conftool/dbconfig/20260528-061138-fceratto.json * 06:10 fceratto@cumin1003: dbctl commit (dc=all): 'Set s8 eqiad as read-only for maintenance - [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93299 and previous config saved to /var/cache/conftool/dbconfig/20260528-061048-fceratto.json * 06:10 federico3: Starting s8 eqiad failover from db1209 to db1193 - [[phab:T426095|T426095]] * 06:04 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1193 with weight 0 [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93298 and previous config saved to /var/cache/conftool/dbconfig/20260528-060412-fceratto.json * 06:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s8 [[phab:T426095|T426095]] * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 41s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:53 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 00:53 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new subnet in eqsin - pt1979@cumin2002" * 00:53 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new subnet in eqsin - pt1979@cumin2002" * 00:49 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 00:25 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294470{{!}}Activate conductwiki (T426984)]] (duration: 07m 12s) * 00:21 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 00:20 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1294470{{!}}Activate conductwiki (T426984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 00:18 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1294470{{!}}Activate conductwiki (T426984)]] * 00:12 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294438{{!}}Init conductwiki (T426984)]] (duration: 07m 25s) * 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 00:08 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 00:06 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1294438{{!}}Init conductwiki (T426984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 00:04 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1294438{{!}}Init conductwiki (T426984)]] * 00:04 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] == 2026-05-27 == * 23:13 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294432{{!}}Exclude more content from selection (T426308)]], [[gerrit:1285523{{!}}Remove MinervaNightMode config after skin cleanup (T426689)]] (duration: 08m 42s) * 23:09 jdlrobson@deploy1003: jdlrobson, h2o, egardner: Continuing with deployment * 23:06 jdlrobson@deploy1003: jdlrobson, h2o, egardner: Backport for [[gerrit:1294432{{!}}Exclude more content from selection (T426308)]], [[gerrit:1285523{{!}}Remove MinervaNightMode config after skin cleanup (T426689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:04 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294432{{!}}Exclude more content from selection (T426308)]], [[gerrit:1285523{{!}}Remove MinervaNightMode config after skin cleanup (T426689)]] * 22:58 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294435{{!}}passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] (duration: 07m 49s) * 22:55 ladsgroup@cumin1003: END (PASS) - Cookbook sre.mysql.sanitarium_restart (exit_code=0) * 22:54 catrope@deploy1003: catrope: Continuing with deployment * 22:52 catrope@deploy1003: catrope: Backport for [[gerrit:1294435{{!}}passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:50 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1294435{{!}}passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] * 22:46 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294360{{!}}Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322{{!}}Thumbnails are not being optimized in large mode (T427237)]] (duration: 06m 54s) * 22:42 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 22:41 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1294360{{!}}Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322{{!}}Thumbnails are not being optimized in large mode (T427237)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:40 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart * 22:40 ladsgroup@cumin1003: END (FAIL) - Cookbook sre.mysql.sanitarium_restart (exit_code=99) * 22:40 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart * 22:39 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294360{{!}}Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322{{!}}Thumbnails are not being optimized in large mode (T427237)]] * 22:39 ladsgroup@deploy1003: Finished scap sync-world: Add conduct.wikimedia.org ([[phab:T426984|T426984]]) (duration: 07m 16s) * 22:35 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 22:34 ladsgroup@deploy1003: ladsgroup: Add conduct.wikimedia.org ([[phab:T426984|T426984]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:33 ladsgroup@deploy1003: Started scap sync-world: Add conduct.wikimedia.org ([[phab:T426984|T426984]]) * 22:13 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294370{{!}}Carousel only on articles (T427336)]] (duration: 10m 00s) * 22:09 egardner@deploy1003: egardner: Continuing with deployment * 22:05 egardner@deploy1003: egardner: Backport for [[gerrit:1294370{{!}}Carousel only on articles (T427336)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:03 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1294370{{!}}Carousel only on articles (T427336)]] * 21:37 bking@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 15 days, 0:00:00 on relforge[1008-1010].eqiad.wmnet with reason: non-production environment * 21:20 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 21:20 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 21:20 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 21:19 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 21:04 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1288370{{!}}Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819{{!}}Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] (duration: 07m 38s) * 20:59 ebernhardson@deploy1003: matmarex, ebernhardson, pppery: Continuing with deployment * 20:58 ebernhardson@deploy1003: matmarex, ebernhardson, pppery: Backport for [[gerrit:1288370{{!}}Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819{{!}}Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:56 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1288370{{!}}Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819{{!}}Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] * 20:51 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294373{{!}}identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374{{!}}Revert^2 "cirrus: AB test query suggester variants" (T407432)]] (duration: 07m 30s) * 20:47 ebernhardson@deploy1003: ebernhardson: Continuing with deployment * 20:46 ebernhardson@deploy1003: ebernhardson: Backport for [[gerrit:1294373{{!}}identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374{{!}}Revert^2 "cirrus: AB test query suggester variants" (T407432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:44 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294373{{!}}identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374{{!}}Revert^2 "cirrus: AB test query suggester variants" (T407432)]] * 20:43 swfrench-wmf: reprepro include dh-php_5.5+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 20:39 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts lvs1016.eqiad.wmnet * 20:39 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:39 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brett@cumin2002" * 20:38 swfrench-wmf: reprepro include php-defaults_94+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 20:37 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brett@cumin2002" * 20:31 brett@cumin2002: START - Cookbook sre.dns.netbox * 20:27 swfrench-wmf: reprepro include php8.3_8.3.31-1+wmf12u2 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 20:25 brett@cumin2002: START - Cookbook sre.hosts.decommission for hosts lvs1016.eqiad.wmnet * 20:25 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294342{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344{{!}}frwiki: restrict Article Guidance experiment to junior editors (T426871)]] (duration: 08m 11s) * 20:21 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host lvs1016.eqiad.wmnet with OS bullseye * 20:21 sbisson@deploy1003: sbisson: Continuing with deployment * 20:20 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1020.eqiad.wmnet * 20:19 sbisson@deploy1003: sbisson: Backport for [[gerrit:1294342{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344{{!}}frwiki: restrict Article Guidance experiment to junior editors (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be v * 20:17 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294342{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344{{!}}frwiki: restrict Article Guidance experiment to junior editors (T426871)]] * 20:14 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet * 20:05 cmooney@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 12355 * 20:04 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 12355 * 19:51 brett@cumin2002: START - Cookbook sre.hosts.reimage for host lvs1016.eqiad.wmnet with OS bullseye * 19:48 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 19:45 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 19:45 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 19:32 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp6016.drmrs.wmnet,cp[1112,1114].eqiad.wmnet,cp[5024,5031-5032].eqsin.wmnet<nowiki>}</nowiki> and A:cp * 19:32 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5032.eqsin.wmnet * 19:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 19:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 19:01 joal@deploy1003: Finished deploy [analytics/refinery@96cf761] (thin): Regular analytics weekly train THIN [analytics/refinery@96cf761f] (duration: 02m 08s) * 18:59 joal@deploy1003: Started deploy [analytics/refinery@96cf761] (thin): Regular analytics weekly train THIN [analytics/refinery@96cf761f] * 18:58 joal@deploy1003: Finished deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] (duration: 05m 01s) * 18:53 joal@deploy1003: Started deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] * 18:53 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294376{{!}}Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375{{!}}auth: Mark the hidden token field used for reauth as skippable (T427398)]] (duration: 07m 41s) * 18:49 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5031.eqsin.wmnet * 18:49 catrope@deploy1003: catrope: Continuing with deployment * 18:47 catrope@deploy1003: catrope: Backport for [[gerrit:1294376{{!}}Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375{{!}}auth: Mark the hidden token field used for reauth as skippable (T427398)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:45 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1294376{{!}}Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375{{!}}auth: Mark the hidden token field used for reauth as skippable (T427398)]] * 18:40 joal@deploy1003: Finished deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] (duration: 01m 05s) * 18:39 joal@deploy1003: Started deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] * 18:37 joal@deploy1003: Finished deploy [analytics/refinery@96cf761] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96cf761f] (duration: 02m 04s) * 18:35 joal@deploy1003: Started deploy [analytics/refinery@96cf761] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96cf761f] * 18:29 swfrench@deploy1003: Finished scap sync-world: Helmfile-only deployment to clean up unused mesh listeners (duration: 06m 12s) * 18:25 swfrench@deploy1003: swfrench: Continuing with deployment * 18:24 swfrench@deploy1003: swfrench: Helmfile-only deployment to clean up unused mesh listeners synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:23 swfrench@deploy1003: Started scap sync-world: Helmfile-only deployment to clean up unused mesh listeners * 18:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93296 and previous config saved to /var/cache/conftool/dbconfig/20260527-181923-fceratto.json * 18:13 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 18:12 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 18:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 18:11 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 18:11 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 18:10 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 18:10 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 18:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93295 and previous config saved to /var/cache/conftool/dbconfig/20260527-180915-fceratto.json * 18:09 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 18:09 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293776{{!}}ProductionServices: Revert to discovery shellbox listeners]] (duration: 10m 24s) * 18:08 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1017.eqiad.wmnet * 18:08 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1017.eqiad.wmnet * 18:07 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5024.eqsin.wmnet * 18:03 swfrench@deploy1003: swfrench: Continuing with deployment * 18:02 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 18:02 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 18:02 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 18:01 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:01 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 18:01 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 18:00 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 18:00 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293776{{!}}ProductionServices: Revert to discovery shellbox listeners]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:00 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93294 and previous config saved to /var/cache/conftool/dbconfig/20260527-175908-fceratto.json * 17:58 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293776{{!}}ProductionServices: Revert to discovery shellbox listeners]] * 17:55 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93293 and previous config saved to /var/cache/conftool/dbconfig/20260527-174900-fceratto.json * 17:43 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293774{{!}}ProductionServices: Temporarily use shellbox in codfw]] (duration: 15m 01s) * 17:38 swfrench@deploy1003: swfrench: Continuing with deployment * 17:31 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293774{{!}}ProductionServices: Temporarily use shellbox in codfw]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:28 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293774{{!}}ProductionServices: Temporarily use shellbox in codfw]] * 17:25 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp1114.eqiad.wmnet * 17:18 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:15 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:15 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:14 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:05 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293775{{!}}ProductionServices: Temporarily use shellbox in eqiad]] (duration: 08m 44s) * 17:00 swfrench@deploy1003: swfrench: Continuing with deployment * 16:58 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293775{{!}}ProductionServices: Temporarily use shellbox in eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:56 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293775{{!}}ProductionServices: Temporarily use shellbox in eqiad]] * 16:53 atsuko@dns1004: END - running authdns-update * 16:51 atsuko@dns1004: START - running authdns-update * 16:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93292 and previous config saved to /var/cache/conftool/dbconfig/20260527-164846-fceratto.json * 16:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1264.eqiad.wmnet with reason: Maintenance * 16:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93291 and previous config saved to /var/cache/conftool/dbconfig/20260527-164815-fceratto.json * 16:43 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp1112.eqiad.wmnet * 16:41 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1017.eqiad.wmnet with reason: Setting up * 16:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P93290 and previous config saved to /var/cache/conftool/dbconfig/20260527-163808-fceratto.json * 16:37 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2163: Repooling after testing patch * 16:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P93287 and previous config saved to /var/cache/conftool/dbconfig/20260527-162800-fceratto.json * 16:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93285 and previous config saved to /var/cache/conftool/dbconfig/20260527-161753-fceratto.json * 16:14 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 16:13 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 16:13 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 16:12 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 16:11 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93284 and previous config saved to /var/cache/conftool/dbconfig/20260527-161101-fceratto.json * 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: Maintenance * 16:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93283 and previous config saved to /var/cache/conftool/dbconfig/20260527-161034-fceratto.json * 16:10 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 16:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1178: Recovering from failure in cookbook * 16:10 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 16:05 sukhe@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host durum5003.eqsin.wmnet with OS trixie * 16:03 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp6016.drmrs.wmnet * 16:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220', diff saved to https://phabricator.wikimedia.org/P93280 and previous config saved to /var/cache/conftool/dbconfig/20260527-160027-fceratto.json * 15:59 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1017.eqiad.wmnet * 15:53 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2163.codfw.wmnet * 15:53 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2163.codfw.wmnet * 15:52 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs1017.eqiad.wmnet * 15:52 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Repooling after testing patch * 15:52 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp6016.drmrs.wmnet,cp[1112,1114].eqiad.wmnet,cp[5024,5031-5032].eqsin.wmnet<nowiki>}</nowiki> and A:cp * 15:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2163: Testing cookbook * 15:50 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2163: Testing cookbook * 15:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220', diff saved to https://phabricator.wikimedia.org/P93276 and previous config saved to /var/cache/conftool/dbconfig/20260527-155019-fceratto.json * 15:45 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:45 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93274 and previous config saved to /var/cache/conftool/dbconfig/20260527-154011-fceratto.json * 15:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 15:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2163: Migration of db2163.codfw.wmnet completed * 15:32 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Migration of db2163.codfw.wmnet completed * 15:32 cwilliams@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2163: Migration of db2163.codfw.wmnet completed * 15:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1178: Recovering from failure in cookbook * 15:22 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1178.eqiad.wmnet * 15:22 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1178.eqiad.wmnet * 15:19 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 15:19 cdanis: 💙cdanis@cp4047.ulsfo.wmnet ~ 🕦☕ sudo apt install lua5.4-ciderbloom lua5.4-ciderbloom-dbgsym * 15:13 cdanis: 💙cdanis@cp5026.eqsin.wmnet ~ 🕚☕ sudo apt install lua5.4-ciderbloom lua5.4-ciderbloom-dbgsym * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:11 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:11 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:11 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Icinga wait failed during run * 15:10 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:10 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:10 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:09 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:09 cdanis: 💔cdanis@apt1002.wikimedia.org ~ 🕚☕ sudo -i reprepro --component main --restrict cidergrinder update trixie-wikimedia * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:05 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93268 and previous config saved to /var/cache/conftool/dbconfig/20260527-150508-fceratto.json * 15:05 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1220.eqiad.wmnet with reason: Maintenance * 15:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93267 and previous config saved to /var/cache/conftool/dbconfig/20260527-150438-fceratto.json * 14:59 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Migration of db2163.codfw.wmnet completed * 14:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P93264 and previous config saved to /var/cache/conftool/dbconfig/20260527-145430-fceratto.json * 14:54 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 14:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2163.codfw.wmnet with OS trixie * 14:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply * 14:50 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply * 14:46 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290926{{!}}Re-enable ReadingLists QuickSurvey (T426781)]] (duration: 08m 32s) * 14:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1178.eqiad.wmnet with OS trixie * 14:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P93263 and previous config saved to /var/cache/conftool/dbconfig/20260527-144423-fceratto.json * 14:42 aude@deploy1003: aude: Continuing with deployment * 14:40 aude@deploy1003: aude: Backport for [[gerrit:1290926{{!}}Re-enable ReadingLists QuickSurvey (T426781)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:38 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db2189.codfw.wmnet with reason: crashed [[phab:T427376|T427376]] * 14:38 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1290926{{!}}Re-enable ReadingLists QuickSurvey (T426781)]] * 14:35 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290924{{!}}Make logging of title and page ID optional (T426457)]] (duration: 11m 30s) * 14:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93262 and previous config saved to /var/cache/conftool/dbconfig/20260527-143416-fceratto.json * 14:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2163.codfw.wmnet with reason: host reimage * 14:29 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2163.codfw.wmnet with reason: host reimage * 14:29 aude@deploy1003: aude: Continuing with deployment * 14:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1178.eqiad.wmnet with reason: host reimage * 14:27 aude@deploy1003: aude: Backport for [[gerrit:1290924{{!}}Make logging of title and page ID optional (T426457)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:27 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93260 and previous config saved to /var/cache/conftool/dbconfig/20260527-142659-fceratto.json * 14:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1179.eqiad.wmnet with reason: Maintenance * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:23 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1290924{{!}}Make logging of title and page ID optional (T426457)]] * 14:22 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1178.eqiad.wmnet with reason: host reimage * 14:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1033.eqiad.wmnet with reason: Maintenance * 14:18 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294247{{!}}Update Direct Reporting email (T427358)]] (duration: 33m 01s) * 14:10 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2163.codfw.wmnet with OS trixie * 14:09 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1178.eqiad.wmnet with OS trixie * 14:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2163: Upgrading db2163.codfw.wmnet * 14:08 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2163: Upgrading db2163.codfw.wmnet * 14:08 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1178: Upgrading db1178.eqiad.wmnet * 14:07 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1178: Upgrading db1178.eqiad.wmnet * 14:06 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:06 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:06 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:06 stran@deploy1003: stran: Continuing with deployment * 14:02 stran@deploy1003: stran: Backport for [[gerrit:1294247{{!}}Update Direct Reporting email (T427358)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:56 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 13:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2164: Migration of db2164.codfw.wmnet completed * 13:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1192: Migration of db1192.eqiad.wmnet completed * 13:45 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1294247{{!}}Update Direct Reporting email (T427358)]] * 13:40 phuedx@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294217{{!}}ext.wikimediaEvents: Add hoisting error detection test (T427092)]] (duration: 11m 35s) * 13:36 phuedx@deploy1003: phuedx: Continuing with deployment * 13:30 phuedx@deploy1003: phuedx: Backport for [[gerrit:1294217{{!}}ext.wikimediaEvents: Add hoisting error detection test (T427092)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:28 phuedx@deploy1003: Started scap sync-world: Backport for [[gerrit:1294217{{!}}ext.wikimediaEvents: Add hoisting error detection test (T427092)]] * 13:21 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290781{{!}}mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264{{!}}MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] (duration: 13m 23s) * 13:15 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2189: Test * 13:15 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2189: Test * 13:15 mlitn@deploy1003: krinkle, mlitn: Continuing with deployment * 13:13 mlitn@deploy1003: krinkle, mlitn: Backport for [[gerrit:1290781{{!}}mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264{{!}}MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 13:10 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2164: Migration of db2164.codfw.wmnet completed * 13:08 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1290781{{!}}mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264{{!}}MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] * 13:06 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 13:05 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db2212.codfw.wmnet with reason: failed to reboot [[phab:T427388|T427388]] [[phab:T426633|T426633]] * 13:05 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1192: Migration of db1192.eqiad.wmnet completed * 13:01 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2164.codfw.wmnet with OS trixie * 12:57 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1192.eqiad.wmnet with OS trixie * 12:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2164.codfw.wmnet with reason: host reimage * 12:40 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1192.eqiad.wmnet with reason: host reimage * 12:40 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2164.codfw.wmnet with reason: host reimage * 12:35 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1192.eqiad.wmnet with reason: host reimage * 12:28 Amir1: deleting binlogs older than a year * 12:22 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2164.codfw.wmnet with OS trixie * 12:21 cmooney@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 36692 * 12:21 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1192.eqiad.wmnet with OS trixie * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1077 * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1080 * 12:20 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1077 * 12:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2164: Upgrading db2164.codfw.wmnet * 12:20 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 36692 * 12:20 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1080 * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1078 * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1079 * 12:20 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2164: Upgrading db2164.codfw.wmnet * 12:19 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:19 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1079 * 12:19 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1078 * 12:19 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:19 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1192: Upgrading db1192.eqiad.wmnet * 12:19 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:18 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1192: Upgrading db1192.eqiad.wmnet * 12:18 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:15 jclark@cumin1003: START - Cookbook sre.dns.netbox * 12:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2165: Migration of db2165.codfw.wmnet completed * 12:14 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:14 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:14 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:12 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db2189: Test * 12:11 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2189: Test * 12:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1193: Migration of db1193.eqiad.wmnet completed * 12:09 jclark@cumin1003: START - Cookbook sre.dns.netbox * 12:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93243 and previous config saved to /var/cache/conftool/dbconfig/20260527-120452-fceratto.json * 12:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: Maintenance * 12:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93242 and previous config saved to /var/cache/conftool/dbconfig/20260527-120205-fceratto.json * 12:01 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 11:58 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 11:58 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "is everything alright? /cc effie - ayounsi@cumin1003" * 11:58 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "is everything alright? /cc effie - ayounsi@cumin1003" * 11:56 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 11:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P93239 and previous config saved to /var/cache/conftool/dbconfig/20260527-115157-fceratto.json * 11:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P93237 and previous config saved to /var/cache/conftool/dbconfig/20260527-114149-fceratto.json * 11:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93235 and previous config saved to /var/cache/conftool/dbconfig/20260527-113142-fceratto.json * 11:29 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2165: Migration of db2165.codfw.wmnet completed * 11:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1193: Migration of db1193.eqiad.wmnet completed * 11:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93231 and previous config saved to /var/cache/conftool/dbconfig/20260527-112327-fceratto.json * 11:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2188.codfw.wmnet with reason: Maintenance * 11:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93230 and previous config saved to /var/cache/conftool/dbconfig/20260527-112257-fceratto.json * 11:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2165.codfw.wmnet with OS trixie * 11:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1193.eqiad.wmnet with OS trixie * 11:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P93229 and previous config saved to /var/cache/conftool/dbconfig/20260527-111250-fceratto.json * 11:10 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:10 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:08 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:08 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:02 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P93227 and previous config saved to /var/cache/conftool/dbconfig/20260527-110242-fceratto.json * 11:02 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:02 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 11:01 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 11:01 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2165.codfw.wmnet with reason: host reimage * 11:00 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2189', diff saved to https://phabricator.wikimedia.org/P93226 and previous config saved to /var/cache/conftool/dbconfig/20260527-110016-marostegui.json * 10:58 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1193.eqiad.wmnet with reason: host reimage * 10:57 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2165.codfw.wmnet with reason: host reimage * 10:56 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 10:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93225 and previous config saved to /var/cache/conftool/dbconfig/20260527-105235-fceratto.json * 10:52 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1193.eqiad.wmnet with reason: host reimage * 10:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1050: repool after maintenance * 10:45 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2176 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93223 and previous config saved to /var/cache/conftool/dbconfig/20260527-104518-fceratto.json * 10:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2176.codfw.wmnet with reason: Maintenance * 10:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93222 and previous config saved to /var/cache/conftool/dbconfig/20260527-104449-fceratto.json * 10:39 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2165.codfw.wmnet with OS trixie * 10:38 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1193.eqiad.wmnet with OS trixie * 10:36 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1193: Upgrading db1193.eqiad.wmnet * 10:35 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1193: Upgrading db1193.eqiad.wmnet * 10:35 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2165: Upgrading db2165.codfw.wmnet * 10:35 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2165: Upgrading db2165.codfw.wmnet * 10:34 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P93218 and previous config saved to /var/cache/conftool/dbconfig/20260527-103441-fceratto.json * 10:29 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:29 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P93217 and previous config saved to /var/cache/conftool/dbconfig/20260527-102434-fceratto.json * 10:22 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:21 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93215 and previous config saved to /var/cache/conftool/dbconfig/20260527-101426-fceratto.json * 10:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1203: Migration of db1203.eqiad.wmnet completed * 10:10 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2166: Migration of db2166.codfw.wmnet completed * 10:08 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93212 and previous config saved to /var/cache/conftool/dbconfig/20260527-100701-fceratto.json * 10:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance * 10:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93211 and previous config saved to /var/cache/conftool/dbconfig/20260527-100632-fceratto.json * 10:05 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1050: repool after maintenance * 10:04 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:02 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1050.eqiad.wmnet with OS trixie * 09:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P93208 and previous config saved to /var/cache/conftool/dbconfig/20260527-095624-fceratto.json * 09:47 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 09:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P93206 and previous config saved to /var/cache/conftool/dbconfig/20260527-094616-fceratto.json * 09:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1050.eqiad.wmnet with reason: host reimage * 09:43 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 09:41 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1050.eqiad.wmnet with reason: host reimage * 09:38 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 09:38 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 09:37 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 09:37 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 09:36 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 09:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93203 and previous config saved to /var/cache/conftool/dbconfig/20260527-093609-fceratto.json * 09:34 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 09:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93202 and previous config saved to /var/cache/conftool/dbconfig/20260527-092842-fceratto.json * 09:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance * 09:28 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1203: Migration of db1203.eqiad.wmnet completed * 09:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93200 and previous config saved to /var/cache/conftool/dbconfig/20260527-092814-fceratto.json * 09:27 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1050.eqiad.wmnet with OS trixie * 09:26 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1050: Upgrading es1050.eqiad.wmnet * 09:25 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1050: Upgrading es1050.eqiad.wmnet * 09:25 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1050: repool after maintenance * 09:25 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1050: repool after maintenance * 09:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2166: Migration of db2166.codfw.wmnet completed * 09:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2051: repool after maintenance * 09:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1203.eqiad.wmnet with OS trixie * 09:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P93196 and previous config saved to /var/cache/conftool/dbconfig/20260527-091806-fceratto.json * 09:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2166.codfw.wmnet with OS trixie * 09:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P93194 and previous config saved to /var/cache/conftool/dbconfig/20260527-090759-fceratto.json * 09:03 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp3074.* * 09:03 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp3066.* * 09:03 fabfur: repooling cp3074 and cp3066 ([[phab:T419825|T419825]]) * 09:02 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp6015.drmrs.wmnet * 09:02 slyngshede@cumin1003: START - Cookbook sre.hosts.remove-downtime for cp6015.drmrs.wmnet * 09:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1203.eqiad.wmnet with reason: host reimage * 09:02 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: name=cp6015.* * 08:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2166.codfw.wmnet with reason: host reimage * 08:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93193 and previous config saved to /var/cache/conftool/dbconfig/20260527-085751-fceratto.json * 08:55 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1203.eqiad.wmnet with reason: host reimage * 08:54 Emperor: restart swift on ms-fe2011 [[phab:T360913|T360913]] * 08:54 jayme@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:54 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2166.codfw.wmnet with reason: host reimage * 08:54 jayme@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 08:51 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 08:51 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 08:51 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp3066.* * 08:51 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp3074.* * 08:51 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 08:50 fabfur: depooling and installing haproxy-awslc on cp3074 and cp3066 ([[phab:T419825|T419825]]) * 08:50 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93191 and previous config saved to /var/cache/conftool/dbconfig/20260527-085024-fceratto.json * 08:50 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance * 08:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93190 and previous config saved to /var/cache/conftool/dbconfig/20260527-085005-fceratto.json * 08:41 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1203.eqiad.wmnet with OS trixie * 08:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P93189 and previous config saved to /var/cache/conftool/dbconfig/20260527-083957-fceratto.json * 08:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2051: repool after maintenance * 08:37 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 08:36 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1203: Upgrading db1203.eqiad.wmnet * 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1004.wikimedia.org * 08:36 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1203: Upgrading db1203.eqiad.wmnet * 08:36 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:35 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2166.codfw.wmnet with OS trixie * 08:35 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2051.codfw.wmnet with OS trixie * 08:34 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2166: Upgrading db2166.codfw.wmnet * 08:33 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2166: Upgrading db2166.codfw.wmnet * 08:33 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader1004.wikimedia.org * 08:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org * 08:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P93185 and previous config saved to /var/cache/conftool/dbconfig/20260527-082950-fceratto.json * 08:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org * 08:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93184 and previous config saved to /var/cache/conftool/dbconfig/20260527-081942-fceratto.json * 08:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2051.codfw.wmnet with reason: host reimage * 08:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2051.codfw.wmnet with reason: host reimage * 08:11 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 08:11 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2153 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93183 and previous config saved to /var/cache/conftool/dbconfig/20260527-081112-fceratto.json * 08:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance * 08:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93182 and previous config saved to /var/cache/conftool/dbconfig/20260527-081054-fceratto.json * 08:07 jmm@dns1004: END - running authdns-update * 08:05 jmm@dns1004: START - running authdns-update * 08:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248', diff saved to https://phabricator.wikimedia.org/P93181 and previous config saved to /var/cache/conftool/dbconfig/20260527-080046-fceratto.json * 07:59 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2051.codfw.wmnet with OS trixie * 07:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248', diff saved to https://phabricator.wikimedia.org/P93180 and previous config saved to /var/cache/conftool/dbconfig/20260527-075039-fceratto.json * 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1026.eqiad.wmnet * 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1026.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:43 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1026.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2051: Upgrading es2051.codfw.wmnet * 07:42 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2051: Upgrading es2051.codfw.wmnet * 07:41 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93178 and previous config saved to /var/cache/conftool/dbconfig/20260527-074031-fceratto.json * 07:40 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294125{{!}}Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126{{!}}Add script to demote ineligible members of restricted global groups (T425395)]] (duration: 06m 42s) * 07:36 mszwarc@deploy1003: mszwarc: Continuing with deployment * 07:35 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1294125{{!}}Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126{{!}}Add script to demote ineligible members of restricted global groups (T425395)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93177 and previous config saved to /var/cache/conftool/dbconfig/20260527-073504-fceratto.json * 07:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2248.codfw.wmnet with reason: Maintenance * 07:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93176 and previous config saved to /var/cache/conftool/dbconfig/20260527-073434-fceratto.json * 07:33 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1294125{{!}}Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126{{!}}Add script to demote ineligible members of restricted global groups (T425395)]] * 07:28 jmm@cumin2002: START - Cookbook sre.dns.netbox * 07:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P93175 and previous config saved to /var/cache/conftool/dbconfig/20260527-072426-fceratto.json * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) * 07:23 marostegui@cumin1003: Removing pc1014 from zarcillo [[phab:T427190|T427190]] * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1014.eqiad.wmnet * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 07:23 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 07:18 marostegui@cumin1003: START - Cookbook sre.dns.netbox * 07:15 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1026.eqiad.wmnet * 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1025.eqiad.wmnet * 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P93174 and previous config saved to /var/cache/conftool/dbconfig/20260527-071418-fceratto.json * 07:13 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1014.eqiad.wmnet * 07:13 marostegui@cumin1003: START - Cookbook sre.mysql.decommission * 07:13 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org * 07:07 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2055: repool after maintenance * 07:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org * 07:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1003.wikimedia.org * 07:06 jmm@cumin2002: START - Cookbook sre.dns.netbox * 07:06 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1190.eqiad.wmnet with reason: Maintenance on db1190 * 07:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93172 and previous config saved to /var/cache/conftool/dbconfig/20260527-070410-fceratto.json * 07:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader1003.wikimedia.org * 06:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93171 and previous config saved to /var/cache/conftool/dbconfig/20260527-065545-fceratto.json * 06:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2247.codfw.wmnet with reason: Maintenance * 06:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93170 and previous config saved to /var/cache/conftool/dbconfig/20260527-065526-fceratto.json * 06:54 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1025.eqiad.wmnet * 06:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P93168 and previous config saved to /var/cache/conftool/dbconfig/20260527-064519-fceratto.json * 06:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P93166 and previous config saved to /var/cache/conftool/dbconfig/20260527-063511-fceratto.json * 06:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93165 and previous config saved to /var/cache/conftool/dbconfig/20260527-062503-fceratto.json * 06:22 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2055: repool after maintenance * 06:21 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 06:21 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2055.codfw.wmnet with OS trixie * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2246 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93163 and previous config saved to /var/cache/conftool/dbconfig/20260527-061643-fceratto.json * 06:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2246.codfw.wmnet with reason: Maintenance * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93162 and previous config saved to /var/cache/conftool/dbconfig/20260527-061613-fceratto.json * 06:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P93161 and previous config saved to /var/cache/conftool/dbconfig/20260527-060606-fceratto.json * 06:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2055.codfw.wmnet with reason: host reimage * 05:56 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2055.codfw.wmnet with reason: host reimage * 05:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P93160 and previous config saved to /var/cache/conftool/dbconfig/20260527-055558-fceratto.json * 05:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93159 and previous config saved to /var/cache/conftool/dbconfig/20260527-054550-fceratto.json * 05:41 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2055.codfw.wmnet with OS trixie * 05:40 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2055: Upgrading es2055.codfw.wmnet * 05:40 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2055: Upgrading es2055.codfw.wmnet * 05:40 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:38 moritzm: remove ganeti1026 from eqiad Ganeti cluster [[phab:T424680|T424680]] * 05:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2245 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93157 and previous config saved to /var/cache/conftool/dbconfig/20260527-053727-fceratto.json * 05:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: Maintenance * 05:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93156 and previous config saved to /var/cache/conftool/dbconfig/20260527-053708-fceratto.json * 05:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P93155 and previous config saved to /var/cache/conftool/dbconfig/20260527-052700-fceratto.json * 05:26 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1014 from dbctl [[phab:T427270|T427270]]', diff saved to https://phabricator.wikimedia.org/P93154 and previous config saved to /var/cache/conftool/dbconfig/20260527-052624-marostegui.json * 05:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P93153 and previous config saved to /var/cache/conftool/dbconfig/20260527-051653-fceratto.json * 05:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93152 and previous config saved to /var/cache/conftool/dbconfig/20260527-050645-fceratto.json * 04:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2237 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93151 and previous config saved to /var/cache/conftool/dbconfig/20260527-045827-fceratto.json * 04:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: Maintenance * 04:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93150 and previous config saved to /var/cache/conftool/dbconfig/20260527-045759-fceratto.json * 04:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P93149 and previous config saved to /var/cache/conftool/dbconfig/20260527-044751-fceratto.json * 04:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P93148 and previous config saved to /var/cache/conftool/dbconfig/20260527-043744-fceratto.json * 04:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93147 and previous config saved to /var/cache/conftool/dbconfig/20260527-042737-fceratto.json * 04:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2236 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93146 and previous config saved to /var/cache/conftool/dbconfig/20260527-041921-fceratto.json * 04:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2236.codfw.wmnet with reason: Maintenance * 04:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93145 and previous config saved to /var/cache/conftool/dbconfig/20260527-041852-fceratto.json * 04:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P93144 and previous config saved to /var/cache/conftool/dbconfig/20260527-040844-fceratto.json * 03:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P93143 and previous config saved to /var/cache/conftool/dbconfig/20260527-035836-fceratto.json * 03:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93142 and previous config saved to /var/cache/conftool/dbconfig/20260527-034828-fceratto.json * 03:40 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93141 and previous config saved to /var/cache/conftool/dbconfig/20260527-034008-fceratto.json * 03:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2219.codfw.wmnet with reason: Maintenance * 03:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93140 and previous config saved to /var/cache/conftool/dbconfig/20260527-033938-fceratto.json * 03:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P93139 and previous config saved to /var/cache/conftool/dbconfig/20260527-032931-fceratto.json * 03:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P93138 and previous config saved to /var/cache/conftool/dbconfig/20260527-031923-fceratto.json * 03:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93137 and previous config saved to /var/cache/conftool/dbconfig/20260527-030915-fceratto.json * 03:00 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93136 and previous config saved to /var/cache/conftool/dbconfig/20260527-030045-fceratto.json * 03:00 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2210.codfw.wmnet with reason: Maintenance * 03:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93135 and previous config saved to /var/cache/conftool/dbconfig/20260527-030016-fceratto.json * 02:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P93134 and previous config saved to /var/cache/conftool/dbconfig/20260527-025008-fceratto.json * 02:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P93133 and previous config saved to /var/cache/conftool/dbconfig/20260527-024000-fceratto.json * 02:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93132 and previous config saved to /var/cache/conftool/dbconfig/20260527-022953-fceratto.json * 02:21 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93131 and previous config saved to /var/cache/conftool/dbconfig/20260527-022133-fceratto.json * 02:21 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2206.codfw.wmnet with reason: Maintenance * 02:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93130 and previous config saved to /var/cache/conftool/dbconfig/20260527-022100-fceratto.json * 02:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P93129 and previous config saved to /var/cache/conftool/dbconfig/20260527-021053-fceratto.json * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 29s) * 02:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P93128 and previous config saved to /var/cache/conftool/dbconfig/20260527-020045-fceratto.json * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93127 and previous config saved to /var/cache/conftool/dbconfig/20260527-015037-fceratto.json * 01:42 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93126 and previous config saved to /var/cache/conftool/dbconfig/20260527-014204-fceratto.json * 01:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2179.codfw.wmnet with reason: Maintenance * 01:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93125 and previous config saved to /var/cache/conftool/dbconfig/20260527-014134-fceratto.json * 01:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P93124 and previous config saved to /var/cache/conftool/dbconfig/20260527-013126-fceratto.json * 01:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P93123 and previous config saved to /var/cache/conftool/dbconfig/20260527-012119-fceratto.json * 01:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93122 and previous config saved to /var/cache/conftool/dbconfig/20260527-011111-fceratto.json * 01:02 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2172 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93121 and previous config saved to /var/cache/conftool/dbconfig/20260527-010234-fceratto.json * 01:02 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance * 01:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93120 and previous config saved to /var/cache/conftool/dbconfig/20260527-010205-fceratto.json * 00:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P93119 and previous config saved to /var/cache/conftool/dbconfig/20260527-005157-fceratto.json * 00:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P93118 and previous config saved to /var/cache/conftool/dbconfig/20260527-004149-fceratto.json * 00:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93117 and previous config saved to /var/cache/conftool/dbconfig/20260527-003141-fceratto.json * 00:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2155 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93116 and previous config saved to /var/cache/conftool/dbconfig/20260527-002309-fceratto.json * 00:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance * 00:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93115 and previous config saved to /var/cache/conftool/dbconfig/20260527-002228-fceratto.json * 00:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P93114 and previous config saved to /var/cache/conftool/dbconfig/20260527-001220-fceratto.json * 00:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P93113 and previous config saved to /var/cache/conftool/dbconfig/20260527-000209-fceratto.json == 2026-05-26 == * 23:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93112 and previous config saved to /var/cache/conftool/dbconfig/20260526-235201-fceratto.json * 23:44 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93111 and previous config saved to /var/cache/conftool/dbconfig/20260526-234451-fceratto.json * 23:44 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance * 23:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93110 and previous config saved to /var/cache/conftool/dbconfig/20260526-234421-fceratto.json * 23:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P93109 and previous config saved to /var/cache/conftool/dbconfig/20260526-233414-fceratto.json * 23:27 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5026.* * 23:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P93108 and previous config saved to /var/cache/conftool/dbconfig/20260526-232406-fceratto.json * 23:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93107 and previous config saved to /var/cache/conftool/dbconfig/20260526-231358-fceratto.json * 23:07 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5026.* * 23:06 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93106 and previous config saved to /var/cache/conftool/dbconfig/20260526-230650-fceratto.json * 23:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance * 23:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93105 and previous config saved to /var/cache/conftool/dbconfig/20260526-230620-fceratto.json * 22:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P93104 and previous config saved to /var/cache/conftool/dbconfig/20260526-225612-fceratto.json * 22:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P93103 and previous config saved to /var/cache/conftool/dbconfig/20260526-224604-fceratto.json * 22:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93101 and previous config saved to /var/cache/conftool/dbconfig/20260526-223556-fceratto.json * 22:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93100 and previous config saved to /var/cache/conftool/dbconfig/20260526-222848-fceratto.json * 22:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance * 22:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93099 and previous config saved to /var/cache/conftool/dbconfig/20260526-222828-fceratto.json * 22:23 robh@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts cp6015.drmrs.wmnet * 22:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P93098 and previous config saved to /var/cache/conftool/dbconfig/20260526-221819-fceratto.json * 22:10 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1009.eqiad.wmnet with OS trixie * 22:08 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1008.eqiad.wmnet with OS trixie * 22:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P93097 and previous config saved to /var/cache/conftool/dbconfig/20260526-220811-fceratto.json * 22:04 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293701{{!}}MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] (duration: 09m 30s) * 22:03 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1009.eqiad.wmnet with reason: host reimage * 22:00 egardner@deploy1003: egardner, mfossati: Continuing with deployment * 21:59 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1008.eqiad.wmnet with reason: host reimage * 21:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93096 and previous config saved to /var/cache/conftool/dbconfig/20260526-215803-fceratto.json * 21:57 egardner@deploy1003: egardner, mfossati: Backport for [[gerrit:1293701{{!}}MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:56 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp6015.drmrs.wmnet * 21:56 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1010.eqiad.wmnet with OS trixie * 21:56 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp6015.drmrs.wmnet * 21:55 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1293701{{!}}MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] * 21:54 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1009.eqiad.wmnet with reason: host reimage * 21:51 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1008.eqiad.wmnet with reason: host reimage * 21:50 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93095 and previous config saved to /var/cache/conftool/dbconfig/20260526-215043-fceratto.json * 21:50 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance * 21:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93094 and previous config saved to /var/cache/conftool/dbconfig/20260526-215011-fceratto.json * 21:49 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1010.eqiad.wmnet with reason: host reimage * 21:47 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp6015.drmrs.wmnet * 21:44 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1009 * 21:44 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host relforge1009 * 21:43 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host relforge1009 * 21:43 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) relforge1009.eqiad.wmnet 120.48.64.10.in-addr.arpa 0.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:43 bking@cumin2002: START - Cookbook sre.dns.wipe-cache relforge1009.eqiad.wmnet 120.48.64.10.in-addr.arpa 0.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:43 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:42 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1009 - bking@cumin2002" * 21:42 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1010.eqiad.wmnet with reason: host reimage * 21:42 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1009 - bking@cumin2002" * 21:41 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1008 * 21:40 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host relforge1008 * 21:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P93093 and previous config saved to /var/cache/conftool/dbconfig/20260526-214003-fceratto.json * 21:36 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host relforge1008 * 21:36 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) relforge1008.eqiad.wmnet 100.32.64.10.in-addr.arpa 0.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:36 bking@cumin2002: START - Cookbook sre.dns.wipe-cache relforge1008.eqiad.wmnet 100.32.64.10.in-addr.arpa 0.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:36 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:36 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1008 - bking@cumin2002" * 21:36 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1008 - bking@cumin2002" * 21:35 bking@cumin2002: START - Cookbook sre.dns.netbox * 21:32 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1010 * 21:32 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1010 * 21:31 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1010.eqiad.wmnet with OS trixie * 21:31 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1009 * 21:30 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1009.eqiad.wmnet with OS trixie * 21:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P93092 and previous config saved to /var/cache/conftool/dbconfig/20260526-212955-fceratto.json * 21:29 bking@cumin2002: START - Cookbook sre.dns.netbox * 21:29 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1008 * 21:29 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1008.eqiad.wmnet with OS trixie * 21:27 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist "all.dblist - mediamoderation-continuous-scan.dblist - preinstall.dblist" extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` in tmux session - [[phab:T421688|T421688]] * 21:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93091 and previous config saved to /var/cache/conftool/dbconfig/20260526-211948-fceratto.json * 21:19 jhathaway: dmarc ingress test run mx-in1001 * 21:15 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_codfw and A:cp * 21:15 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2057.codfw.wmnet * 21:14 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_codfw and A:cp * 21:14 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2058.codfw.wmnet * 21:12 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2222 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93090 and previous config saved to /var/cache/conftool/dbconfig/20260526-211238-fceratto.json * 21:12 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance * 21:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93089 and previous config saved to /var/cache/conftool/dbconfig/20260526-211207-fceratto.json * 21:06 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 21:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P93088 and previous config saved to /var/cache/conftool/dbconfig/20260526-210159-fceratto.json * 20:55 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on phab2003.codfw.wmnet with reason: WIP * 20:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P93087 and previous config saved to /var/cache/conftool/dbconfig/20260526-205152-fceratto.json * 20:50 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:50 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 20:50 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 20:45 dzahn@cumin2002: START - Cookbook sre.dns.netbox * 20:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93086 and previous config saved to /var/cache/conftool/dbconfig/20260526-204143-fceratto.json * 20:38 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2055.codfw.wmnet * 20:34 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93085 and previous config saved to /var/cache/conftool/dbconfig/20260526-203430-fceratto.json * 20:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance * 20:34 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2056.codfw.wmnet * 20:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93084 and previous config saved to /var/cache/conftool/dbconfig/20260526-203357-fceratto.json * 20:32 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:32 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:32 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:31 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P93083 and previous config saved to /var/cache/conftool/dbconfig/20260526-202349-fceratto.json * 20:18 alexsanford@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293161{{!}}Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794{{!}}Re-enable ReadingLists survey on beta cluster (T426781)]] (duration: 09m 14s) * 20:14 alexsanford@deploy1003: alexsanford, aude: Continuing with deployment * 20:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P93082 and previous config saved to /var/cache/conftool/dbconfig/20260526-201341-fceratto.json * 20:11 alexsanford@deploy1003: alexsanford, aude: Backport for [[gerrit:1293161{{!}}Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794{{!}}Re-enable ReadingLists survey on beta cluster (T426781)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:09 alexsanford@deploy1003: Started scap sync-world: Backport for [[gerrit:1293161{{!}}Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794{{!}}Re-enable ReadingLists survey on beta cluster (T426781)]] * 20:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93081 and previous config saved to /var/cache/conftool/dbconfig/20260526-200333-fceratto.json * 19:59 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2053.codfw.wmnet * 19:58 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2029.codfw.wmnet with OS trixie * 19:57 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2028.codfw.wmnet with OS trixie * 19:56 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2208 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93080 and previous config saved to /var/cache/conftool/dbconfig/20260526-195632-fceratto.json * 19:56 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance * 19:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93079 and previous config saved to /var/cache/conftool/dbconfig/20260526-195557-fceratto.json * 19:55 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2054.codfw.wmnet * 19:51 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:51 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P93078 and previous config saved to /var/cache/conftool/dbconfig/20260526-194549-fceratto.json * 19:45 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 19:44 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:43 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2029 * 19:43 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 * 19:43 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 * 19:43 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 19:40 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb2014.codfw.wmnet with OS trixie * 19:40 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:40 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb2013.codfw.wmnet with OS trixie * 19:40 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:39 brett@cumin2002: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 19:38 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 19:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P93077 and previous config saved to /var/cache/conftool/dbconfig/20260526-193541-fceratto.json * 19:35 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:35 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 19:30 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 19:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93076 and previous config saved to /var/cache/conftool/dbconfig/20260526-192533-fceratto.json * 19:24 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:21 dzahn@cumin2002: START - Cookbook sre.dns.netbox * 19:20 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2051.codfw.wmnet * 19:19 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:19 brett@cumin2002: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 19:18 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93075 and previous config saved to /var/cache/conftool/dbconfig/20260526-191818-fceratto.json * 19:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance * 19:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93074 and previous config saved to /var/cache/conftool/dbconfig/20260526-191748-fceratto.json * 19:16 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2052.codfw.wmnet * 19:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P93073 and previous config saved to /var/cache/conftool/dbconfig/20260526-190740-fceratto.json * 19:07 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb2014.codfw.wmnet with reason: host reimage * 19:03 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb2013.codfw.wmnet with reason: host reimage * 18:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1026.eqiad.wmnet * 18:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P93072 and previous config saved to /var/cache/conftool/dbconfig/20260526-185732-fceratto.json * 18:56 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2014.codfw.wmnet with reason: host reimage * 18:56 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2013.codfw.wmnet with reason: host reimage * 18:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93071 and previous config saved to /var/cache/conftool/dbconfig/20260526-184724-fceratto.json * 18:44 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb2014.codfw.wmnet with OS trixie * 18:43 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb2013.codfw.wmnet with OS trixie * 18:41 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host rdb2014.codfw.wmnet with OS trixie * 18:41 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2049.codfw.wmnet * 18:40 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93070 and previous config saved to /var/cache/conftool/dbconfig/20260526-184009-fceratto.json * 18:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance * 18:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93069 and previous config saved to /var/cache/conftool/dbconfig/20260526-183939-fceratto.json * 18:37 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2050.codfw.wmnet * 18:30 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 18:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P93068 and previous config saved to /var/cache/conftool/dbconfig/20260526-182931-fceratto.json * 18:29 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:29 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_magru-v4 - dzahn@cumin2002" * 18:29 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_magru-v4 - dzahn@cumin2002" * 18:24 dzahn@cumin2002: START - Cookbook sre.dns.netbox * 18:21 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 18:21 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 18:21 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 18:20 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 18:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P93066 and previous config saved to /var/cache/conftool/dbconfig/20260526-181923-fceratto.json * 18:15 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 18:15 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 18:15 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 18:15 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 18:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93065 and previous config saved to /var/cache/conftool/dbconfig/20260526-180915-fceratto.json * 18:02 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93064 and previous config saved to /var/cache/conftool/dbconfig/20260526-180205-fceratto.json * 18:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2159.codfw.wmnet with reason: Maintenance * 18:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93063 and previous config saved to /var/cache/conftool/dbconfig/20260526-180132-fceratto.json * 18:00 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2047.codfw.wmnet * 17:59 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2048.codfw.wmnet * 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:54 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:54 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:54 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P93062 and previous config saved to /var/cache/conftool/dbconfig/20260526-175124-fceratto.json * 17:42 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293779{{!}}Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] (duration: 07m 25s) * 17:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P93060 and previous config saved to /var/cache/conftool/dbconfig/20260526-174117-fceratto.json * 17:39 mvernon@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ms-be2089.codfw.wmnet * 17:37 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 17:37 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:36 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:36 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:36 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1293779{{!}}Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:36 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:34 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1293779{{!}}Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] * 17:33 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:33 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:33 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:33 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93059 and previous config saved to /var/cache/conftool/dbconfig/20260526-173109-fceratto.json * 17:27 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:26 jclark@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:25 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:25 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:25 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:24 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:24 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1001 to eqiad - jclark@cumin1003" * 17:24 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:24 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1001 to eqiad - jclark@cumin1003" * 17:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93058 and previous config saved to /var/cache/conftool/dbconfig/20260526-172332-fceratto.json * 17:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2227.codfw.wmnet with reason: Maintenance * 17:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93057 and previous config saved to /var/cache/conftool/dbconfig/20260526-172303-fceratto.json * 17:21 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2045.codfw.wmnet * 17:20 jclark@cumin1003: START - Cookbook sre.dns.netbox * 17:20 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2046.codfw.wmnet * 17:18 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:17 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:17 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:17 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:17 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:17 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:17 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:17 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:16 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:15 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 17:14 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:14 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:14 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:14 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:13 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:13 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:13 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:13 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:13 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P93056 and previous config saved to /var/cache/conftool/dbconfig/20260526-171255-fceratto.json * 17:11 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:11 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:11 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:11 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:07 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:05 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:05 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:05 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:05 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P93055 and previous config saved to /var/cache/conftool/dbconfig/20260526-170247-fceratto.json * 17:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:57 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:55 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:52 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93054 and previous config saved to /var/cache/conftool/dbconfig/20260526-165240-fceratto.json * 16:50 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:50 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:50 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:50 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:45 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:45 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:45 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:45 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:45 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:44 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:44 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2209 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93053 and previous config saved to /var/cache/conftool/dbconfig/20260526-164421-fceratto.json * 16:44 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:44 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1002 to eqiad - jclark@cumin1003" * 16:44 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2209.codfw.wmnet with reason: Maintenance * 16:44 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1002 to eqiad - jclark@cumin1003" * 16:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93052 and previous config saved to /var/cache/conftool/dbconfig/20260526-164352-fceratto.json * 16:42 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2043.codfw.wmnet * 16:41 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2044.codfw.wmnet * 16:40 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:40 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:40 jclark@cumin1003: START - Cookbook sre.dns.netbox * 16:40 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:40 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:40 brett: reboot lvs 101[345].eqiad.wmnet * 16:39 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:39 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:39 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:39 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:37 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:37 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 16:37 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:37 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:37 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:37 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 16:37 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 16:36 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 16:36 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:36 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:36 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:36 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:35 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 16:34 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 16:34 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:34 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:34 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:34 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:33 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_codfw and A:cp * 16:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P93051 and previous config saved to /var/cache/conftool/dbconfig/20260526-163344-fceratto.json * 16:33 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_codfw and A:cp * 16:31 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:31 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:30 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:30 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P93050 and previous config saved to /var/cache/conftool/dbconfig/20260526-162336-fceratto.json * 16:13 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2089.codfw.wmnet * 16:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93049 and previous config saved to /var/cache/conftool/dbconfig/20260526-161328-fceratto.json * 16:11 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:11 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:10 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:10 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:07 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search,name=eqiad * 16:06 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:06 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:06 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:06 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93047 and previous config saved to /var/cache/conftool/dbconfig/20260526-160450-fceratto.json * 16:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2194.codfw.wmnet with reason: Maintenance * 16:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93046 and previous config saved to /var/cache/conftool/dbconfig/20260526-160420-fceratto.json * 16:03 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:03 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:03 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] (duration: 00m 28s) * 16:02 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] * 16:00 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:00 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:00 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:00 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:55 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:55 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:55 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:55 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] (duration: 00m 22s) * 15:55 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:55 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] * 15:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P93045 and previous config saved to /var/cache/conftool/dbconfig/20260526-155413-fceratto.json * 15:46 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search,name=eqiad * 15:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P93044 and previous config saved to /var/cache/conftool/dbconfig/20260526-154405-fceratto.json * 15:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93043 and previous config saved to /var/cache/conftool/dbconfig/20260526-153357-fceratto.json * 15:30 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:30 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:30 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:30 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93042 and previous config saved to /var/cache/conftool/dbconfig/20260526-152629-fceratto.json * 15:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2190.codfw.wmnet with reason: Maintenance * 15:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93041 and previous config saved to /var/cache/conftool/dbconfig/20260526-152559-fceratto.json * 15:24 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:24 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:24 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:24 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:23 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:22 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:22 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:22 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P93040 and previous config saved to /var/cache/conftool/dbconfig/20260526-151552-fceratto.json * 15:12 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2196: Rack maintenance completed * 15:10 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2196.codfw.wmnet * 15:10 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2196.codfw.wmnet * 15:07 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw * 15:06 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2222: Rack maintenance completed * 15:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P93037 and previous config saved to /var/cache/conftool/dbconfig/20260526-150546-fceratto.json * 15:04 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2221: Rack maintenance completed * 15:04 brennen@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab1004 for [[phab:T427286|T427286]] (duration: 00m 39s) * 15:03 brennen@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab1004 for [[phab:T427286|T427286]] * 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2002 for [[phab:T427286|T427286]] (duration: 00m 45s) * 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2002 for [[phab:T427286|T427286]] * 15:02 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2002.codfw.wmnet with reason: Phabricator deploy * 15:01 bjensen: uploading prometheus-memcached-exporter_0.16.0-1_amd64 on apt1002 * 15:01 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator deploy * 15:00 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2223: switch maintenance * 14:56 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2196: Rack maintenance completed * 14:55 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2221.codfw.wmnet * 14:55 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2221.codfw.wmnet * 14:55 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2222.codfw.wmnet * 14:55 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2222.codfw.wmnet * 14:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93033 and previous config saved to /var/cache/conftool/dbconfig/20260526-145538-fceratto.json * 14:55 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet * 14:54 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1026.eqiad.wmnet * 14:52 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet * 14:52 moritzm: remove ganeti1025 from eqiad Ganeti cluster [[phab:T424680|T424680]] * 14:51 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2030.codfw.wmnet to cluster codfw and group A * 14:51 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2222: Rack maintenance completed * 14:49 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:49 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2221: Rack maintenance completed * 14:49 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2030.codfw.wmnet to cluster codfw and group A * 14:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2029.codfw.wmnet to cluster codfw and group A * 14:47 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2029.codfw.wmnet to cluster codfw and group A * 14:47 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2177 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93030 and previous config saved to /var/cache/conftool/dbconfig/20260526-144718-fceratto.json * 14:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance * 14:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93029 and previous config saved to /var/cache/conftool/dbconfig/20260526-144651-fceratto.json * 14:45 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=wdqs-scholarly,name=codfw * 14:45 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=wdqs-scholarly,name=codfw * 14:43 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw * 14:40 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:40 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2167: Migration of db2167.codfw.wmnet completed * 14:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P93026 and previous config saved to /var/cache/conftool/dbconfig/20260526-143643-fceratto.json * 14:31 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1054.eqiad.wmnet with OS trixie * 14:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P93023 and previous config saved to /var/cache/conftool/dbconfig/20260526-142636-fceratto.json * 14:26 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:25 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:24 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1014: Rack maintenance completed * 14:24 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.parsercache (exit_code=99) * 14:24 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache * 14:24 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc1014: Rack maintenance completed * 14:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1025.eqiad.wmnet * 14:19 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for backup2015.codfw.wmnet,db2197.codfw.wmnet * 14:19 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for backup2015.codfw.wmnet,db2197.codfw.wmnet * 14:18 jynus: restarting mediabackups@codfw after maintenance on a codfw backup media storage server [[phab:T426199|T426199]] * 14:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93021 and previous config saved to /var/cache/conftool/dbconfig/20260526-141628-fceratto.json * 14:16 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:14 fabfur: repooled cp2043 ([[phab:T426199|T426199]]) * 14:14 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2223: switch maintenance * 14:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1054.eqiad.wmnet with reason: host reimage * 14:14 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2043.* * 14:13 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293710{{!}}Site info should output thumblimits as array (T427066)]] (duration: 06m 40s) * 14:12 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:10 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1054.eqiad.wmnet with reason: host reimage * 14:10 fabfur@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2011.codfw.wmnet * 14:10 fabfur@cumin1003: START - Cookbook sre.hosts.remove-downtime for lvs2011.codfw.wmnet * 14:09 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 14:09 fabfur: restoring lvs2011 as primary ([[phab:T426199|T426199]]) * 14:08 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1293710{{!}}Site info should output thumblimits as array (T427066)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:08 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 14:08 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 14:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93017 and previous config saved to /var/cache/conftool/dbconfig/20260526-140748-fceratto.json * 14:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance * 14:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93016 and previous config saved to /var/cache/conftool/dbconfig/20260526-140718-fceratto.json * 14:07 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1293710{{!}}Site info should output thumblimits as array (T427066)]] * 14:05 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.decommission (exit_code=99) * 14:05 marostegui@cumin1003: Removing pc1013 from zarcillo [[phab:T427190|T427190]] * 14:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1013.eqiad.wmnet * 14:04 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:04 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 14:04 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 14:00 marostegui@cumin1003: START - Cookbook sre.dns.netbox * 13:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P93014 and previous config saved to /var/cache/conftool/dbconfig/20260526-135711-fceratto.json * 13:56 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1054.eqiad.wmnet with OS trixie * 13:55 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2167: Migration of db2167.codfw.wmnet completed * 13:53 Amir1: drop flaggedrevs tables on cawikinews ([[phab:T423577|T423577]]) * 13:49 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1013.eqiad.wmnet * 13:49 marostegui@cumin1003: START - Cookbook sre.mysql.decommission * 13:48 Lucas_WMDE: UTC afternoon backport+config window done * 13:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P93012 and previous config saved to /var/cache/conftool/dbconfig/20260526-134703-fceratto.json * 13:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2167.codfw.wmnet with OS trixie * 13:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93011 and previous config saved to /var/cache/conftool/dbconfig/20260526-133656-fceratto.json * 13:36 XioNoX: reboot lsw1-a2-codfw for software upgrade - [[phab:T426199|T426199]] * 13:36 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2223: switch maintenance * 13:35 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2223: switch maintenance * 13:35 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2222: switch maintenance * 13:35 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2222: switch maintenance * 13:35 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2221: switch maintenance * 13:35 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293662{{!}}Enable IRS Direct Reporting on testwiki (T425025)]] (duration: 09m 28s) * 13:34 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2221: switch maintenance * 13:34 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2196: switch maintenance * 13:34 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2196: switch maintenance * 13:31 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 13:30 stran@deploy1003: stran: Continuing with deployment * 13:29 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 13:29 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93006 and previous config saved to /var/cache/conftool/dbconfig/20260526-132927-fceratto.json * 13:29 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2167.codfw.wmnet with reason: host reimage * 13:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance * 13:29 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 34 hosts with reason: Switch maintenance * 13:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93005 and previous config saved to /var/cache/conftool/dbconfig/20260526-132857-fceratto.json * 13:28 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lsw1-a2-codfw,lsw1-a2-codfw IPv6,lsw1-a2-codfw.mgmt with reason: Switch maintenance * 13:27 stran@deploy1003: stran: Backport for [[gerrit:1293662{{!}}Enable IRS Direct Reporting on testwiki (T425025)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:25 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1293662{{!}}Enable IRS Direct Reporting on testwiki (T425025)]] * 13:25 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2167.codfw.wmnet with reason: host reimage * 13:22 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293706{{!}}Disable the `no` language code for translation (T424613)]] (duration: 08m 30s) * 13:22 ladsgroup@dns1004: END - running authdns-update * 13:20 ladsgroup@dns1004: START - running authdns-update * 13:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P93004 and previous config saved to /var/cache/conftool/dbconfig/20260526-131850-fceratto.json * 13:18 lucaswerkmeister-wmde@deploy1003: jhsoby, lucaswerkmeister-wmde: Continuing with deployment * 13:16 lucaswerkmeister-wmde@deploy1003: jhsoby, lucaswerkmeister-wmde: Backport for [[gerrit:1293706{{!}}Disable the `no` language code for translation (T424613)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:14 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1293706{{!}}Disable the `no` language code for translation (T424613)]] * 13:12 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293177{{!}}Instrumentation: log new articles namespace and source (T422146)]] (duration: 07m 09s) * 13:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P93003 and previous config saved to /var/cache/conftool/dbconfig/20260526-130842-fceratto.json * 13:08 sbisson@deploy1003: sbisson: Continuing with deployment * 13:07 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2167.codfw.wmnet with OS trixie * 13:07 sbisson@deploy1003: sbisson: Backport for [[gerrit:1293177{{!}}Instrumentation: log new articles namespace and source (T422146)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:05 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2167: Upgrading db2167.codfw.wmnet * 13:05 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1293177{{!}}Instrumentation: log new articles namespace and source (T422146)]] * 13:04 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2167: Upgrading db2167.codfw.wmnet * 13:04 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:04 kart_: Update Recommendation API to 2026-05-26-074931-production * 13:03 kartik@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 13:00 topranks: deactivate CR BGP to doh2002 to test backup path via doh2001 * 12:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93000 and previous config saved to /var/cache/conftool/dbconfig/20260526-125834-fceratto.json * 12:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2226 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92999 and previous config saved to /var/cache/conftool/dbconfig/20260526-125135-fceratto.json * 12:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance * 12:51 kartik@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92998 and previous config saved to /var/cache/conftool/dbconfig/20260526-125105-fceratto.json * 12:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P92997 and previous config saved to /var/cache/conftool/dbconfig/20260526-124059-fceratto.json * 12:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2003.wikimedia.org * 12:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1214: Migration of db1214.eqiad.wmnet completed * 12:33 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc2003.wikimedia.org * 12:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P92995 and previous config saved to /var/cache/conftool/dbconfig/20260526-123052-fceratto.json * 12:26 fabfur: depooled cp204 for network activity ([[phab:T426199|T426199]]) * 12:26 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2043.* * 12:24 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-a1-codfw,ssw1-a1-codfw IPv6,ssw1-a1-codfw.mgmt with reason: Switch maintenance * 12:24 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mirror1001.wikimedia.org * 12:23 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:23 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:22 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92993 and previous config saved to /var/cache/conftool/dbconfig/20260526-122044-fceratto.json * 12:20 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:19 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mirror1001.wikimedia.org * 12:13 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2225 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92991 and previous config saved to /var/cache/conftool/dbconfig/20260526-121336-fceratto.json * 12:13 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance * 12:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92990 and previous config saved to /var/cache/conftool/dbconfig/20260526-121306-fceratto.json * 12:09 fabfur@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2011.codfw.wmnet with reason: Planned downtime for rack maintenance * 12:08 fabfur: downtime, disable puppet and stop pybal for rack maintenance ([[phab:T426199|T426199]]) * 12:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2181: Migration of db2181.codfw.wmnet completed * 12:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P92987 and previous config saved to /var/cache/conftool/dbconfig/20260526-120258-fceratto.json * 12:01 XioNoX: start ssw1-a1-codfw network maintenance (no impact expected as the spines are redundant) * 11:59 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293167{{!}}hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055{{!}}hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] (duration: 15m 26s) * 11:56 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on backup2015.codfw.wmnet,db2197.codfw.wmnet with reason: network maintenance * 11:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1005.eqiad.wmnet * 11:55 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment * 11:54 jynus: stopping mediabackups@codfw for maintenance on a codfw backup media storage server [[phab:T426199|T426199]] * 11:54 jmm@dns1004: END - running authdns-update * 11:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P92985 and previous config saved to /var/cache/conftool/dbconfig/20260526-115251-fceratto.json * 11:52 jmm@dns1004: START - running authdns-update * 11:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1005.eqiad.wmnet * 11:49 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1214: Migration of db1214.eqiad.wmnet completed * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1004.eqiad.wmnet * 11:47 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet * 11:46 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1293167{{!}}hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055{{!}}hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:45 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1004.eqiad.wmnet * 11:44 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1293167{{!}}hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055{{!}}hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] * 11:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92983 and previous config saved to /var/cache/conftool/dbconfig/20260526-114243-fceratto.json * 11:42 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet * 11:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1214.eqiad.wmnet with OS trixie * 11:35 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293691{{!}}Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] (duration: 06m 46s) * 11:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92981 and previous config saved to /var/cache/conftool/dbconfig/20260526-113542-fceratto.json * 11:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance * 11:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92980 and previous config saved to /var/cache/conftool/dbconfig/20260526-113521-fceratto.json * 11:31 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 11:31 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1293691{{!}}Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1222: Migration of db1222.eqiad.wmnet completed * 11:29 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1293691{{!}}Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] * 11:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P92978 and previous config saved to /var/cache/conftool/dbconfig/20260526-112513-fceratto.json * 11:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1214.eqiad.wmnet with reason: host reimage * 11:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc4 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92977 and previous config saved to /var/cache/conftool/dbconfig/20260526-112326-marostegui.json * 11:22 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2181: Migration of db2181.codfw.wmnet completed * 11:22 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1024 to dbctl [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92975 and previous config saved to /var/cache/conftool/dbconfig/20260526-112215-marostegui.json * 11:20 fceratto@cumin1003: dbctl commit (dc=all): 'Switchover es2042 es2041 for [[phab:T426199|T426199]]', diff saved to https://phabricator.wikimedia.org/P92974 and previous config saved to /var/cache/conftool/dbconfig/20260526-112028-fceratto.json * 11:17 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1214.eqiad.wmnet with reason: host reimage * 11:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P92972 and previous config saved to /var/cache/conftool/dbconfig/20260526-111506-fceratto.json * 11:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2181.codfw.wmnet with OS trixie * 11:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92971 and previous config saved to /var/cache/conftool/dbconfig/20260526-110458-fceratto.json * 11:02 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1214.eqiad.wmnet with OS trixie * 11:00 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293095{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] (duration: 15m 50s) * 11:00 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1214: Upgrading db1214.eqiad.wmnet * 10:59 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1214: Upgrading db1214.eqiad.wmnet * 10:59 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92968 and previous config saved to /var/cache/conftool/dbconfig/20260526-105755-fceratto.json * 10:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance * 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92967 and previous config saved to /var/cache/conftool/dbconfig/20260526-105726-fceratto.json * 10:56 jiji@deploy1003: jiji: Continuing with deployment * 10:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2181.codfw.wmnet with reason: host reimage * 10:51 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2181.codfw.wmnet with reason: host reimage * 10:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P92966 and previous config saved to /var/cache/conftool/dbconfig/20260526-104718-fceratto.json * 10:46 jiji@deploy1003: jiji: Backport for [[gerrit:1293095{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:44 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1293095{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] * 10:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P92964 and previous config saved to /var/cache/conftool/dbconfig/20260526-103711-fceratto.json * 10:36 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2181.codfw.wmnet with OS trixie * 10:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply * 10:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply * 10:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92963 and previous config saved to /var/cache/conftool/dbconfig/20260526-102703-fceratto.json * 10:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1226: Migration of db1226.eqiad.wmnet completed * 10:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2181: Upgrading db2181.codfw.wmnet * 10:24 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2181: Upgrading db2181.codfw.wmnet * 10:24 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92960 and previous config saved to /var/cache/conftool/dbconfig/20260526-101936-fceratto.json * 10:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance * 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92959 and previous config saved to /var/cache/conftool/dbconfig/20260526-101842-fceratto.json * 10:16 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-codfw@codfw * 10:16 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 10:15 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 10:10 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293668{{!}}hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] (duration: 06m 42s) * 10:09 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-codfw@codfw * 10:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92957 and previous config saved to /var/cache/conftool/dbconfig/20260526-100834-fceratto.json * 10:06 kharlan@deploy1003: kharlan: Continuing with deployment * 10:05 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293668{{!}}hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:03 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293668{{!}}hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] * 10:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2195: Migration of db2195.codfw.wmnet completed * 10:01 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>kubestage200*<nowiki>}</nowiki> and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw) * 10:01 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet * 10:01 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet * 10:00 jmm@cumin2002: END (PASS) - Cookbook sre.netbox.restart-reboot (exit_code=0) rolling reboot on A:netbox * 09:58 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 09:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92955 and previous config saved to /var/cache/conftool/dbconfig/20260526-095827-fceratto.json * 09:58 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 09:58 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 09:57 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 09:56 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-eqiad@eqiad * 09:56 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:55 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 09:55 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:55 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:55 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet * 09:54 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet * 09:54 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet * 09:54 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet * 09:53 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>kubestage100*<nowiki>}</nowiki> and (A:wikikube-staging-master-eqiad or A:wikikube-staging-worker-eqiad) * 09:53 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet * 09:53 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet * 09:52 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-eqiad@eqiad * 09:52 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293665{{!}}hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] (duration: 08m 07s) * 09:51 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2043.* * 09:51 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2044.* * 09:48 fabfur: repooling cp2043 and cp2044 (haproxy-awslc) ([[phab:T419825|T419825]]) * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92953 and previous config saved to /var/cache/conftool/dbconfig/20260526-094819-fceratto.json * 09:47 kharlan@deploy1003: kharlan: Continuing with deployment * 09:46 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet * 09:45 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293665{{!}}hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:44 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs3009.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:44 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293665{{!}}hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] * 09:41 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet * 09:41 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet * 09:41 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet * 09:41 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92951 and previous config saved to /var/cache/conftool/dbconfig/20260526-094115-fceratto.json * 09:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2229.codfw.wmnet with reason: Maintenance * 09:41 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs3009.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92950 and previous config saved to /var/cache/conftool/dbconfig/20260526-094045-fceratto.json * 09:40 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1226: Migration of db1226.eqiad.wmnet completed * 09:39 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-codfw@codfw * 09:39 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:38 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:34 fabfur: depooling cp2044 to install haproxy-awslc ([[phab:T419825|T419825]]) * 09:34 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet * 09:34 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet * 09:34 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2044.* * 09:33 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet * 09:33 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet * 09:33 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet * 09:33 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2043.* * 09:32 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293661{{!}}hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] (duration: 06m 52s) * 09:32 fabfur: depooling cp2043 to install haproxy-awslc ([[phab:T419825|T419825]]) * 09:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1226.eqiad.wmnet with OS trixie * 09:30 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-codfw@codfw * 09:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P92947 and previous config saved to /var/cache/conftool/dbconfig/20260526-093031-fceratto.json * 09:29 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet * 09:29 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet * 09:29 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet * 09:28 kharlan@deploy1003: kharlan: Continuing with deployment * 09:28 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs3008.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:28 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293661{{!}}hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:27 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet * 09:26 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet * 09:26 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet * 09:26 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet * 09:26 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293661{{!}}hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] * 09:25 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs3008.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:25 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs3010.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet * 09:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet * 09:21 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs3010.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:20 fabfur: start rebooting esams liberica instances ([[phab:T426563|T426563]]) * 09:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P92946 and previous config saved to /var/cache/conftool/dbconfig/20260526-092024-fceratto.json * 09:20 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet * 09:16 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2195: Migration of db2195.codfw.wmnet completed * 09:15 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet * 09:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1226.eqiad.wmnet with reason: host reimage * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>kubestage100*<nowiki>}</nowiki> and (A:wikikube-staging-master-eqiad or A:wikikube-staging-worker-eqiad) * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>kubestage200*<nowiki>}</nowiki> and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw) * 09:14 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293658{{!}}Fix TypeError in Mandatory2FAChecker (T427251)]] (duration: 06m 47s) * 09:10 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1226.eqiad.wmnet with reason: host reimage * 09:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92944 and previous config saved to /var/cache/conftool/dbconfig/20260526-091016-fceratto.json * 09:09 mszwarc@deploy1003: mszwarc: Continuing with deployment * 09:09 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1293658{{!}}Fix TypeError in Mandatory2FAChecker (T427251)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2195.codfw.wmnet with OS trixie * 09:07 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293658{{!}}Fix TypeError in Mandatory2FAChecker (T427251)]] * 09:06 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs4009.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 09:03 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92943 and previous config saved to /var/cache/conftool/dbconfig/20260526-090315-fceratto.json * 09:03 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2224.codfw.wmnet with reason: Maintenance * 09:03 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs4009.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92942 and previous config saved to /var/cache/conftool/dbconfig/20260526-090256-fceratto.json * 08:57 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs4008.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 08:56 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox.discovery.wmnet. on all recursors * 08:56 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors * 08:55 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1226.eqiad.wmnet with OS trixie * 08:53 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs4008.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 08:53 fabfur: start rebooting ulsfo liberica instances ([[phab:T426563|T426563]]) * 08:53 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293594{{!}}Allow to remove passkeys when there's only one standard 2FA method (T426872)]] (duration: 07m 23s) * 08:53 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs5005.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:53 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1226: Upgrading db1226.eqiad.wmnet * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P92941 and previous config saved to /var/cache/conftool/dbconfig/20260526-085248-fceratto.json * 08:51 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox.discovery.wmnet. on all recursors * 08:51 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors * 08:51 jmm@cumin2002: START - Cookbook sre.netbox.restart-reboot rolling reboot on A:netbox * 08:50 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1226: Upgrading db1226.eqiad.wmnet * 08:50 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs5005.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:50 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2195.codfw.wmnet with reason: host reimage * 08:49 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1222: Migration of db1222.eqiad.wmnet completed * 08:48 mszwarc@deploy1003: mszwarc: Continuing with deployment * 08:47 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1293594{{!}}Allow to remove passkeys when there's only one standard 2FA method (T426872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:46 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293594{{!}}Allow to remove passkeys when there's only one standard 2FA method (T426872)]] * 08:43 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs5004.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet * 08:43 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2195.codfw.wmnet with reason: host reimage * 08:43 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1292032{{!}}Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964{{!}}hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] (duration: 09m 56s) * 08:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P92939 and previous config saved to /var/cache/conftool/dbconfig/20260526-084240-fceratto.json * 08:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1222.eqiad.wmnet with OS trixie * 08:40 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs5004.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:40 fabfur: start rebooting eqsin liberica instances ([[phab:T426563|T426563]]) * 08:39 kartik@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 08:39 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet * 08:39 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 08:39 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs5006.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:35 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs5006.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1024.eqiad.wmnet * 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1024.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 08:35 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1292032{{!}}Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964{{!}}hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:33 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs6002.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:33 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1292032{{!}}Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964{{!}}hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] * 08:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92938 and previous config saved to /var/cache/conftool/dbconfig/20260526-083233-fceratto.json * 08:30 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs6002.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:25 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2217 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92937 and previous config saved to /var/cache/conftool/dbconfig/20260526-082531-fceratto.json * 08:25 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2217.codfw.wmnet with reason: Maintenance * 08:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92936 and previous config saved to /var/cache/conftool/dbconfig/20260526-082458-fceratto.json * 08:23 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2195.codfw.wmnet with OS trixie * 08:23 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage * 08:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2195: Upgrading db2195.codfw.wmnet * 08:20 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2195: Upgrading db2195.codfw.wmnet * 08:19 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:18 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage * 08:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P92934 and previous config saved to /var/cache/conftool/dbconfig/20260526-081451-fceratto.json * 08:13 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs6001.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 08:10 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs6001.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:09 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1024.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 08:04 jmm@cumin2002: START - Cookbook sre.dns.netbox * 08:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P92932 and previous config saved to /var/cache/conftool/dbconfig/20260526-080443-fceratto.json * 08:01 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1222.eqiad.wmnet with OS trixie * 08:00 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs6003.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet * 07:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet * 07:59 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1024.eqiad.wmnet * 07:59 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1023.eqiad.wmnet * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1023.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:59 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 07:59 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:58 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1023.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:56 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs6003.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 07:56 fabfur: start rebooting drmrs liberica instances ([[phab:T426563|T426563]]) * 07:56 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs7002.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:54 jmm@cumin2002: START - Cookbook sre.dns.netbox * 07:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92931 and previous config saved to /var/cache/conftool/dbconfig/20260526-075435-fceratto.json * 07:52 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs7002.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:51 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1047.eqiad.wmnet * 07:51 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:51 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1047.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:49 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1023.eqiad.wmnet * 07:47 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2193 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92930 and previous config saved to /var/cache/conftool/dbconfig/20260526-074739-fceratto.json * 07:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2193.codfw.wmnet with reason: Maintenance * 07:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92929 and previous config saved to /var/cache/conftool/dbconfig/20260526-074710-fceratto.json * 07:46 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet * 07:45 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet * 07:45 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:45 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs7001.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:44 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1025.eqiad.wmnet * 07:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:43 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:41 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs7001.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:40 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1046.eqiad.wmnet * 07:40 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1046.eqiad.wmnet * 07:38 arthurtaylor@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291951{{!}}Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] (duration: 12m 01s) * 07:38 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1047.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P92928 and previous config saved to /var/cache/conftool/dbconfig/20260526-073702-fceratto.json * 07:37 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet * 07:36 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet * 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance * 07:35 fabfur: start rebooting magru liberica instances ([[phab:T426563|T426563]]) * 07:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92926 and previous config saved to /var/cache/conftool/dbconfig/20260526-073459-fceratto.json * 07:32 arthurtaylor@deploy1003: arthurtaylor: Continuing with deployment * 07:31 arthurtaylor@deploy1003: arthurtaylor: Backport for [[gerrit:1291951{{!}}Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:30 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1046.eqiad.wmnet * 07:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20260526-072643-fceratto.json * 07:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1046.eqiad.wmnet * 07:26 arthurtaylor@deploy1003: Started scap sync-world: Backport for [[gerrit:1291951{{!}}Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] * 07:25 jiji@cumin1003: START - Cookbook sre.dns.netbox * 07:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P92924 and previous config saved to /var/cache/conftool/dbconfig/20260526-072452-fceratto.json * 07:24 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet * 07:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1047.eqiad.wmnet * 07:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1047.eqiad.wmnet * 07:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92923 and previous config saved to /var/cache/conftool/dbconfig/20260526-071635-fceratto.json * 07:15 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet * 07:15 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1026.eqiad.wmnet * 07:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P92922 and previous config saved to /var/cache/conftool/dbconfig/20260526-071444-fceratto.json * 07:13 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet * 07:11 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1025.eqiad.wmnet * 07:10 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1025.eqiad.wmnet * 07:09 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92921 and previous config saved to /var/cache/conftool/dbconfig/20260526-070946-fceratto.json * 07:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2180.codfw.wmnet with reason: Maintenance * 07:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92920 and previous config saved to /var/cache/conftool/dbconfig/20260526-070916-fceratto.json * 07:09 moritzm: failover Ganeti master in eqiad to ganeti1048 * 07:09 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1047.eqiad.wmnet * 07:07 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1046.eqiad.wmnet * 07:07 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:06 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1046.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org * 07:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92919 and previous config saved to /var/cache/conftool/dbconfig/20260526-070436-fceratto.json * 07:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet * 07:04 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1046.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet * 07:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org * 06:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P92918 and previous config saved to /var/cache/conftool/dbconfig/20260526-065909-fceratto.json * 06:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org * 06:58 jiji@cumin1003: START - Cookbook sre.dns.netbox * 06:58 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet * 06:55 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet * 06:53 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1046.eqiad.wmnet * 06:53 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1045.eqiad.wmnet * 06:53 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:53 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1045.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 06:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org * 06:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P92917 and previous config saved to /var/cache/conftool/dbconfig/20260526-064901-fceratto.json * 06:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92916 and previous config saved to /var/cache/conftool/dbconfig/20260526-064833-fceratto.json * 06:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1222.eqiad.wmnet with reason: Maintenance * 06:47 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1222: Switchover * 06:41 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92914 and previous config saved to /var/cache/conftool/dbconfig/20260526-063853-fceratto.json * 06:35 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 06:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92912 and previous config saved to /var/cache/conftool/dbconfig/20260526-063155-fceratto.json * 06:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance * 06:28 fceratto@cumin1003: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance * 06:23 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1222: Switchover * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1222 [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92910 and previous config saved to /var/cache/conftool/dbconfig/20260526-061656-fceratto.json * 06:15 fceratto@dns1005: END - running authdns-update * 06:14 fceratto@dns1005: START - running authdns-update * 06:11 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1162 to s2 primary and set section read-write [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92909 and previous config saved to /var/cache/conftool/dbconfig/20260526-061114-fceratto.json * 06:10 fceratto@cumin1003: dbctl commit (dc=all): 'Set s2 eqiad as read-only for maintenance - [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92908 and previous config saved to /var/cache/conftool/dbconfig/20260526-061021-fceratto.json * 06:10 federico3: Starting s2 eqiad failover from db1222 to db1162 - [[phab:T425622|T425622]] * 06:04 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1162 with weight 0 [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92907 and previous config saved to /var/cache/conftool/dbconfig/20260526-060443-fceratto.json * 06:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s2 [[phab:T425622|T425622]] * 06:02 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:02 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:01 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:00 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 05:15 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1014.eqiad.wmnet: Maintenance on pc4 * 05:15 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 05:15 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 05:15 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1014.eqiad.wmnet: Maintenance on pc4 * 05:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2024.codfw.wmnet,pc[1014,1024].eqiad.wmnet with reason: Maintenance on pc4 * 04:37 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 04:34 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.1 (duration: 02m 32s) * 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] (duration: 36m 24s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 20s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-25 == * 21:00 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1045.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:49 jiji@cumin1003: START - Cookbook sre.dns.netbox * 20:38 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1045.eqiad.wmnet * 20:37 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1044.eqiad.wmnet * 20:37 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:37 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1044.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:25 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1044.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:15 moritzm: truncate krb5kdc.log1 (which made log rotation fail) * 20:06 jiji@cumin1003: START - Cookbook sre.dns.netbox * 19:57 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1044.eqiad.wmnet * 19:25 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1043.eqiad.wmnet * 19:25 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:25 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1043.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 19:22 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1043.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 18:49 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_eqiad * 18:49 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1115.eqiad.wmnet * 18:34 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5023.eqsin.wmnet [reason: manually pooling after reboot as icinga was down] * 18:33 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5030.eqsin.wmnet [reason: manually pooling after reboot as icinga was down] * 18:22 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp5030*<nowiki>}</nowiki> and A:cp * 18:22 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5030.eqsin.wmnet * 18:15 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp5023*<nowiki>}</nowiki> and A:cp * 18:15 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5023.eqsin.wmnet * 18:10 jiji@cumin1003: START - Cookbook sre.dns.netbox * 18:10 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp5030*<nowiki>}</nowiki> and A:cp * 18:09 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp1113*<nowiki>}</nowiki> and A:cp * 18:09 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet * 18:09 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet * 18:03 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp1113*<nowiki>}</nowiki> and A:cp * 18:02 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp5023*<nowiki>}</nowiki> and A:cp * 18:01 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-text_eqiad * 18:01 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-upload_eqsin * 18:01 sukhe: sre.cdn.roll-reboot cookbooks stalled due to icinga reboot * 18:00 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-text_eqsin * 17:35 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1043.eqiad.wmnet * 17:31 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1110.eqiad.wmnet [reason: manually pooling after reboot as icinga was down] * 17:30 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1042.eqiad.wmnet * 17:30 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:30 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1042.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:29 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1111.eqiad.wmnet * 17:28 sukhe: sukhe@alert1002:~$ sudo systemctl restart icinga.service * 17:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92903 and previous config saved to /var/cache/conftool/dbconfig/20260525-171310-fceratto.json * 17:11 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1042.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:06 jiji@cumin1003: START - Cookbook sre.dns.netbox * 17:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P92902 and previous config saved to /var/cache/conftool/dbconfig/20260525-170302-fceratto.json * 16:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P92901 and previous config saved to /var/cache/conftool/dbconfig/20260525-165255-fceratto.json * 16:51 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1042.eqiad.wmnet * 16:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92900 and previous config saved to /var/cache/conftool/dbconfig/20260525-164247-fceratto.json * 16:42 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1041.eqiad.wmnet * 16:42 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:42 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1041.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:41 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1041.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:40 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5021.eqsin.wmnet * 16:39 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5029.eqsin.wmnet * 16:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92899 and previous config saved to /var/cache/conftool/dbconfig/20260525-163559-fceratto.json * 16:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2158.codfw.wmnet with reason: Maintenance * 16:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92898 and previous config saved to /var/cache/conftool/dbconfig/20260525-163512-fceratto.json * 16:34 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1108.eqiad.wmnet * 16:30 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1109.eqiad.wmnet * 16:26 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249', diff saved to https://phabricator.wikimedia.org/P92897 and previous config saved to /var/cache/conftool/dbconfig/20260525-162505-fceratto.json * 16:20 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1041.eqiad.wmnet * 16:20 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1040.eqiad.wmnet * 16:20 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:20 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1040.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:16 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1040.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249', diff saved to https://phabricator.wikimedia.org/P92896 and previous config saved to /var/cache/conftool/dbconfig/20260525-161457-fceratto.json * 16:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92895 and previous config saved to /var/cache/conftool/dbconfig/20260525-160450-fceratto.json * 16:02 jiji@cumin1003: START - Cookbook sre.dns.netbox * 15:59 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2249 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92894 and previous config saved to /var/cache/conftool/dbconfig/20260525-155930-fceratto.json * 15:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2249.codfw.wmnet with reason: Maintenance * 15:57 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5020.eqsin.wmnet * 15:57 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5028.eqsin.wmnet * 15:52 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1106.eqiad.wmnet * 15:51 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1107.eqiad.wmnet * 15:29 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1040.eqiad.wmnet * 15:29 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1039.eqiad.wmnet * 15:29 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:29 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1039.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:27 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1039.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:17 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1013 from dbctl [[phab:T427190|T427190]]', diff saved to https://phabricator.wikimedia.org/P92893 and previous config saved to /var/cache/conftool/dbconfig/20260525-151718-marostegui.json * 15:15 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5019.eqsin.wmnet * 15:15 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5027.eqsin.wmnet * 15:12 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1104.eqiad.wmnet * 15:11 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1105.eqiad.wmnet * 15:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92892 and previous config saved to /var/cache/conftool/dbconfig/20260525-150309-fceratto.json * 14:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P92891 and previous config saved to /var/cache/conftool/dbconfig/20260525-145301-fceratto.json * 14:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P92890 and previous config saved to /var/cache/conftool/dbconfig/20260525-144253-fceratto.json * 14:33 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1102.eqiad.wmnet * 14:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92889 and previous config saved to /var/cache/conftool/dbconfig/20260525-143246-fceratto.json * 14:32 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5026.eqsin.wmnet * 14:32 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5018.eqsin.wmnet * 14:31 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1103.eqiad.wmnet * 14:25 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2228 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92888 and previous config saved to /var/cache/conftool/dbconfig/20260525-142551-fceratto.json * 14:25 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance * 14:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92887 and previous config saved to /var/cache/conftool/dbconfig/20260525-142520-fceratto.json * 14:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P92885 and previous config saved to /var/cache/conftool/dbconfig/20260525-141513-fceratto.json * 14:12 jiji@cumin1003: START - Cookbook sre.dns.netbox * 14:06 sukhe: curl localhost:9090/pools/inference-staging-grpc_30051 shows ml-staging200[1-3].codfw.wmnet as enabled and pooled: [[phab:T424049|T424049]] * 14:05 sukhe: sukhe@lvs2013:~$ sudo systemctl restart pybal.service: [[phab:T424049|T424049]] * 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P92884 and previous config saved to /var/cache/conftool/dbconfig/20260525-140505-fceratto.json * 14:03 sukhe: sudo cumin 'A:lvs and A:lvs-low-traffic-codfw' 'run-puppet-agent --enable "adding new ml-serve (grpc) [[phab:T424049|T424049]]"' * 14:02 sukhe: sukhe@lvs2014:~$ sudo systemctl restart pybal.service": [[phab:T424049|T424049]] * 14:02 sukhe: sukhe@lvs2014:~$ sudo systemctl restart pybal.service * 14:00 sukhe: sudo cumin 'A:lvs and A:lvs-secondary-codfw' 'run-puppet-agent --enable "adding new ml-serve (grpc) [[phab:T424049|T424049]]"' * 13:59 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1039.eqiad.wmnet * 13:58 sukhe: sudo cumin 'A:lvs and A:eqiad' 'run-puppet-agent --enable "adding new ml-serve (grpc) [[phab:T424049|T424049]]": NOOP change, since service is codfw only * 13:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92882 and previous config saved to /var/cache/conftool/dbconfig/20260525-135458-fceratto.json * 13:52 Msz2001: Everything deployed, UTC afternoon config+backport window done * 13:52 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293119{{!}}Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] (duration: 09m 43s) * 13:51 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1101.eqiad.wmnet * 13:51 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1100.eqiad.wmnet * 13:50 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5025.eqsin.wmnet * 13:50 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5017.eqsin.wmnet * 13:49 kart_: Updated Recommendation API to 2026-05-21-044522-production * 13:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2223 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92881 and previous config saved to /var/cache/conftool/dbconfig/20260525-134807-fceratto.json * 13:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2223.codfw.wmnet with reason: Maintenance * 13:47 mszwarc@deploy1003: vadymts1, mszwarc: Continuing with deployment * 13:47 kartik@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 13:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92880 and previous config saved to /var/cache/conftool/dbconfig/20260525-134737-fceratto.json * 13:45 mszwarc@deploy1003: vadymts1, mszwarc: Backport for [[gerrit:1293119{{!}}Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:45 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1162: Reboot * 13:43 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293119{{!}}Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] * 13:40 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqiad * 13:39 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqiad * 13:38 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290813{{!}}Article Guidance: enable experiment on phase 2 wikis (T426871)]] (duration: 08m 14s) * 13:38 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqsin * 13:38 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqsin * 13:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P92878 and previous config saved to /var/cache/conftool/dbconfig/20260525-133729-fceratto.json * 13:34 sbisson@deploy1003: sbisson: Continuing with deployment * 13:33 kartik@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 13:32 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1038.eqiad.wmnet * 13:32 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:32 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1038.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 13:31 sbisson@deploy1003: sbisson: Backport for [[gerrit:1290813{{!}}Article Guidance: enable experiment on phase 2 wikis (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:30 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1290813{{!}}Article Guidance: enable experiment on phase 2 wikis (T426871)]] * 13:27 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293094{{!}}Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953{{!}}Update logo, wordmark and tagline for zghwiki (T426406)]] (duration: 07m 43s) * 13:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P92876 and previous config saved to /var/cache/conftool/dbconfig/20260525-132722-fceratto.json * 13:23 mszwarc@deploy1003: mszwarc, jhsoby: Continuing with deployment * 13:21 mszwarc@deploy1003: mszwarc, jhsoby: Backport for [[gerrit:1293094{{!}}Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953{{!}}Update logo, wordmark and tagline for zghwiki (T426406)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:20 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1038.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 13:20 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293094{{!}}Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953{{!}}Update logo, wordmark and tagline for zghwiki (T426406)]] * 13:19 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291966{{!}}Modify various configurations for English Wikibooks (T426992)]] (duration: 15m 53s) * 13:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92875 and previous config saved to /var/cache/conftool/dbconfig/20260525-131714-fceratto.json * 13:12 mszwarc@deploy1003: vadymts1, mszwarc: Continuing with deployment * 13:12 jiji@cumin1003: START - Cookbook sre.dns.netbox * 13:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92873 and previous config saved to /var/cache/conftool/dbconfig/20260525-131023-fceratto.json * 13:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2211.codfw.wmnet with reason: Maintenance * 13:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92872 and previous config saved to /var/cache/conftool/dbconfig/20260525-130950-fceratto.json * 13:07 mszwarc@deploy1003: vadymts1, mszwarc: Backport for [[gerrit:1291966{{!}}Modify various configurations for English Wikibooks (T426992)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:03 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1291966{{!}}Modify various configurations for English Wikibooks (T426992)]] * 12:59 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1162: Reboot * 12:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P92870 and previous config saved to /var/cache/conftool/dbconfig/20260525-125942-fceratto.json * 12:59 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1162: Reboot * 12:59 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1162: Reboot * 12:58 kart_: Updated cxserver to 2026-05-24-103047-production ([[phab:T426808|T426808]], [[phab:T373418|T373418]]) * 12:56 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply * 12:56 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply * 12:54 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db1162: Reboot * 12:54 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1162: Reboot * 12:54 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply * 12:53 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply * 12:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1162.eqiad.wmnet with reason: Reboot * 12:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P92868 and previous config saved to /var/cache/conftool/dbconfig/20260525-124934-fceratto.json * 12:40 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply * 12:39 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply * 12:39 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1038.eqiad.wmnet * 12:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92867 and previous config saved to /var/cache/conftool/dbconfig/20260525-123927-fceratto.json * 12:32 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2192 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92866 and previous config saved to /var/cache/conftool/dbconfig/20260525-123239-fceratto.json * 12:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2192.codfw.wmnet with reason: Maintenance * 12:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92865 and previous config saved to /var/cache/conftool/dbconfig/20260525-123208-fceratto.json * 12:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P92864 and previous config saved to /var/cache/conftool/dbconfig/20260525-122201-fceratto.json * 12:17 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1037.eqiad.wmnet * 12:17 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:17 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1037.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 12:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P92863 and previous config saved to /var/cache/conftool/dbconfig/20260525-121153-fceratto.json * 12:10 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1037.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 12:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92862 and previous config saved to /var/cache/conftool/dbconfig/20260525-120145-fceratto.json * 11:58 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2178 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92861 and previous config saved to /var/cache/conftool/dbconfig/20260525-115504-fceratto.json * 11:54 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2178.codfw.wmnet with reason: Maintenance * 11:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92860 and previous config saved to /var/cache/conftool/dbconfig/20260525-115434-fceratto.json * 11:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P92859 and previous config saved to /var/cache/conftool/dbconfig/20260525-114426-fceratto.json * 11:43 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1037.eqiad.wmnet * 11:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P92858 and previous config saved to /var/cache/conftool/dbconfig/20260525-113419-fceratto.json * 11:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2160.codfw.wmnet with OS trixie * 11:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92857 and previous config saved to /var/cache/conftool/dbconfig/20260525-112411-fceratto.json * 11:17 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2171 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92856 and previous config saved to /var/cache/conftool/dbconfig/20260525-111717-fceratto.json * 11:17 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: Maintenance * 11:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92855 and previous config saved to /var/cache/conftool/dbconfig/20260525-111648-fceratto.json * 11:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P92854 and previous config saved to /var/cache/conftool/dbconfig/20260525-110640-fceratto.json * 11:05 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2160.codfw.wmnet with reason: host reimage * 11:00 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2160.codfw.wmnet with reason: host reimage * 10:58 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:57 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:56 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P92853 and previous config saved to /var/cache/conftool/dbconfig/20260525-105633-fceratto.json * 10:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92852 and previous config saved to /var/cache/conftool/dbconfig/20260525-104625-fceratto.json * 10:43 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2160.codfw.wmnet with OS trixie * 10:41 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc3 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92851 and previous config saved to /var/cache/conftool/dbconfig/20260525-104141-marostegui.json * 10:40 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1023 to pc3 as master [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92850 and previous config saved to /var/cache/conftool/dbconfig/20260525-104055-marostegui.json * 10:40 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1023 to dbctl', diff saved to https://phabricator.wikimedia.org/P92849 and previous config saved to /var/cache/conftool/dbconfig/20260525-104027-marostegui.json * 10:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92848 and previous config saved to /var/cache/conftool/dbconfig/20260525-103944-fceratto.json * 10:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance * 10:31 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply * 10:30 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply * 10:27 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 10:18 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 10:16 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1011.eqiad.wmnet * 10:08 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1011.eqiad.wmnet * 10:08 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1007.eqiad.wmnet * 09:59 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1007.eqiad.wmnet * 09:59 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1006.eqiad.wmnet * 09:57 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:49 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1006.eqiad.wmnet * 09:48 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:46 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:45 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:40 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:40 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:28 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:17 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:13 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92847 and previous config saved to /var/cache/conftool/dbconfig/20260525-091302-fceratto.json * 09:12 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231', diff saved to https://phabricator.wikimedia.org/P92846 and previous config saved to /var/cache/conftool/dbconfig/20260525-090255-fceratto.json * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231', diff saved to https://phabricator.wikimedia.org/P92845 and previous config saved to /var/cache/conftool/dbconfig/20260525-085247-fceratto.json * 08:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92844 and previous config saved to /var/cache/conftool/dbconfig/20260525-084239-fceratto.json * 08:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92843 and previous config saved to /var/cache/conftool/dbconfig/20260525-083540-fceratto.json * 08:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2231.codfw.wmnet with reason: Maintenance * 08:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92842 and previous config saved to /var/cache/conftool/dbconfig/20260525-083511-fceratto.json * 08:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215', diff saved to https://phabricator.wikimedia.org/P92841 and previous config saved to /var/cache/conftool/dbconfig/20260525-082504-fceratto.json * 08:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215', diff saved to https://phabricator.wikimedia.org/P92840 and previous config saved to /var/cache/conftool/dbconfig/20260525-081456-fceratto.json * 08:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92839 and previous config saved to /var/cache/conftool/dbconfig/20260525-080448-fceratto.json * 07:57 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2215 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92838 and previous config saved to /var/cache/conftool/dbconfig/20260525-075739-fceratto.json * 07:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2215.codfw.wmnet with reason: Maintenance * 07:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92837 and previous config saved to /var/cache/conftool/dbconfig/20260525-075708-fceratto.json * 07:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196', diff saved to https://phabricator.wikimedia.org/P92836 and previous config saved to /var/cache/conftool/dbconfig/20260525-074700-fceratto.json * 07:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196', diff saved to https://phabricator.wikimedia.org/P92835 and previous config saved to /var/cache/conftool/dbconfig/20260525-073653-fceratto.json * 07:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92834 and previous config saved to /var/cache/conftool/dbconfig/20260525-072645-fceratto.json * 07:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92833 and previous config saved to /var/cache/conftool/dbconfig/20260525-071953-fceratto.json * 07:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2196.codfw.wmnet with reason: Maintenance * 07:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92832 and previous config saved to /var/cache/conftool/dbconfig/20260525-071924-fceratto.json * 07:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186', diff saved to https://phabricator.wikimedia.org/P92831 and previous config saved to /var/cache/conftool/dbconfig/20260525-070917-fceratto.json * 07:03 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2233.codfw.wmnet with OS trixie * 06:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186', diff saved to https://phabricator.wikimedia.org/P92830 and previous config saved to /var/cache/conftool/dbconfig/20260525-065909-fceratto.json * 06:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92829 and previous config saved to /var/cache/conftool/dbconfig/20260525-064902-fceratto.json * 06:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92828 and previous config saved to /var/cache/conftool/dbconfig/20260525-064305-fceratto.json * 06:42 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance * 06:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2233.codfw.wmnet with reason: host reimage * 06:35 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2233.codfw.wmnet with reason: host reimage * 06:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2233.codfw.wmnet with OS trixie * 06:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2233.codfw.wmnet with reason: Reimage to Trixie * 06:17 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 06:17 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:15 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2160.codfw.wmnet with reason: Reboot upgrade m2 * 06:15 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2233.codfw.wmnet with reason: Reboot upgrade m2 * 06:08 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy1027.eqiad.wmnet with reason: Reboot * 05:18 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2023.codfw.wmnet,pc[1013,1023].eqiad.wmnet with reason: Maintenance on pc3 * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1013.eqiad.wmnet: Maintenance on pc3 * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 05:17 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 05:17 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1013.eqiad.wmnet: Maintenance on pc3 * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 43s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-24 == * 19:08 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down * 02:06 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 23s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-23 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 35s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-22 == * 23:39 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 23:39 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 23:39 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 23:39 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 23:38 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 23:37 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 23:37 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 23:37 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 22:20 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 22:12 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 22:11 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 20:29 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 20:28 inflatador: bking@deploy1003 set eqiad prod cirrus `node_concurrent_recoveries` up to 7 from 4 [[phab:T426585|T426585]] * 20:27 inflatador: bking@deploy1003 set codfw prod cirrus `node_concurrent_recoveries` back down to 4 from 7 [[phab:T426585|T426585]] * 18:39 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 17:34 topranks: enable ttl protection on esams CRs IBGP session * 17:28 topranks: enable ttl protection on ulsfo CRs IBGP session * 16:50 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 16:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 16:16 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 16:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 16:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 15:58 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 15:14 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 15:02 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 15:02 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 14:34 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudnet2008-dev.codfw.wmnet * 14:34 andrew@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:34 andrew@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2008-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 14:33 andrew@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2008-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 14:33 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb[1020,1022-1025].eqiad.wmnet * 14:29 andrew@cumin2002: START - Cookbook sre.dns.netbox * 14:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 14:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 14:23 andrew@cumin2002: START - Cookbook sre.hosts.decommission for hosts cloudnet2008-dev.codfw.wmnet * 14:23 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudnet2007-dev.codfw.wmnet * 14:23 andrew@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:23 andrew@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2007-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 14:03 andrew@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2007-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 13:59 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb[1020,1022-1025].eqiad.wmnet * 13:58 andrew@cumin2002: START - Cookbook sre.dns.netbox * 13:53 andrew@cumin2002: START - Cookbook sre.hosts.decommission for hosts cloudnet2007-dev.codfw.wmnet * 13:52 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1018.eqiad.wmnet * 13:50 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 13:50 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 13:46 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1018.eqiad.wmnet * 13:25 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for clouddb1018.eqiad.wmnet * 13:25 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1018.eqiad.wmnet * 13:25 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for 6 hosts * 13:16 inflatador: bking@deploy1002 set search_codfw cluster recovery settings from 4 to 7 [[phab:T426560|T426560]] * 13:15 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for 6 hosts * 13:15 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 13:11 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp5017.eqsin.wmnet<nowiki>}</nowiki> and A:cp * 13:11 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5017.eqsin.wmnet * 13:10 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1017.eqiad.wmnet * 13:09 elukey: uploaded spicerack_12.6.0 to apt.wikimedia.org bookworm-wikimedia * 13:08 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for clouddb1017.eqiad.wmnet * 12:59 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp5017.eqsin.wmnet<nowiki>}</nowiki> and A:cp * 12:57 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp308[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 12:57 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3081.esams.wmnet * 12:54 isaranto@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 12:41 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 12:15 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3080.esams.wmnet * 12:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 12:03 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp308[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[2-3].esams.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3073.esams.wmnet * 11:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2154: Migration of db2154.codfw.wmnet completed * 11:19 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3072.esams.wmnet * 11:15 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 11:11 fnegri@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb1017.eqiad.wmnet with reason: Rebooting clouddb1017 * 11:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1172: Migration of db1172.eqiad.wmnet completed * 11:07 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[2-3].esams.wmnet<nowiki>}</nowiki> and A:cp * 11:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1058.eqiad.wmnet * 11:01 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 11:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3079.esams.wmnet * 10:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1058.eqiad.wmnet * 10:55 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 10:55 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 10:48 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 10:47 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 10:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1024.eqiad.wmnet * 10:43 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:43 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:43 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:42 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:42 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:42 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2154: Migration of db2154.codfw.wmnet completed * 10:42 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:41 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1024.eqiad.wmnet * 10:37 moritzm: remove ganeti1024 foom eqiad Ganeti cluster [[phab:T424680|T424680]] * 10:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2154.codfw.wmnet with OS trixie * 10:31 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2010.codfw.wmnet with OS trixie * 10:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1024.eqiad.wmnet * 10:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1172: Migration of db1172.eqiad.wmnet completed * 10:19 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3078.esams.wmnet * 10:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2154.codfw.wmnet with reason: host reimage * 10:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1172.eqiad.wmnet with OS trixie * 10:15 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1017.eqiad.wmnet * 10:13 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2154.codfw.wmnet with reason: host reimage * 10:07 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 10:06 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 10:06 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3071.esams.wmnet * 09:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1172.eqiad.wmnet with reason: host reimage * 09:56 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2154.codfw.wmnet with OS trixie * 09:55 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage * 09:53 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1172.eqiad.wmnet with reason: host reimage * 09:51 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage * 09:39 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2154: Upgrading db2154.codfw.wmnet * 09:39 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2154: Upgrading db2154.codfw.wmnet * 09:38 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:38 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1172.eqiad.wmnet with OS trixie * 09:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1172: Upgrading db1172.eqiad.wmnet * 09:34 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1172: Upgrading db1172.eqiad.wmnet * 09:34 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:34 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2009.codfw.wmnet with OS trixie * 09:33 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS trixie * 09:26 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 09:26 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 09:26 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3070.esams.wmnet * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 09:16 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie * 09:14 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 09:11 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[6-7].esams.wmnet<nowiki>}</nowiki> and A:cp * 09:11 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3077.esams.wmnet * 09:04 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 09:03 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie * 08:47 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 08:46 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie * 08:40 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 08:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 08:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 08:30 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3076.esams.wmnet * 08:18 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[6-7].esams.wmnet<nowiki>}</nowiki> and A:cp * 08:15 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti1058.eqiad.wmnet on all recursors * 08:15 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:15 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change records for ganeti1058 - cmooney@cumin1003" * 08:15 cmooney@cumin1003: START - Cookbook sre.dns.wipe-cache ganeti1058.eqiad.wmnet on all recursors * 08:15 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change records for ganeti1058 - cmooney@cumin1003" * 08:09 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 08:07 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp306[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 08:07 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3069.esams.wmnet * 08:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 08:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 07:31 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1024.eqiad.wmnet * 07:26 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3068.esams.wmnet * 07:14 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp306[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 07:11 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1057.eqiad.wmnet to cluster eqiad and group A * 07:10 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3075.esams.wmnet<nowiki>}</nowiki> and A:cp * 07:10 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3075.esams.wmnet * 07:06 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1057.eqiad.wmnet to cluster eqiad and group A * 07:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1057.eqiad.wmnet * 07:02 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1057 * 07:01 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1057 * 06:58 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3075.esams.wmnet<nowiki>}</nowiki> and A:cp * 06:58 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3067.esams.wmnet<nowiki>}</nowiki> and A:cp * 06:58 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3067.esams.wmnet * 06:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1057.eqiad.wmnet * 06:46 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3067.esams.wmnet<nowiki>}</nowiki> and A:cp * 06:13 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1024.eqiad.wmnet * 06:08 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1024.eqiad.wmnet * 06:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org * 06:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org * 05:25 marostegui@dns1004: END - running authdns-update * 05:24 marostegui@dns1004: START - running authdns-update * 05:23 marostegui: Failover m5-master [[phab:T426633|T426633]] * 05:19 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy1028.eqiad.wmnet with reason: Reboot * 05:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy2005.codfw.wmnet with reason: Reboot * 05:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1012.eqiad.wmnet * 05:11 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 05:11 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1012.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 05:06 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1012.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 05:03 marostegui@cumin1003: START - Cookbook sre.dns.netbox * 04:56 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1012.eqiad.wmnet == 2026-05-21 == * 23:43 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290954{{!}}Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957{{!}}Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958{{!}}Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960{{!}}Drop unused wpReportIncidentLocalLinks]] (duration: 06m 42s) * 23:38 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 23:38 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1290954{{!}}Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957{{!}}Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958{{!}}Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960{{!}}Drop unused wpReportIncidentLocalLinks]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified * 23:36 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1290954{{!}}Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957{{!}}Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958{{!}}Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960{{!}}Drop unused wpReportIncidentLocalLinks]] * 22:26 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul2002.codfw.wmnet with OS trixie * 22:08 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul2002.codfw.wmnet with reason: host reimage * 22:03 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul2002.codfw.wmnet with reason: host reimage * 22:02 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 21:49 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 21:49 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 21:44 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host zuul2002.codfw.wmnet with OS trixie * 21:25 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:25 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 20:26 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 20:16 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 19:22 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-reboot (exit_code=0) rolling reboot on A:restbase * 19:10 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:59 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:53 papaul: rebooting msw1-codfw * 18:50 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:39 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:52 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:52 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:50 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:49 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:49 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:48 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:46 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 17:46 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 17:43 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:43 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:43 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:42 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:42 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:41 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:41 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:41 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:41 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:41 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:41 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:41 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:40 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:40 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:40 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:39 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 * 17:39 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:38 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down * 17:37 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 17:36 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:36 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:30 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:25 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:25 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:24 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:23 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:22 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1016.eqiad.wmnet * 17:22 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:14 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2031.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:14 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2030.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:13 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1016.eqiad.wmnet * 17:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:08 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repool pc2 ([[phab:T421705|T421705]])', diff saved to https://phabricator.wikimedia.org/P92810 and previous config saved to /var/cache/conftool/dbconfig/20260521-170823-ladsgroup.json * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:07 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2031.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:07 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2030.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:06 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:03 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:03 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:00 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2029 * 16:58 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2031 * 16:58 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:58 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 * 16:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 * 16:55 papaul: rebooting msw-d3-codfw * 16:55 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 16:52 papaul: rebooting msw-c7-codfw * 16:51 papaul: rebooting msw-c6-codfw * 16:48 papaul: rebooting msw-b7-codfw * 16:48 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1014.eqiad.wmnet * 16:45 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1014.eqiad.wmnet * 16:43 papaul: rebooting msw-b6-codfw * 16:40 papaul: rebooting msw-a1-codfw * 16:37 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 16:37 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1014.eqiad.wmnet * 16:37 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:36 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:35 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 16:35 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2030 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2030 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 * 16:34 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 16:34 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:33 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2028 to codfw - jhancock@cumin2002" * 16:33 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2028 to codfw - jhancock@cumin2002" * 16:26 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 16:24 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on pc1022.eqiad.wmnet with reason: Move to nftables * 16:24 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on pc2022.codfw.wmnet with reason: Move to nftables * 16:18 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2048: Repooling * 16:18 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depool pc2 ([[phab:T421705|T421705]])', diff saved to https://phabricator.wikimedia.org/P92807 and previous config saved to /var/cache/conftool/dbconfig/20260521-161808-ladsgroup.json * 16:15 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:15 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:15 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:15 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:52 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 15:42 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es2048: Repooling * 15:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92804 and previous config saved to /var/cache/conftool/dbconfig/20260521-154108-fceratto.json * 15:39 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:38 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:34 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:34 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:34 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:34 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:34 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92803 and previous config saved to /var/cache/conftool/dbconfig/20260521-153400-fceratto.json * 15:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2048.codfw.wmnet with reason: Maintenance * 15:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92802 and previous config saved to /var/cache/conftool/dbconfig/20260521-153331-fceratto.json * 15:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:25 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:24 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:24 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:24 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:24 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040', diff saved to https://phabricator.wikimedia.org/P92801 and previous config saved to /var/cache/conftool/dbconfig/20260521-152323-fceratto.json * 15:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1045.eqiad.wmnet * 15:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1045.eqiad.wmnet * 15:19 claime: Enabling puppet on A:cp-text - [[phab:T426323|T426323]] * 15:15 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1045.eqiad.wmnet * 15:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040', diff saved to https://phabricator.wikimedia.org/P92800 and previous config saved to /var/cache/conftool/dbconfig/20260521-151316-fceratto.json * 15:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1014.eqiad.wmnet * 15:11 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1045.eqiad.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2034.codfw.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2034.codfw.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1037.eqiad.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1037.eqiad.wmnet * 15:07 elukey@cumin1003: END (PASS) - Cookbook sre.misc-clusters.restart-reboot-config-master (exit_code=0) rolling reboot on A:config-master * 15:06 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1014.eqiad.wmnet * 15:05 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors * 15:05 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors * 15:04 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290805{{!}}hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] (duration: 10m 11s) * 15:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92799 and previous config saved to /var/cache/conftool/dbconfig/20260521-150308-fceratto.json * 15:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1037.eqiad.wmnet * 15:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2034.codfw.wmnet * 15:00 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors * 15:00 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors * 15:00 elukey@cumin1003: START - Cookbook sre.misc-clusters.restart-reboot-config-master rolling reboot on A:config-master * 15:00 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 15:00 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-lab1002.eqiad.wmnet * 14:59 elukey@cumin1003: END (PASS) - Cookbook sre.pki.restart-reboot (exit_code=0) rolling reboot on A:pki * 14:57 claime: Disabling puppet on A:cp-text - [[phab:T426323|T426323]] * 14:56 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1290805{{!}}hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:55 klausman@cumin1003: START - Cookbook sre.hosts.reboot-single for host ml-lab1002.eqiad.wmnet * 14:54 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-build1001.eqiad.wmnet * 14:54 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1290805{{!}}hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] * 14:54 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet * 14:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1013.eqiad.wmnet * 14:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1037.eqiad.wmnet * 14:53 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1028.eqiad.wmnet * 14:53 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>ml-serve1001.eqiad.wmnet<nowiki>}</nowiki> and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) * 14:53 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1001.eqiad.wmnet * 14:53 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1001.eqiad.wmnet * 14:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1028.eqiad.wmnet * 14:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92798 and previous config saved to /var/cache/conftool/dbconfig/20260521-145132-fceratto.json * 14:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2040.codfw.wmnet with reason: Maintenance * 14:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92797 and previous config saved to /var/cache/conftool/dbconfig/20260521-145103-fceratto.json * 14:50 klausman@cumin1003: START - Cookbook sre.hosts.reboot-single for host ml-build1001.eqiad.wmnet * 14:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: Migration of db2241.codfw.wmnet completed * 14:48 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1001.eqiad.wmnet * 14:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1013.eqiad.wmnet * 14:46 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1028.eqiad.wmnet * 14:45 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:44 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:42 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1001.eqiad.wmnet * 14:42 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>ml-serve1001.eqiad.wmnet<nowiki>}</nowiki> and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) * 14:42 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1028.eqiad.wmnet * 14:42 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-worker-eqiad * 14:42 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1011.eqiad.wmnet * 14:42 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1011.eqiad.wmnet * 14:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039', diff saved to https://phabricator.wikimedia.org/P92795 and previous config saved to /var/cache/conftool/dbconfig/20260521-144055-fceratto.json * 14:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1012.eqiad.wmnet * 14:38 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet. on all recursors * 14:37 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet. on all recursors * 14:37 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1011.eqiad.wmnet * 14:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1027.eqiad.wmnet * 14:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1027.eqiad.wmnet * 14:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1011.eqiad.wmnet * 14:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1012.eqiad.wmnet * 14:32 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1010.eqiad.wmnet * 14:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1010.eqiad.wmnet * 14:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039', diff saved to https://phabricator.wikimedia.org/P92793 and previous config saved to /var/cache/conftool/dbconfig/20260521-143045-fceratto.json * 14:30 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet. on all recursors * 14:30 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet. on all recursors * 14:29 elukey@cumin1003: START - Cookbook sre.pki.restart-reboot rolling reboot on A:pki * 14:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1027.eqiad.wmnet * 14:27 slyngshede@cumin1003: END (FAIL) - Cookbook sre.cdn.roll-reboot (exit_code=1) rolling reboot on P<nowiki>{</nowiki>cp601[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 14:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1027.eqiad.wmnet * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1054.eqiad.wmnet * 14:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1054.eqiad.wmnet * 14:24 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1010.eqiad.wmnet * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1011.eqiad.wmnet * 14:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92792 and previous config saved to /var/cache/conftool/dbconfig/20260521-142037-fceratto.json * 14:19 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1054.eqiad.wmnet * 14:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1054.eqiad.wmnet * 14:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1053.eqiad.wmnet * 14:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1053.eqiad.wmnet * 14:14 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1010.eqiad.wmnet * 14:14 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1009.eqiad.wmnet * 14:14 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1009.eqiad.wmnet * 14:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:13 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1011.eqiad.wmnet * 14:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:12 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2218: repool after maintenance * 14:11 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1053.eqiad.wmnet * 14:09 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92789 and previous config saved to /var/cache/conftool/dbconfig/20260521-140906-fceratto.json * 14:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2039.codfw.wmnet with reason: Maintenance * 14:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92788 and previous config saved to /var/cache/conftool/dbconfig/20260521-140837-fceratto.json * 14:08 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1009.eqiad.wmnet * 14:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:07 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1053.eqiad.wmnet * 14:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1035.eqiad.wmnet * 14:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1035.eqiad.wmnet * 14:04 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2241: Migration of db2241.codfw.wmnet completed * 14:03 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1009.eqiad.wmnet * 14:03 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1008.eqiad.wmnet * 14:03 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1008.eqiad.wmnet * 14:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2241.codfw.wmnet with OS trixie * 13:59 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 13:59 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1035.eqiad.wmnet * 13:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92786 and previous config saved to /var/cache/conftool/dbconfig/20260521-135830-fceratto.json * 13:58 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1008.eqiad.wmnet * 13:53 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1008.eqiad.wmnet * 13:53 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1007.eqiad.wmnet * 13:53 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1007.eqiad.wmnet * 13:51 Lucas_WMDE: UTC afternoon backport+config window done * 13:51 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290743{{!}}composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347{{!}}Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342{{!}}fix: simplify to show only one icon type for password reveal (T419413)]] (duration: 07m 20s) * 13:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92784 and previous config saved to /var/cache/conftool/dbconfig/20260521-134822-fceratto.json * 13:48 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1007.eqiad.wmnet * 13:47 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 13:46 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Continuing with deployment * 13:45 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 13:45 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Backport for [[gerrit:1290743{{!}}composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347{{!}}Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342{{!}}fix: simplify to show only one icon type for password reveal (T419413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes * 13:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2241.codfw.wmnet with reason: host reimage * 13:44 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 13:43 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1290743{{!}}composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347{{!}}Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342{{!}}fix: simplify to show only one icon type for password reveal (T419413)]] * 13:43 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 13:43 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1007.eqiad.wmnet * 13:42 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1006.eqiad.wmnet * 13:42 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1006.eqiad.wmnet * 13:41 dbrant@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290035{{!}}docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] (duration: 06m 52s) * 13:41 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 13:40 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2241.codfw.wmnet with reason: host reimage * 13:39 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1035.eqiad.wmnet * 13:38 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in codfw/ml-serve-codfw: maintenance * 13:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92782 and previous config saved to /var/cache/conftool/dbconfig/20260521-133815-fceratto.json * 13:37 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1006.eqiad.wmnet * 13:37 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in codfw/ml-serve-codfw: maintenance * 13:37 dbrant@deploy1003: dbrant: Continuing with deployment * 13:36 dbrant@deploy1003: dbrant: Backport for [[gerrit:1290035{{!}}docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1032.eqiad.wmnet * 13:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1032.eqiad.wmnet * 13:35 dbrant@deploy1003: Started scap sync-world: Backport for [[gerrit:1290035{{!}}docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] * 13:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1006.eqiad.wmnet * 13:32 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1005.eqiad.wmnet * 13:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1005.eqiad.wmnet * 13:31 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290014{{!}}Enable AG on phase 2 wikis (T426871)]] (duration: 09m 11s) * 13:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92781 and previous config saved to /var/cache/conftool/dbconfig/20260521-133116-fceratto.json * 13:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1048.eqiad.wmnet with reason: Maintenance * 13:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92780 and previous config saved to /var/cache/conftool/dbconfig/20260521-133048-fceratto.json * 13:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1032.eqiad.wmnet * 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1032.eqiad.wmnet * 13:27 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1005.eqiad.wmnet * 13:27 sbisson@deploy1003: sbisson: Continuing with deployment * 13:27 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2218: repool after maintenance * 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1031.eqiad.wmnet * 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1031.eqiad.wmnet * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:25 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2241.codfw.wmnet with OS trixie * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:24 sbisson@deploy1003: sbisson: Backport for [[gerrit:1290014{{!}}Enable AG on phase 2 wikis (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:23 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: Upgrading db2241.codfw.wmnet * 13:23 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2241: Upgrading db2241.codfw.wmnet * 13:23 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:22 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1290014{{!}}Enable AG on phase 2 wikis (T426871)]] * 13:22 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1005.eqiad.wmnet * 13:22 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1004.eqiad.wmnet * 13:22 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1004.eqiad.wmnet * 13:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040', diff saved to https://phabricator.wikimedia.org/P92778 and previous config saved to /var/cache/conftool/dbconfig/20260521-132041-fceratto.json * 13:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1031.eqiad.wmnet * 13:20 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290088{{!}}Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032{{!}}Enable 'flood' user group at en.wikiversity (T426882)]] (duration: 11m 55s) * 13:18 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet * 13:17 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1018.eqiad.wmnet with OS trixie * 13:16 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1031.eqiad.wmnet * 13:16 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1039: Repooling * 13:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1030.eqiad.wmnet * 13:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1030.eqiad.wmnet * 13:15 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Continuing with deployment * 13:15 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1004.eqiad.wmnet * 13:14 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet * 13:11 eevans@cumin1003: START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:restbase * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . * 13:10 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1004.eqiad.wmnet * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . * 13:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040', diff saved to https://phabricator.wikimedia.org/P92776 and previous config saved to /var/cache/conftool/dbconfig/20260521-131033-fceratto.json * 13:10 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1003.eqiad.wmnet * 13:10 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1003.eqiad.wmnet * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 13:10 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2241 [[phab:T426936|T426936]]', diff saved to https://phabricator.wikimedia.org/P92775 and previous config saved to /var/cache/conftool/dbconfig/20260521-131025-cwilliams.json * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'readability' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'logo-detection' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . * 13:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1030.eqiad.wmnet * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . * 13:10 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Backport for [[gerrit:1290088{{!}}Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032{{!}}Enable 'flood' user group at en.wikiversity (T426882)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'llm' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . * 13:08 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1290088{{!}}Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032{{!}}Enable 'flood' user group at en.wikiversity (T426882)]] * 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet * 13:06 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp601[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 13:06 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3074.esams.wmnet<nowiki>}</nowiki> and A:cp * 13:06 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3074.esams.wmnet * 13:06 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2162 to x3 primary [[phab:T426936|T426936]]', diff saved to https://phabricator.wikimedia.org/P92774 and previous config saved to /var/cache/conftool/dbconfig/20260521-130609-cwilliams.json * 13:04 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 13:04 cezmunsta: Starting x3 codfw failover from db2241 to db2162 - [[phab:T426936|T426936]] * 13:04 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1003.eqiad.wmnet * 13:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1030.eqiad.wmnet * 13:03 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet * 13:00 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 13:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92772 and previous config saved to /var/cache/conftool/dbconfig/20260521-130018-fceratto.json * 12:59 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1003.eqiad.wmnet * 12:59 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1018.eqiad.wmnet with reason: host reimage * 12:59 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1002.eqiad.wmnet * 12:59 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1002.eqiad.wmnet * 12:58 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:57 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:56 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2162 with weight 0 [[phab:T426936|T426936]]', diff saved to https://phabricator.wikimedia.org/P92771 and previous config saved to /var/cache/conftool/dbconfig/20260521-125645-cwilliams.json * 12:56 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Primary switchover x3 [[phab:T426936|T426936]] * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:55 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1029.eqiad.wmnet * 12:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1029.eqiad.wmnet * 12:54 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3074.esams.wmnet<nowiki>}</nowiki> and A:cp * 12:54 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1002.eqiad.wmnet * 12:54 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp600[7-8].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 12:54 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6008.drmrs.wmnet * 12:53 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:52 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1018.eqiad.wmnet with reason: host reimage * 12:51 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:49 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1002.eqiad.wmnet * 12:49 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-worker-eqiad * 12:48 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1029.eqiad.wmnet * 12:48 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3066.esams.wmnet<nowiki>}</nowiki> and A:cp * 12:48 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3066.esams.wmnet * 12:47 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:47 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92770 and previous config saved to /var/cache/conftool/dbconfig/20260521-124707-fceratto.json * 12:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1040.eqiad.wmnet with reason: Maintenance * 12:46 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es1039: Repooling * 12:46 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:45 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1029.eqiad.wmnet * 12:45 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:44 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:43 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:43 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290727{{!}}hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] (duration: 07m 54s) * 12:42 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92768 and previous config saved to /var/cache/conftool/dbconfig/20260521-124014-fceratto.json * 12:39 kharlan@deploy1003: kharlan: Continuing with deployment * 12:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1052.eqiad.wmnet * 12:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1052.eqiad.wmnet * 12:37 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1018.eqiad.wmnet with OS trixie * 12:37 kharlan@deploy1003: kharlan: Backport for [[gerrit:1290727{{!}}hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:36 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:36 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3066.esams.wmnet<nowiki>}</nowiki> and A:cp * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:35 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1290727{{!}}hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:34 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1017.eqiad.wmnet with OS trixie * 12:34 kart_: Updated cxserver to 2026-05-20-034002-production ([[phab:T388690|T388690]], [[phab:T404295|T404295]], [[phab:T391703|T391703]], [[phab:T426605|T426605]]) * 12:34 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet * 12:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1052.eqiad.wmnet * 12:30 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply * 12:30 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply * 12:30 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet * 12:29 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:29 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92767 and previous config saved to /var/cache/conftool/dbconfig/20260521-122905-fceratto.json * 12:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1039.eqiad.wmnet with reason: Maintenance * 12:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92766 and previous config saved to /var/cache/conftool/dbconfig/20260521-122839-fceratto.json * 12:27 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply * 12:27 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply * 12:26 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:23 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-staging-worker * 12:23 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2003.codfw.wmnet * 12:23 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2003.codfw.wmnet * 12:22 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1052.eqiad.wmnet * 12:21 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply * 12:21 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply * 12:21 moritzm: installing nginx security updates * 12:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1051.eqiad.wmnet * 12:20 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in codfw/ml-serve-codfw: maintenance * 12:19 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1017.eqiad.wmnet with reason: host reimage * 12:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1051.eqiad.wmnet * 12:19 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in codfw/ml-serve-codfw: maintenance * 12:19 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in codfw/ml-staging-codfw: maintenance * 12:19 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in codfw/ml-staging-codfw: maintenance * 12:19 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in codfw/ml-staging-codfw: maintenance * 12:18 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in codfw/ml-staging-codfw: maintenance * 12:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047', diff saved to https://phabricator.wikimedia.org/P92765 and previous config saved to /var/cache/conftool/dbconfig/20260521-121832-fceratto.json * 12:17 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2003.codfw.wmnet * 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet * 12:15 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1017.eqiad.wmnet with reason: host reimage * 12:14 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1051.eqiad.wmnet * 12:13 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6007.drmrs.wmnet * 12:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet * 12:10 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1051.eqiad.wmnet * 12:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047', diff saved to https://phabricator.wikimedia.org/P92764 and previous config saved to /var/cache/conftool/dbconfig/20260521-120824-fceratto.json * 12:07 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2003.codfw.wmnet * 12:07 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2002.codfw.wmnet * 12:07 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2002.codfw.wmnet * 12:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1050.eqiad.wmnet * 12:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1050.eqiad.wmnet * 12:02 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp600[7-8].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp601[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6014.drmrs.wmnet * 12:00 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1017.eqiad.wmnet with OS trixie * 12:00 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2002.codfw.wmnet * 11:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1002.wikimedia.org * 11:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92763 and previous config saved to /var/cache/conftool/dbconfig/20260521-115817-fceratto.json * 11:57 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1050.eqiad.wmnet * 11:53 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt1002.wikimedia.org * 11:51 taavi: disabling puppet on C:bird to roll out {{Gerrit|1289919}} * 11:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92762 and previous config saved to /var/cache/conftool/dbconfig/20260521-115112-fceratto.json * 11:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2047.codfw.wmnet with reason: Maintenance * 11:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1050.eqiad.wmnet * 11:50 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2002.codfw.wmnet * 11:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92761 and previous config saved to /var/cache/conftool/dbconfig/20260521-115043-fceratto.json * 11:50 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2001.codfw.wmnet * 11:50 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2001.codfw.wmnet * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1049.eqiad.wmnet * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt2002.wikimedia.org * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1049.eqiad.wmnet * 11:45 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2001.codfw.wmnet * 11:45 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker-exp1001.eqiad.wmnet * 11:44 kartik@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 11:44 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1049.eqiad.wmnet * 11:43 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt2002.wikimedia.org * 11:42 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet * 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet * 11:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037', diff saved to https://phabricator.wikimedia.org/P92760 and previous config saved to /var/cache/conftool/dbconfig/20260521-114036-fceratto.json * 11:39 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker-exp1001.eqiad.wmnet * 11:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker-exp2001.codfw.wmnet * 11:38 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host testreduce1002.eqiad.wmnet * 11:37 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1049.eqiad.wmnet * 11:36 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet * 11:36 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet * 11:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1038.eqiad.wmnet * 11:35 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2001.codfw.wmnet * 11:35 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-staging-worker * 11:35 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet * 11:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1038.eqiad.wmnet * 11:34 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host testreduce1002.eqiad.wmnet * 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker-exp2001.codfw.wmnet * 11:32 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet * 11:31 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet * 11:30 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt-staging2001.codfw.wmnet * 11:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037', diff saved to https://phabricator.wikimedia.org/P92759 and previous config saved to /var/cache/conftool/dbconfig/20260521-113028-fceratto.json * 11:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2014.codfw.wmnet * 11:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1038.eqiad.wmnet * 11:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt-staging2001.codfw.wmnet * 11:26 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet * 11:24 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1038.eqiad.wmnet * 11:24 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1034.eqiad.wmnet * 11:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1034.eqiad.wmnet * 11:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2014.codfw.wmnet * 11:20 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6013.drmrs.wmnet * 11:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92758 and previous config saved to /var/cache/conftool/dbconfig/20260521-112021-fceratto.json * 11:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1034.eqiad.wmnet * 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling reboot on A:ldap-replicas-eqiad * 11:13 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2013.codfw.wmnet * 11:11 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1034.eqiad.wmnet * 11:09 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp601[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92757 and previous config saved to /var/cache/conftool/dbconfig/20260521-110851-fceratto.json * 11:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2037.codfw.wmnet with reason: Maintenance * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92756 and previous config saved to /var/cache/conftool/dbconfig/20260521-110822-fceratto.json * 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1033.eqiad.wmnet * 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1033.eqiad.wmnet * 11:05 jmm@cumin2002: START - Cookbook sre.ldap.roll-restart-reboot-replica rolling reboot on A:ldap-replicas-eqiad * 11:05 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2013.codfw.wmnet * 11:04 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp600[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 11:04 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6006.drmrs.wmnet * 11:02 jmm@cumin2002: END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling reboot on A:ldap-replicas-codfw * 11:00 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1033.eqiad.wmnet * 10:59 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1016.eqiad.wmnet with reason: host reimage * 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036', diff saved to https://phabricator.wikimedia.org/P92753 and previous config saved to /var/cache/conftool/dbconfig/20260521-105815-fceratto.json * 10:57 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1033.eqiad.wmnet * 10:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1044.eqiad.wmnet * 10:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1044.eqiad.wmnet * 10:55 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1016.eqiad.wmnet with reason: host reimage * 10:54 jmm@cumin2002: START - Cookbook sre.ldap.roll-restart-reboot-replica rolling reboot on A:ldap-replicas-codfw * 10:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2012.codfw.wmnet * 10:51 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:51 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:51 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1044.eqiad.wmnet * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036', diff saved to https://phabricator.wikimedia.org/P92752 and previous config saved to /var/cache/conftool/dbconfig/20260521-104807-fceratto.json * 10:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2012.codfw.wmnet * 10:46 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1044.eqiad.wmnet * 10:44 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290709{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] (duration: 08m 02s) * 10:43 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:41 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:40 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet * 10:40 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:39 jiji@deploy1003: jiji: Continuing with deployment * 10:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92751 and previous config saved to /var/cache/conftool/dbconfig/20260521-103759-fceratto.json * 10:37 jiji@deploy1003: jiji: Backport for [[gerrit:1290709{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:36 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1290709{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] * 10:35 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet * 10:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1043.eqiad.wmnet * 10:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1043.eqiad.wmnet * 10:34 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:29 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 10:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1043.eqiad.wmnet * 10:27 dcausse: [[phab:T423993|T423993]]: reindexing all archive indices * 10:27 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . * 10:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92749 and previous config saved to /var/cache/conftool/dbconfig/20260521-102630-fceratto.json * 10:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance * 10:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1043.eqiad.wmnet * 10:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92748 and previous config saved to /var/cache/conftool/dbconfig/20260521-102601-fceratto.json * 10:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2011.codfw.wmnet * 10:24 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6005.drmrs.wmnet * 10:22 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1042.eqiad.wmnet * 10:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1042.eqiad.wmnet * 10:17 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2011.codfw.wmnet * 10:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1042.eqiad.wmnet * 10:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047', diff saved to https://phabricator.wikimedia.org/P92747 and previous config saved to /var/cache/conftool/dbconfig/20260521-101552-fceratto.json * 10:15 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:14 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . * 10:13 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1042.eqiad.wmnet * 10:13 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1041.eqiad.wmnet * 10:12 moritzm: installing postgresql security updates * 10:12 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp600[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 10:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1041.eqiad.wmnet * 10:10 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet * 10:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org * 10:09 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 10:08 fnegri@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb1013.eqiad.wmnet * 10:08 fnegri@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb1013.eqiad.wmnet * 10:07 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1013.eqiad.wmnet * 10:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1041.eqiad.wmnet * 10:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047', diff saved to https://phabricator.wikimedia.org/P92746 and previous config saved to /var/cache/conftool/dbconfig/20260521-100545-fceratto.json * 10:05 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet * 10:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1041.eqiad.wmnet * 10:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1040.eqiad.wmnet * 10:04 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet * 10:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1040.eqiad.wmnet * 10:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org * 10:01 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1013.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 10:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1040.eqiad.wmnet * 10:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org * 09:59 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet * 09:58 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-codfw * 09:58 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2005.codfw.wmnet * 09:58 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2005.codfw.wmnet * 09:56 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1040.eqiad.wmnet * 09:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1039.eqiad.wmnet * 09:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1039.eqiad.wmnet * 09:56 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 09:56 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:55 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ml-serve1013.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 09:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92745 and previous config saved to /var/cache/conftool/dbconfig/20260521-095536-fceratto.json * 09:54 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1384.eqiad.wmnet * 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org * 09:54 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:54 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:52 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2005.codfw.wmnet * 09:52 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2005.codfw.wmnet * 09:52 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply * 09:52 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2004.codfw.wmnet * 09:52 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2004.codfw.wmnet * 09:51 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply * 09:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1039.eqiad.wmnet * 09:49 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1384.eqiad.wmnet * 09:49 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 09:49 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1383.eqiad.wmnet * 09:48 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1039.eqiad.wmnet * 09:48 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1036.eqiad.wmnet * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92744 and previous config saved to /var/cache/conftool/dbconfig/20260521-094829-fceratto.json * 09:48 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1036.eqiad.wmnet * 09:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1047.eqiad.wmnet with reason: Maintenance * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92743 and previous config saved to /var/cache/conftool/dbconfig/20260521-094801-fceratto.json * 09:47 fnegri@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1013.eqiad.wmnet * 09:47 fnegri@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1013.eqiad.wmnet with reason: Rebooting clouddb1013 [[phab:T426563|T426563]] * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2004.codfw.wmnet * 09:45 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2004.codfw.wmnet * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet * 09:45 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet * 09:45 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet * 09:44 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1383.eqiad.wmnet * 09:44 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 09:44 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1382.eqiad.wmnet * 09:42 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2002.codfw.wmnet * 09:40 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1036.eqiad.wmnet * 09:39 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet * 09:38 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1382.eqiad.wmnet * 09:38 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1381.eqiad.wmnet * 09:38 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1036.eqiad.wmnet * 09:38 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2003.codfw.wmnet * 09:38 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2003.codfw.wmnet * 09:38 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2002.codfw.wmnet * 09:38 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2002.codfw.wmnet * 09:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037', diff saved to https://phabricator.wikimedia.org/P92742 and previous config saved to /var/cache/conftool/dbconfig/20260521-093754-fceratto.json * 09:37 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 09:37 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet * 09:37 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet * 09:37 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet * 09:37 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet * 09:36 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host build2002.codfw.wmnet * 09:36 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 09:35 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp601[1-2].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 09:35 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6012.drmrs.wmnet * 09:34 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet * 09:33 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet * 09:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1381.eqiad.wmnet * 09:33 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1380.eqiad.wmnet * 09:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1023.eqiad.wmnet * 09:31 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet * 09:31 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2002.codfw.wmnet * 09:31 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2002.codfw.wmnet * 09:31 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2001.codfw.wmnet * 09:31 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2001.codfw.wmnet * 09:30 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet * 09:30 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet * 09:30 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet * 09:30 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet * 09:29 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet * 09:29 jayme@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad * 09:29 jayme@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw * 09:29 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet * 09:28 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet * 09:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037', diff saved to https://phabricator.wikimedia.org/P92741 and previous config saved to /var/cache/conftool/dbconfig/20260521-092746-fceratto.json * 09:27 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1380.eqiad.wmnet * 09:27 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1379.eqiad.wmnet * 09:27 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet * 09:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1023.eqiad.wmnet * 09:25 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet * 09:24 jayme@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw * 09:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1056.eqiad.wmnet to cluster eqiad and group A * 09:23 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet * 09:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad * 09:22 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1379.eqiad.wmnet * 09:22 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1378.eqiad.wmnet * 09:21 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2001.codfw.wmnet * 09:21 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2001.codfw.wmnet * 09:21 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-codfw * 09:21 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1056.eqiad.wmnet to cluster eqiad and group A * 09:20 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 09:18 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-jumbo1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:18 moritzm: remove ganeti1023 foom eqiad Ganeti cluster [[phab:T424680|T424680]] * 09:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92740 and previous config saved to /var/cache/conftool/dbconfig/20260521-091738-fceratto.json * 09:16 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1378.eqiad.wmnet * 09:16 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1377.eqiad.wmnet * 09:12 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1377.eqiad.wmnet * 09:12 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1376.eqiad.wmnet * 09:07 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1036: Repooling * 09:07 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1376.eqiad.wmnet * 09:07 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1375.eqiad.wmnet * 09:06 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92738 and previous config saved to /var/cache/conftool/dbconfig/20260521-090609-fceratto.json * 09:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1037.eqiad.wmnet with reason: Maintenance * 09:02 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1375.eqiad.wmnet * 09:01 btullis@cumin1003: START - Cookbook sre.hosts.provision for host kafka-jumbo1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:55 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6011.drmrs.wmnet * 08:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1023.eqiad.wmnet * 08:47 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 08:47 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1256: Migration of db1256.eqiad.wmnet completed * 08:44 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp601[1-2].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 08:42 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp600[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 08:42 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6004.drmrs.wmnet * 08:37 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es1036: Repooling * 08:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92733 and previous config saved to /var/cache/conftool/dbconfig/20260521-082951-fceratto.json * 08:29 hashar@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.3 refs [[phab:T423912|T423912]] * 08:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92731 and previous config saved to /var/cache/conftool/dbconfig/20260521-081642-fceratto.json * 08:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1036.eqiad.wmnet with reason: Maintenance * 08:02 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1256: Migration of db1256.eqiad.wmnet completed * 08:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6003.drmrs.wmnet * 08:00 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1256.eqiad.wmnet with OS trixie * 07:52 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp600[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:51 marostegui@dns1004: END - running authdns-update * 07:50 marostegui@dns1004: START - running authdns-update * 07:48 marostegui: Failover m3-master [[phab:T426633|T426633]] * 07:47 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1023.eqiad.wmnet * 07:46 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp6010.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:46 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6010.drmrs.wmnet * 07:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster1005.eqiad.wmnet to plain * 07:44 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster1005.eqiad.wmnet to plain * 07:43 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1256.eqiad.wmnet with reason: host reimage * 07:42 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster1005.eqiad.wmnet to drbd * 07:38 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1256.eqiad.wmnet with reason: host reimage * 07:35 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp6010.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:35 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp6002.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:35 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6002.drmrs.wmnet * 07:27 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster1005.eqiad.wmnet to drbd * 07:24 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp6002.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:24 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1256.eqiad.wmnet with OS trixie * 07:22 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1256: Upgrading db1256.eqiad.wmnet * 07:21 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1256: Upgrading db1256.eqiad.wmnet * 07:21 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to plain * 07:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to plain * 07:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbproxy1025.eqiad.wmnet with reason: Rebooting * 07:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to drbd * 06:54 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to drbd * 06:53 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to plain * 06:52 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to plain * 06:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to drbd * 06:42 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists1004.wikimedia.org * 06:40 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org * 06:39 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1003.eqiad.wmnet * 06:34 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org * 06:34 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host lists1004.wikimedia.org * 06:33 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host vrts1003.eqiad.wmnet * 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to drbd * 06:23 arnaudb@cumin1003: END (FAIL) - Cookbook sre.gerrit.reboot-gerrit (exit_code=99) Rebooting Gerrit on gerrit2003 * 06:22 arnaudb@cumin1003: START - Cookbook sre.gerrit.reboot-gerrit Rebooting Gerrit on gerrit2003 * 06:15 marostegui@dns1004: END - running authdns-update * 06:14 marostegui: Failover m2-master [[phab:T426633|T426633]] * 06:13 marostegui@dns1004: START - running authdns-update * 05:39 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1012 from dbctl [[phab:T426930|T426930]]', diff saved to https://phabricator.wikimedia.org/P92728 and previous config saved to /var/cache/conftool/dbconfig/20260521-053858-marostegui.json * 05:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc2 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92727 and previous config saved to /var/cache/conftool/dbconfig/20260521-053000-marostegui.json * 05:29 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1022 to pc2 master [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92726 and previous config saved to /var/cache/conftool/dbconfig/20260521-052905-marostegui.json * 05:21 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc1012.eqiad.wmnet with reason: Cloning * 02:41 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on planet1003.eqiad.wmnet with reason: debug wip * 02:11 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 29s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:29 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1027.eqiad.wmnet * 01:22 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1027.eqiad.wmnet * 00:55 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> de239c9ozh3gywvuu3mo7vs43fgotma 2426645 2426644 2026-06-14T02:06:55Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 34s) 2426645 wikitext text/x-wiki == 2026-06-14 == * 02:06 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-06-13 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 35s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-06-12 == * 19:54 dwisehaupt@dns1004: END - running authdns-update * 19:52 dwisehaupt@dns1004: START - running authdns-update * 18:33 dwisehaupt@dns1006: END - running authdns-update * 18:32 dwisehaupt@dns1006: START - running authdns-update * 16:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:10 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:10 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:59 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 15:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:43 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1301371{{!}}Hotfix for T428620 (T428620)]] (duration: 11m 17s) * 14:36 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 14:35 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1301371{{!}}Hotfix for T428620 (T428620)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:31 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1301371{{!}}Hotfix for T428620 (T428620)]] * 14:29 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:28 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 13:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 12:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 12:22 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:22 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:22 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:22 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 12:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 12:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 12:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 12:04 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:04 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:04 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 12:02 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of prometheus5003.eqsin.wmnet to drbd * 12:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus5003.eqsin.wmnet to drbd * 11:40 moritzm: installing Linux 5.10.257 on Bullseye hosts * 11:36 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 11:35 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:24 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 11:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:56 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 10:56 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 10:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:49 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 10:49 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 10:40 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply * 10:37 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply * 10:36 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply * 10:35 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply * 10:35 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply * 10:35 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply * 10:12 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 10:12 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 10:08 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 09:59 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 09:58 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 09:57 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 06:13 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) * 06:11 jmm@cumin2002: START - Cookbook sre.puppet.disable-merges * 03:07 ryankemper: [[phab:T427951|T427951]] sorry, `[eqiad,codfw].mediawiki.page_html_content_change.rc0` (accidentally a word) * 03:06 ryankemper: [[phab:T427951|T427951]] Deleted all 20 unused dev/test topics on kafka-jumbo (verified empty first); 2 (`[eqiad,codfw]page_html_content_change.rc0`) were immediately auto-recreated empty by a still-running `dse-k8s` enrichment consumer; awaiting owner confirmation before final re-delete * 02:01 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 01m 13s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:00 bblack@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () == 2026-06-11 == * 22:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 22:26 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 22:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 22:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 22:05 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300906{{!}}Restore MediaViewer toggle in Special:Preferences (T428742)]] (duration: 30m 51s) * 21:58 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host releases2003.codfw.wmnet with OS trixie * 21:52 egardner@deploy1003: egardner: Continuing with deployment * 21:51 egardner@deploy1003: egardner: Backport for [[gerrit:1300906{{!}}Restore MediaViewer toggle in Special:Preferences (T428742)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:34 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1300906{{!}}Restore MediaViewer toggle in Special:Preferences (T428742)]] * 21:34 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet with reason: host reimage * 21:29 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300913{{!}}Avoid the escaping from nowiki processing (T398967)]] (duration: 09m 09s) * 21:28 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on releases2003.codfw.wmnet with reason: host reimage * 21:25 arlolra@deploy1003: arlolra: Continuing with deployment * 21:22 arlolra@deploy1003: arlolra: Backport for [[gerrit:1300913{{!}}Avoid the escaping from nowiki processing (T398967)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:20 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1300913{{!}}Avoid the escaping from nowiki processing (T398967)]] * 21:07 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300911{{!}}hCaptcha: Enable for badlogin for all small wikis (T426875)]], [[gerrit:1300905{{!}}RadioRangeBallot: Fix strict mode issue (T428947)]] (duration: 10m 43s) * 21:06 bblack@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text and not P<nowiki>{</nowiki>cp7008*<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 21:01 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 21:00 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300911{{!}}hCaptcha: Enable for badlogin for all small wikis (T426875)]], [[gerrit:1300905{{!}}RadioRangeBallot: Fix strict mode issue (T428947)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:56 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300911{{!}}hCaptcha: Enable for badlogin for all small wikis (T426875)]], [[gerrit:1300905{{!}}RadioRangeBallot: Fix strict mode issue (T428947)]] * 20:51 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300842{{!}}Donor Delight Badge: Unify on "Remove badge" language across treatments (T427313)]], [[gerrit:1300843{{!}}[A11y] Donor Badge: Remove Badge button disappears too quickly (T428646)]], [[gerrit:1300896{{!}}Donor Delight Badge, styles: Amending to final design review feedback (T427313)]] (duration: 34m 10s) * 20:39 jdrewniak@deploy1003: annet, jdrewniak: Continuing with deployment * 20:35 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host releases2003.codfw.wmnet with OS trixie * 20:34 jdrewniak@deploy1003: annet, jdrewniak: Backport for [[gerrit:1300842{{!}}Donor Delight Badge: Unify on "Remove badge" language across treatments (T427313)]], [[gerrit:1300843{{!}}[A11y] Donor Badge: Remove Badge button disappears too quickly (T428646)]], [[gerrit:1300896{{!}}Donor Delight Badge, styles: Amending to final design review feedback (T427313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug * 20:17 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1300842{{!}}Donor Delight Badge: Unify on "Remove badge" language across treatments (T427313)]], [[gerrit:1300843{{!}}[A11y] Donor Badge: Remove Badge button disappears too quickly (T428646)]], [[gerrit:1300896{{!}}Donor Delight Badge, styles: Amending to final design review feedback (T427313)]] * 19:12 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 18:12 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 18:12 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 17:52 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300865{{!}}UploadWizard.config.php: Fix cc-by-4.0-heirs msg issue (T428935 T405146)]] (duration: 08m 15s) * 17:48 reedy@deploy1003: reedy: Continuing with deployment * 17:46 reedy@deploy1003: reedy: Backport for [[gerrit:1300865{{!}}UploadWizard.config.php: Fix cc-by-4.0-heirs msg issue (T428935 T405146)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:44 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1300865{{!}}UploadWizard.config.php: Fix cc-by-4.0-heirs msg issue (T428935 T405146)]] * 17:26 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 17:25 blake@deploy1003: Scap cancelled without rolling back. * 17:25 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 17:24 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 17:24 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 17:24 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 17:24 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 17:23 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 17:23 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 17:23 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 17:23 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 17:23 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 17:23 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 17:20 blake@deploy1003: blake: apache config update ([[phab:T428772|T428772]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:20 blake@deploy1003: Started scap sync-world: apache config update ([[phab:T428772|T428772]]) * 17:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 17:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2212: Migration of db2212.codfw.wmnet completed * 17:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 17:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1235: Migration of db1235.eqiad.wmnet completed * 17:08 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 16:45 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:43 dzahn@dns1005: END - running authdns-update * 16:42 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:41 dzahn@dns1005: START - running authdns-update * 16:41 mutante: releases.wikimedia.org - switching backend from codfw to eqiad - releases1003 is now the source of rsync for uploaded releases files (use releases.discovery.wmnet to not have to think about it) - [[phab:T418299|T418299]] * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts rdb2007.codfw.wmnet * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts rdb1011.eqiad.wmnet * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) * 16:34 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb2009.codfw.wmnet * 16:34 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:34 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2009.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:33 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Migration of db2212.codfw.wmnet completed * 16:27 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2009.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:27 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1235: Migration of db1235.eqiad.wmnet completed * 16:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2212.codfw.wmnet with OS trixie * 16:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1235.eqiad.wmnet with OS trixie * 16:13 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:07 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:06 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 16:05 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 16:05 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 16:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 16:04 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2212.codfw.wmnet with reason: host reimage * 16:01 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 16:01 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:01 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 16:01 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 16:00 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 16:00 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 16:00 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 16:00 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2212.codfw.wmnet with reason: host reimage * 15:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1235.eqiad.wmnet with reason: host reimage * 15:58 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 15:58 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 15:57 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 15:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 15:57 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 15:57 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 15:56 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2009.codfw.wmnet * 15:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:55 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb1011.eqiad.wmnet * 15:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 15:55 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2007.codfw.wmnet * 15:54 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 15:54 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1235.eqiad.wmnet with reason: host reimage * 15:54 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 15:53 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 15:53 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 15:40 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 15:40 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2212.codfw.wmnet with OS trixie * 15:39 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 15:39 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1235.eqiad.wmnet with OS trixie * 15:36 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 15:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1235: Upgrading db1235.eqiad.wmnet * 15:35 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 15:35 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1235: Upgrading db1235.eqiad.wmnet * 15:35 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:32 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 15:32 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:31 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 15:30 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300822{{!}}T428849: temporarily disable noisy warnings in HandleParsoidSectionLinks (T428849 T417530)]] (duration: 11m 29s) * 15:27 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2212: Upgrading db2212.codfw.wmnet * 15:26 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2212: Upgrading db2212.codfw.wmnet * 15:26 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:26 cscott@deploy1003: cscott: Continuing with deployment * 15:26 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1235: Upgrading db1235.eqiad.wmnet * 15:25 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1235: Upgrading db1235.eqiad.wmnet * 15:25 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:21 cscott@deploy1003: cscott: Backport for [[gerrit:1300822{{!}}T428849: temporarily disable noisy warnings in HandleParsoidSectionLinks (T428849 T417530)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:19 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1300822{{!}}T428849: temporarily disable noisy warnings in HandleParsoidSectionLinks (T428849 T417530)]] * 15:18 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 15:17 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 15:13 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 15:13 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 15:13 moritzm: installing libdbi-perl security updates * 14:53 moritzm: installing Bind security updates (just client-side tools/libraries) * 14:51 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry * 14:48 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry * 14:43 moritzm: installing Poppler security updates * 14:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:33 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 14:32 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 14:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1234: Migration of db1234.eqiad.wmnet completed * 14:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin02 and group 01 * 14:24 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin02 and group 01 * 14:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:18 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet * 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet * 14:00 Lucas_WMDE: UTC afternoon backport+config window done * 13:58 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300733{{!}}stream: webrequest.page_view_stats.dev0 (T428725)]] (duration: 08m 12s) * 13:57 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp5024.* * 13:55 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: name=cp5024.* * 13:55 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp5020.* * 13:54 javiermonton@deploy1003: javiermonton: Continuing with deployment * 13:52 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1300733{{!}}stream: webrequest.page_view_stats.dev0 (T428725)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:51 slyngshede@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P<nowiki>{</nowiki>lvs5004*<nowiki>}</nowiki> and A:liberica * 13:50 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1300733{{!}}stream: webrequest.page_view_stats.dev0 (T428725)]] * 13:50 slyngshede@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P<nowiki>{</nowiki>lvs5004*<nowiki>}</nowiki> and A:liberica * 13:50 slyngs: reloading liberica config on lvs5004 * 13:50 moritzm: installing openssl security updates * 13:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:46 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 13:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm * 13:46 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1234: Migration of db1234.eqiad.wmnet completed * 13:46 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 13:45 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 13:45 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 13:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2202.codfw.wmnet with OS trixie * 13:43 alexsanford@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298890{{!}}Add 2FA enforcement demotion config for phase 3 groups (T423120)]] (duration: 07m 19s) * 13:39 alexsanford@deploy1003: alexsanford: Continuing with deployment * 13:38 alexsanford@deploy1003: alexsanford: Backport for [[gerrit:1298890{{!}}Add 2FA enforcement demotion config for phase 3 groups (T423120)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:36 alexsanford@deploy1003: Started scap sync-world: Backport for [[gerrit:1298890{{!}}Add 2FA enforcement demotion config for phase 3 groups (T423120)]] * 13:36 slyngshede@dns1004: END - running authdns-update * 13:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1234.eqiad.wmnet with OS trixie * 13:34 moritzm: installing dovecot security updates * 13:34 slyngshede@dns1004: START - running authdns-update * 13:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300787{{!}}hCaptcha: Enable for MobileFrontend on all group1 wikis (T425940)]] (duration: 06m 59s) * 13:29 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 13:29 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 13:29 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 13:29 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 13:28 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 13:28 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 13:28 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 13:27 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300787{{!}}hCaptcha: Enable for MobileFrontend on all group1 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:26 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2202.codfw.wmnet with reason: host reimage * 13:25 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300787{{!}}hCaptcha: Enable for MobileFrontend on all group1 wikis (T425940)]] * 13:25 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/Android_FAQ 'Wikimedia Apps/FAQ/Android' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:24 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/Android_FAQ 'Wikimedia Apps/FAQ/Android' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:22 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300736{{!}}fix: correct intake-url and payload type for NCS experiment events (T422295)]] (duration: 06m 51s) * 13:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage * 13:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1234.eqiad.wmnet with reason: host reimage * 13:18 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Continuing with deployment * 13:18 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2202.codfw.wmnet with reason: host reimage * 13:18 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Backport for [[gerrit:1300736{{!}}fix: correct intake-url and payload type for NCS experiment events (T422295)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:18 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 13:17 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 13:16 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1300736{{!}}fix: correct intake-url and payload type for NCS experiment events (T422295)]] * 13:15 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage * 13:14 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/Android_FAQ 'Wikimedia Apps/FAQ/Android' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:13 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:13 gkyziridis@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300731{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] (duration: 08m 47s) * 13:13 andrewbogott: sudo -i reprepro --noskipold --component thirdparty/openstack-trixie-flamingo-backports update trixie-wikimedia * 13:12 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1234.eqiad.wmnet with reason: host reimage * 13:12 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 13:12 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/iOS_FAQ 'Wikimedia Apps/FAQ/iOS' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:12 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 13:12 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 13:11 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 13:11 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 13:11 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 13:11 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 13:11 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 13:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 13:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 13:09 gkyziridis@deploy1003: gkyziridis: Continuing with deployment * 13:06 gkyziridis@deploy1003: gkyziridis: Backport for [[gerrit:1300731{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:06 claime: echo 'https://api.wikimedia.org/service/lw/specs/openapi.yaml' {{!}} mwscript-k8s --attach -- purgeList.php * 13:04 gkyziridis@deploy1003: Started scap sync-world: Backport for [[gerrit:1300731{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] * 13:02 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2202.codfw.wmnet with OS trixie * 13:00 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:57 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1234.eqiad.wmnet with OS trixie * 12:55 moritzm: installing Exim security updates on Bullseye * 12:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ganeti5006 * 12:47 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti5006 * 12:46 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti5006 * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti5006.eqsin.wmnet 9.0.132.10.in-addr.arpa 9.0.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 12:46 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache ganeti5006.eqsin.wmnet 9.0.132.10.in-addr.arpa 9.0.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5006 - jmm@cumin2002" * 12:46 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5006 - jmm@cumin2002" * 12:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1234: Upgrading db1234.eqiad.wmnet * 12:44 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1234: Upgrading db1234.eqiad.wmnet * 12:44 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2188: Migration of db2188.codfw.wmnet completed * 12:29 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "UX improvements - oblivian@cumin1003" * 12:29 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: UX improvements - oblivian@cumin1003 * 12:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1232: Migration of db1232.eqiad.wmnet completed * 12:28 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: UX improvements - oblivian@cumin1003 * 12:28 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "UX improvements - oblivian@cumin1003" * 12:27 jmm@cumin2002: START - Cookbook sre.dns.netbox * 12:26 jmm@cumin2002: START - Cookbook sre.hosts.move-vlan for host ganeti5006 * 12:26 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm * 12:21 moritzm: remove ganeti5006 from eqsin cluster for reimage [[phab:T428229|T428229]] * 12:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet * 12:10 moritzm: installing openjdk-21 security updates on Bookworm * 12:03 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300764{{!}}Remove GrowthExperiments extension from closed wikis (T428884)]] (duration: 06m 53s) * 11:59 urbanecm@deploy1003: urbanecm: Continuing with deployment * 11:58 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1300764{{!}}Remove GrowthExperiments extension from closed wikis (T428884)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:56 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1300764{{!}}Remove GrowthExperiments extension from closed wikis (T428884)]] * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb1012.eqiad.wmnet * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb2010.codfw.wmnet * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:48 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2010.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 11:46 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:46 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb2008.codfw.wmnet * 11:46 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:46 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2188: Migration of db2188.codfw.wmnet completed * 11:44 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 11:43 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:43 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2010.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 11:43 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1232: Migration of db1232.eqiad.wmnet completed * 11:38 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:37 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 11:37 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 11:36 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 11:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2188.codfw.wmnet with OS trixie * 11:35 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb1012.eqiad.wmnet * 11:34 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2008.codfw.wmnet * 11:34 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2010.codfw.wmnet * 11:33 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 11:32 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 11:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1232.eqiad.wmnet with OS trixie * 11:27 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2002.codfw.wmnet * 11:25 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300749{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300751{{!}}hCaptcha: Enable for DiscussionTools on all wikis (T426039)]] (duration: 08m 38s) * 11:21 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 11:19 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300749{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300751{{!}}hCaptcha: Enable for DiscussionTools on all wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2188.codfw.wmnet with reason: host reimage * 11:17 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300749{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300751{{!}}hCaptcha: Enable for DiscussionTools on all wikis (T426039)]] * 11:15 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2188.codfw.wmnet with reason: host reimage * 11:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1232.eqiad.wmnet with reason: host reimage * 11:13 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc2002.codfw.wmnet * 11:13 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet * 11:11 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 11:09 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2001.codfw.wmnet * 11:09 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1232.eqiad.wmnet with reason: host reimage * 11:08 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet * 11:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:04 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc2001.codfw.wmnet * 11:04 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host testreduce1002.eqiad.wmnet * 11:04 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:02 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1262.eqiad.wmnet with reason: crash * 11:00 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 11:00 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host testreduce1002.eqiad.wmnet * 10:59 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 10:59 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 10:58 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 10:55 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2188.codfw.wmnet with OS trixie * 10:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2188: Upgrading db2188.codfw.wmnet * 10:52 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2188: Upgrading db2188.codfw.wmnet * 10:52 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:52 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1232.eqiad.wmnet with OS trixie * 10:48 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1232: Upgrading db1232.eqiad.wmnet * 10:48 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1232: Upgrading db1232.eqiad.wmnet * 10:48 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:33 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:32 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:31 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300734{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300727{{!}}hCaptcha: Enable for DiscussionTools on group 1 wikis (T426039)]] (duration: 11m 01s) * 10:26 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:23 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:23 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:22 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300734{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300727{{!}}hCaptcha: Enable for DiscussionTools on group 1 wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:20 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300734{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300727{{!}}hCaptcha: Enable for DiscussionTools on group 1 wikis (T426039)]] * 10:18 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:18 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:10 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 10:10 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 10:09 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2045.codfw.wmnet with OS trixie * 10:09 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repool es2046', diff saved to https://phabricator.wikimedia.org/P94069 and previous config saved to /var/cache/conftool/dbconfig/20260611-100221-marostegui.json * 10:01 marostegui@cumin1003: dbctl commit (dc=all): 'Depool es2046', diff saved to https://phabricator.wikimedia.org/P94068 and previous config saved to /var/cache/conftool/dbconfig/20260611-100145-marostegui.json * 10:01 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:59 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300580{{!}}ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] (duration: 15m 41s) * 09:54 jiji@deploy1003: jiji: Continuing with deployment * 09:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2045.codfw.wmnet with reason: host reimage * 09:45 jiji@deploy1003: jiji: Backport for [[gerrit:1300580{{!}}ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:43 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1300580{{!}}ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] * 09:42 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2045.codfw.wmnet with reason: host reimage * 09:37 elukey: uploaded spicerack_12.8.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia * 09:26 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS trixie * 09:26 marostegui@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host es2045.codfw.wmnet with OS bookworm * 09:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2176: Migration of db2176.codfw.wmnet completed * 09:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1219: Migration of db1219.eqiad.wmnet completed * 09:11 claime: cumin -x 'A:swift-fe' "disable-puppet 'Disabling puppet for ratelimit deploy - cgoubert'" * 08:57 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS bookworm * 08:39 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2176: Migration of db2176.codfw.wmnet completed * 08:34 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94055) * 08:34 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1219: Migration of db1219.eqiad.wmnet completed * 08:33 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94053) * 08:30 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T428823|T428823]] (duration: 01m 18s) * 08:29 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T428823|T428823]] * 08:27 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2176.codfw.wmnet with OS trixie * 08:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Migration to 10.11.17 * 08:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 08:25 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 08:25 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool pc1021: Migration to 10.11.17 * 08:25 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94052) * 08:24 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): Testing upgrade for [[phab:T428823|T428823]] (duration: 01m 17s) * 08:23 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): Testing upgrade for [[phab:T428823|T428823]] * 08:22 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94051) * 08:22 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1219.eqiad.wmnet with OS trixie * 08:17 moritzm: installing PHP 8.2 security updates * 08:15 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 08:14 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 08:11 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 08:11 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 08:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2176.codfw.wmnet with reason: host reimage * 08:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1013.eqiad.wmnet with OS trixie * 08:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5004.eqsin.wmnet to cluster eqsin02 and group 01 * 08:06 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 08:06 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 08:05 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2021.codfw.wmnet,pc1021.eqiad.wmnet with reason: upgrade * 08:05 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1219.eqiad.wmnet with reason: host reimage * 08:05 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5004.eqsin.wmnet to cluster eqsin02 and group 01 * 08:05 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 08:05 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 08:04 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2176.codfw.wmnet with reason: host reimage * 08:04 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 08:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 08:03 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 08:03 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5004.eqsin.wmnet * 07:58 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1219.eqiad.wmnet with reason: host reimage * 07:56 marostegui: install mariadb 10.11.17 on pc1 [[phab:T427345|T427345]] * 07:54 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1013.eqiad.wmnet with reason: host reimage * 07:50 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1013.eqiad.wmnet with reason: host reimage * 07:49 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 07:49 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 07:49 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5004.eqsin.wmnet * 07:47 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 07:47 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 07:46 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2176.codfw.wmnet with OS trixie * 07:43 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1219.eqiad.wmnet with OS trixie * 07:43 moritzm: imported Jenkins 2.541.3 for thirdparty/ci (Bullseye) and thirdparty/jenkins (Bookworm, Trixie) * 07:42 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 07:35 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1013.eqiad.wmnet with OS trixie * 07:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2176: Upgrading db2176.codfw.wmnet * 07:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1219: Upgrading db1219.eqiad.wmnet * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2176: Upgrading db2176.codfw.wmnet * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1219: Upgrading db1219.eqiad.wmnet * 07:31 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:30 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 07:29 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1163: Repooling * 07:19 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 06:51 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS trixie * 06:50 marostegui@cumin1003: dbctl commit (dc=all): 'Repool es2042', diff saved to https://phabricator.wikimedia.org/P94044 and previous config saved to /var/cache/conftool/dbconfig/20260611-065049-marostegui.json * 06:50 marostegui@cumin1003: dbctl commit (dc=all): 'Depool es2042', diff saved to https://phabricator.wikimedia.org/P94043 and previous config saved to /var/cache/conftool/dbconfig/20260611-065027-marostegui.json * 06:44 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1163: Repooling * 06:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1163 [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94041 and previous config saved to /var/cache/conftool/dbconfig/20260611-064319-fceratto.json * 06:42 fceratto@dns1005: END - running authdns-update * 06:40 fceratto@dns1005: START - running authdns-update * 06:33 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:33 fceratto@cumin1003: MariaDB change: Setting sections s1 as read-write for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:33 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:33 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1184 to s1 primary and set section read-write [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94040 and previous config saved to /var/cache/conftool/dbconfig/20260611-063323-fceratto.json * 06:32 fceratto@cumin1003: dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94039 and previous config saved to /var/cache/conftool/dbconfig/20260611-063251-fceratto.json * 06:32 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:32 fceratto@cumin1003: Dbctl change: Setting sections s1 as read-write for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:32 fceratto@cumin1003: MariaDB change: Setting sections s1 as read-write for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:31 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:31 fceratto@cumin1003: dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94037 and previous config saved to /var/cache/conftool/dbconfig/20260611-063100-fceratto.json * 06:30 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:30 fceratto@cumin1003: MariaDB change: Setting sections s1 as read-only for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:30 fceratto@cumin1003: Dbctl change: Setting sections s1 as read-only for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:29 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:29 federico3: Starting s1 eqiad failover from db1163 to db1184 - [[phab:T426083|T426083]] * 06:22 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1184 with weight 0 [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94035 and previous config saved to /var/cache/conftool/dbconfig/20260611-062224-fceratto.json * 06:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 30 hosts with reason: Primary switchover s1 [[phab:T426083|T426083]] * 05:37 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 05:28 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 05:27 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 05:18 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 05:17 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS trixie * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2045: Upgrading es2045.codfw.wmnet * 05:16 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2045: Upgrading es2045.codfw.wmnet * 05:16 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 44s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:23 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2046.* * 01:19 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync * 01:18 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync * 01:18 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1009.eqiad.wmnet with OS trixie * 01:12 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:12 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 01:12 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:12 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:11 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:11 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 01:11 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:10 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:10 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 01:09 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 01:09 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 01:08 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 01:08 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 01:08 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 01:07 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 01:07 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 01:06 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 01:06 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 01:06 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 01:05 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 01:05 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 01:05 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 01:02 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1009.eqiad.wmnet with reason: host reimage * 00:58 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1009.eqiad.wmnet with reason: host reimage * 00:54 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main1009 * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main1009 * 00:41 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main1009 * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main1009.eqiad.wmnet 37.48.64.10.in-addr.arpa 7.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:41 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main1009.eqiad.wmnet 37.48.64.10.in-addr.arpa 7.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1009 - jasmine@cumin2002" * 00:40 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1009 - jasmine@cumin2002" * 00:39 cdanis@cumin1003: dbctl commit (dc=all): 'depool db1262', diff saved to https://phabricator.wikimedia.org/P94032 and previous config saved to /var/cache/conftool/dbconfig/20260611-003950-cdanis.json * 00:36 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 00:34 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5020.* * 00:30 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main1009 * 00:30 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1009.eqiad.wmnet with OS trixie * 00:03 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5024.* == 2026-06-10 == * 23:53 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5024.* * 23:15 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300154{{!}}Disable ShortUrl on bdwikimedia, bhwiki, bnwiki, bnwikisource, eswikibooks, gomwiki (T107188)]] (duration: 11m 37s) * 23:11 krinkle@deploy1003: krinkle: Continuing with deployment * 23:06 krinkle@deploy1003: krinkle: Backport for [[gerrit:1300154{{!}}Disable ShortUrl on bdwikimedia, bhwiki, bnwiki, bnwikisource, eswikibooks, gomwiki (T107188)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:04 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1300154{{!}}Disable ShortUrl on bdwikimedia, bhwiki, bnwiki, bnwikisource, eswikibooks, gomwiki (T107188)]] * 22:57 ladsgroup@dns1004: END - running authdns-update * 22:55 ladsgroup@dns1004: START - running authdns-update * 22:13 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5024.eqsin.wmnet with OS trixie * 22:13 mutante: gerrit - restarting service for logging change * 22:11 dzahn@cumin2002: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 0:10:00 on gerrit.wikimedia.org with reason: service restart * 22:09 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on gerrit2003.wikimedia.org with reason: service restart * 22:06 mutante: gerrit-spare: restarting gerrit * 22:06 mutante: gerrit-replica: restarting gerrit * 21:44 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage * 21:37 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage * 21:22 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300250{{!}}ExecuteTestAndCacheJob: Fix stdClasses serialised wrongly by JobQueue (T428801)]], [[gerrit:1300248{{!}}tests: Fix StandaloneHooksTest ordering, now broken by DB upgrade]] (duration: 08m 23s) * 21:17 jforrester@deploy1003: jforrester: Continuing with deployment * 21:15 jforrester@deploy1003: jforrester: Backport for [[gerrit:1300250{{!}}ExecuteTestAndCacheJob: Fix stdClasses serialised wrongly by JobQueue (T428801)]], [[gerrit:1300248{{!}}tests: Fix StandaloneHooksTest ordering, now broken by DB upgrade]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:13 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1300250{{!}}ExecuteTestAndCacheJob: Fix stdClasses serialised wrongly by JobQueue (T428801)]], [[gerrit:1300248{{!}}tests: Fix StandaloneHooksTest ordering, now broken by DB upgrade]] * 21:03 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5024 * 21:02 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5024 * 21:02 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300247{{!}}Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] (duration: 06m 51s) * 21:00 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp5024 * 21:00 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5024.eqsin.wmnet 35.0.132.10.in-addr.arpa 5.3.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 21:00 brett@cumin2002: START - Cookbook sre.dns.wipe-cache cp5024.eqsin.wmnet 35.0.132.10.in-addr.arpa 5.3.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 21:00 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:00 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5024 - brett@cumin2002" * 20:59 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5024 - brett@cumin2002" * 20:57 catrope@deploy1003: catrope: Continuing with deployment * 20:57 catrope@deploy1003: catrope: Backport for [[gerrit:1300247{{!}}Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:55 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1300247{{!}}Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] * 20:54 brett@cumin2002: START - Cookbook sre.dns.netbox * 20:50 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5024 * 20:49 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5024.eqsin.wmnet with OS trixie * 20:48 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5020.* * 20:44 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300073{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] (duration: 11m 55s) * 20:40 catrope@deploy1003: catrope, gkyziridis: Continuing with deployment * 20:34 catrope@deploy1003: catrope, gkyziridis: Backport for [[gerrit:1300073{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:32 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1300073{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] * 20:30 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5020.eqsin.wmnet with OS trixie * 20:30 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300226{{!}}[arzwiki] Change the wordmark (T427720)]] (duration: 09m 49s) * 20:25 catrope@deploy1003: gergesshamon, catrope: Continuing with deployment * 20:22 catrope@deploy1003: gergesshamon, catrope: Backport for [[gerrit:1300226{{!}}[arzwiki] Change the wordmark (T427720)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1300226{{!}}[arzwiki] Change the wordmark (T427720)]] * 19:59 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5020.eqsin.wmnet with reason: host reimage * 19:53 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5020.eqsin.wmnet with reason: host reimage * 19:30 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 19:27 bblack@cumin1003: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 19:23 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2046.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:19 brett@cumin2002: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2046.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5020 * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5020 * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2044.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:18 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp5020 * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5020.eqsin.wmnet 24.0.132.10.in-addr.arpa 4.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:18 brett@cumin2002: START - Cookbook sre.dns.wipe-cache cp5020.eqsin.wmnet 24.0.132.10.in-addr.arpa 4.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:17 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5020 - brett@cumin2002" * 19:17 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5020 - brett@cumin2002" * 19:14 brett@cumin2002: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2044.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:11 brett@cumin2002: START - Cookbook sre.dns.netbox * 19:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 19:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2174: Migration of db2174.codfw.wmnet completed * 19:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 19:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1218: Migration of db1218.eqiad.wmnet completed * 18:24 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5020 * 18:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5020.eqsin.wmnet with OS trixie * 18:22 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2174: Migration of db2174.codfw.wmnet completed * 18:20 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 18:17 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1218: Migration of db1218.eqiad.wmnet completed * 18:16 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5018.* * 18:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2174.codfw.wmnet with OS trixie * 18:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1218.eqiad.wmnet with OS trixie * 17:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2174.codfw.wmnet with reason: host reimage * 17:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1218.eqiad.wmnet with reason: host reimage * 17:46 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2010.codfw.wmnet with OS trixie * 17:45 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync * 17:44 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync * 17:44 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2174.codfw.wmnet with reason: host reimage * 17:42 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1218.eqiad.wmnet with reason: host reimage * 17:33 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94021) * 17:29 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2010.codfw.wmnet with reason: host reimage * 17:26 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1218.eqiad.wmnet with OS trixie * 17:26 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2174.codfw.wmnet with OS trixie * 17:25 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:24 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 17:24 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1218: Upgrading db1218.eqiad.wmnet * 17:24 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:24 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:24 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1218: Upgrading db1218.eqiad.wmnet * 17:23 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 17:23 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2174: Upgrading db2174.codfw.wmnet * 17:23 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 17:23 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2010.codfw.wmnet with reason: host reimage * 17:23 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:22 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2174: Upgrading db2174.codfw.wmnet * 17:22 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:22 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 17:22 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 17:22 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 17:22 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text and not P<nowiki>{</nowiki>cp7008*<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 17:21 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 17:21 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 17:19 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 17:19 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 17:18 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 17:18 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 17:17 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 17:17 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 17:17 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 17:13 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 17:12 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox) * 17:03 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 17:03 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1206: Migration of db1206.eqiad.wmnet completed * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2010 * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2010 * 17:02 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2010 * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2010.codfw.wmnet 35.48.192.10.in-addr.arpa 5.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:02 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2010.codfw.wmnet 35.48.192.10.in-addr.arpa 5.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2010 - jasmine@cumin2002" * 17:01 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2010 - jasmine@cumin2002" * 16:57 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 16:50 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2010 * 16:50 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2010.codfw.wmnet with OS trixie * 16:41 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 16:39 bblack@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 16:39 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 16:34 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 16:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5018.eqsin.wmnet with OS trixie * 16:22 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 16:20 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 16:17 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1206: Migration of db1206.eqiad.wmnet completed * 16:15 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 16:15 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 16:14 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 16:12 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 16:12 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 16:11 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 16:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1206.eqiad.wmnet with OS trixie * 16:01 blblack: apt: uploaded libvmod-wmfuniq 0.3.0 for trixie * 15:59 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5018.eqsin.wmnet with reason: host reimage * 15:53 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:52 vriley@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:51 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5018.eqsin.wmnet with reason: host reimage * 15:50 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1206.eqiad.wmnet with reason: host reimage * 15:45 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1206.eqiad.wmnet with reason: host reimage * 15:43 sukhe@cumin1003: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool drmrs [reason: no reason specified, no task ID specified] * 15:42 sukhe@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: no reason specified, no task ID specified] * 15:38 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 15:38 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2173: Migration of db2173.codfw.wmnet completed * 15:34 topranks: drain traffic through cr2-drmrs to reset pic0 * 15:33 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94013) * 15:30 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1206.eqiad.wmnet with OS trixie * 15:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1206: Upgrading db1206.eqiad.wmnet * 15:28 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1206: Upgrading db1206.eqiad.wmnet * 15:27 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:25 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:24 vriley@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:24 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1009 * 15:24 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Harroyo-wmf out of all services on: 2436 hosts * 15:23 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1009 * 15:21 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:20 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist translate extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release * 15:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5018 * 15:19 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5018 * 15:18 vriley@cumin1003: START - Cookbook sre.dns.netbox * 15:18 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp5018 * 15:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5018.eqsin.wmnet 18.0.132.10.in-addr.arpa 8.1.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 15:18 brett@cumin2002: START - Cookbook sre.dns.wipe-cache cp5018.eqsin.wmnet 18.0.132.10.in-addr.arpa 8.1.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 15:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:15 brett@cumin2002: START - Cookbook sre.dns.netbox * 15:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 15:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1195: Migration of db1195.eqiad.wmnet completed * 15:12 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:11 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:11 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin1003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:11 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin1003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:08 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300169{{!}}Fix snak value display for rtl languages (T360854)]], [[gerrit:1300168{{!}}Fix snak value display for rtl languages (T360854)]] (duration: 08m 39s) * 15:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 15:01 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1300169{{!}}Fix snak value display for rtl languages (T360854)]], [[gerrit:1300168{{!}}Fix snak value display for rtl languages (T360854)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:59 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:59 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1300169{{!}}Fix snak value display for rtl languages (T360854)]], [[gerrit:1300168{{!}}Fix snak value display for rtl languages (T360854)]] * 14:58 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:55 Lucas_WMDE: lucaswerkmeister-wmde@deploy1003 $ printf 'https://www.mediawiki.org/keys/%s\n' '' 'keys.txt' 'keys.html' {{!}} mwscript-k8s --attach --comment=[[phab:T423267|T423267]] purgeList mediawikiwiki * 14:54 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist translate extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release, now with correct schema * 14:53 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2173: Migration of db2173.codfw.wmnet completed * 14:50 ayounsi@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin2003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:50 ayounsi@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:49 ayounsi@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:48 ayounsi@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:47 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299614{{!}}Add my public key to mediawiki.org/keys (T423267)]] (duration: 08m 33s) * 14:46 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:42 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, matmarex: Continuing with deployment * 14:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2173.codfw.wmnet with OS trixie * 14:40 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, matmarex: Backport for [[gerrit:1299614{{!}}Add my public key to mediawiki.org/keys (T423267)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:40 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:40 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:38 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1299614{{!}}Add my public key to mediawiki.org/keys (T423267)]] * 14:38 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox and (A:dnsbox) * 14:34 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:34 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:33 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:29 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1195: Migration of db1195.eqiad.wmnet completed * 14:28 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:27 vriley@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:26 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 14:26 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 14:24 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist translate extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release, now with dblist translate * 14:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2173.codfw.wmnet with reason: host reimage * 14:23 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 14:22 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 14:22 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply * 14:21 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply * 14:20 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox) * 14:20 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2173.codfw.wmnet with reason: host reimage * 14:20 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:19 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:19 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:18 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:18 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:18 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:18 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1195.eqiad.wmnet with OS trixie * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 14:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 14:15 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 14:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 14:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 14:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 14:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:13 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:13 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 14:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 14:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 14:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 14:09 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 14:08 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:08 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 14:07 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 14:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 14:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 14:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 14:02 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2173.codfw.wmnet with OS trixie * 14:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 14:00 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1195.eqiad.wmnet with reason: host reimage * 14:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 13:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2173: Upgrading db2173.codfw.wmnet * 13:59 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2173: Upgrading db2173.codfw.wmnet * 13:58 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:58 atsuko@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/ttmserver-export.php --wiki=default --ttmserver eqiad-test # [[phab:T425377|T425377]] populating production index on test cluster to estimate time required for the release * 13:56 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1195.eqiad.wmnet with reason: host reimage * 13:54 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Atieno out of all services on: 2436 hosts * 13:42 Lucas_WMDE: UTC afternoon backport+config window done * 13:42 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1195.eqiad.wmnet with OS trixie * 13:36 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297237{{!}}wmf-config: Update private subnets to include additions (T427393)]] (duration: 07m 20s) * 13:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1195: Upgrading db1195.eqiad.wmnet * 13:33 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling restart_daemons on A:hcaptcha-proxy and A:hcaptcha-proxy * 13:33 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum and A:durum * 13:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2170: Migration of db2170.codfw.wmnet completed * 13:33 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1195: Upgrading db1195.eqiad.wmnet * 13:32 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:32 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, brett: Continuing with deployment * 13:32 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough * 13:31 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply * 13:31 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, brett: Backport for [[gerrit:1297237{{!}}wmf-config: Update private subnets to include additions (T427393)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:31 eevans@deploy1003: helmfile [staging] START helmfile.d/services/data-gateway: apply * 13:29 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1297237{{!}}wmf-config: Update private subnets to include additions (T427393)]] * 13:28 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp5018.eqsin.wmnet with reason: host down * 13:28 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy (exit_code=0) rolling restart_daemons on A:tcpproxy and A:tcpproxy * 13:25 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5018.eqsin.wmnet,service=(cdn{{!}}ats-be) * 13:22 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox) * 13:20 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum and A:durum * 13:20 sukhe@cumin1003: START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling restart_daemons on A:hcaptcha-proxy and A:hcaptcha-proxy * 13:19 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299676{{!}}Enable ULS v2 on group0 wikis]] (duration: 17m 00s) * 13:19 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough * 13:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1186: Migration of db1186.eqiad.wmnet completed * 13:18 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-test: apply * 13:18 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-test: apply * 13:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply * 13:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply * 13:15 sbisson@deploy1003: sbisson, abi: Continuing with deployment * 13:10 sukhe@cumin1003: START - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy rolling restart_daemons on A:tcpproxy and A:tcpproxy * 13:05 sbisson@deploy1003: sbisson, abi: Backport for [[gerrit:1299676{{!}}Enable ULS v2 on group0 wikis]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:03 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1014.eqiad.wmnet with OS trixie * 13:02 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299676{{!}}Enable ULS v2 on group0 wikis]] * 12:47 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2170: Migration of db2170.codfw.wmnet completed * 12:46 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5004.eqsin.wmnet with OS bookworm * 12:46 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:46 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:46 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:45 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1014.eqiad.wmnet with reason: host reimage * 12:42 topranks: re-map DSCP AF41 from 'low' to 'normal' priority qos class on network [[phab:T424640|T424640]] * 12:41 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1014.eqiad.wmnet with reason: host reimage * 12:36 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2170.codfw.wmnet with OS trixie * 12:33 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1186: Migration of db1186.eqiad.wmnet completed * 12:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5004.eqsin.wmnet with reason: host reimage * 12:24 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1014 * 12:24 jiji@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host rdb1014 * 12:23 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1186.eqiad.wmnet with OS trixie * 12:21 jiji@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host rdb1014 * 12:21 jiji@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb1014.eqiad.wmnet 42.48.64.10.in-addr.arpa 2.4.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 12:21 jiji@cumin1003: START - Cookbook sre.dns.wipe-cache rdb1014.eqiad.wmnet 42.48.64.10.in-addr.arpa 2.4.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 12:21 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:21 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host rdb1014 - jiji@cumin1003" * 12:21 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host rdb1014 - jiji@cumin1003" * 12:20 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5004.eqsin.wmnet with reason: host reimage * 12:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2170.codfw.wmnet with reason: host reimage * 12:16 jiji@cumin1003: START - Cookbook sre.dns.netbox * 12:13 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1014 * 12:12 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1014.eqiad.wmnet with OS trixie * 12:12 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2170.codfw.wmnet with reason: host reimage * 12:08 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300104{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643{{!}}wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] (duration: 11m 06s) * 12:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1186.eqiad.wmnet with reason: host reimage * 12:03 reedy@deploy1003: reedy: Continuing with deployment * 12:02 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1186.eqiad.wmnet with reason: host reimage * 11:59 reedy@deploy1003: reedy: Backport for [[gerrit:1300104{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643{{!}}wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes c * 11:57 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1300104{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643{{!}}wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] * 11:53 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2170.codfw.wmnet with OS trixie * 11:51 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ganeti5004 * 11:51 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti5004 * 11:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2170: Upgrading db2170.codfw.wmnet * 11:49 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2170: Upgrading db2170.codfw.wmnet * 11:49 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti5004 * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti5004.eqsin.wmnet 40.0.132.10.in-addr.arpa 0.4.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 11:49 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache ganeti5004.eqsin.wmnet 40.0.132.10.in-addr.arpa 0.4.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5004 - jmm@cumin2002" * 11:49 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5004 - jmm@cumin2002" * 11:49 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:48 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1186.eqiad.wmnet with OS trixie * 11:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1186: Upgrading db1186.eqiad.wmnet * 11:45 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1186: Upgrading db1186.eqiad.wmnet * 11:45 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:38 jmm@cumin2002: START - Cookbook sre.dns.netbox * 11:35 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 11:34 jmm@cumin2002: START - Cookbook sre.hosts.move-vlan for host ganeti5004 * 11:34 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 11:34 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5004.eqsin.wmnet with OS bookworm * 11:34 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 11:33 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1151: Security updates * 11:33 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 11:33 root@cumin1003: START - Cookbook sre.mysql.parsercache * 11:33 root@cumin1003: START - Cookbook sre.mysql.pool pool db1151: Security updates * 11:31 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply * 11:30 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply * 11:30 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply * 11:30 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:23 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 11:23 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:23 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 11:23 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 11:16 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 11:15 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 11:15 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 11:15 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:09 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1151: Security updates * 11:09 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 11:09 root@cumin1003: START - Cookbook sre.mysql.parsercache * 11:09 root@cumin1003: START - Cookbook sre.mysql.depool depool db1151: Security updates * 11:08 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300092{{!}}ProductionServices: re-add poolcounter2006 (T426736)]] (duration: 06m 55s) * 11:04 blake@deploy1003: blake: Continuing with deployment * 11:04 blake@deploy1003: blake: Backport for [[gerrit:1300092{{!}}ProductionServices: re-add poolcounter2006 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:03 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:01 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300092{{!}}ProductionServices: re-add poolcounter2006 (T426736)]] * 10:59 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet * 10:57 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 10:57 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 10:57 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 10:56 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 10:56 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply * 10:56 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply * 10:56 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet * 10:56 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300087{{!}}ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] (duration: 06m 42s) * 10:51 blake@deploy1003: blake: Continuing with deployment * 10:51 moritzm: remove ganeti5004 from eqsin cluster for reimage [[phab:T428229|T428229]] * 10:51 blake@deploy1003: blake: Backport for [[gerrit:1300087{{!}}ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:49 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300087{{!}}ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] * 10:47 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet * 10:47 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 10:46 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 10:46 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 10:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 10:43 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet * 10:43 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300082{{!}}ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] (duration: 07m 38s) * 10:41 moritzm: installing nginx security updates * 10:38 blake@deploy1003: blake: Continuing with deployment * 10:38 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1152: Security updates * 10:38 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:38 root@cumin1003: START - Cookbook sre.mysql.parsercache * 10:38 root@cumin1003: START - Cookbook sre.mysql.pool pool db1152: Security updates * 10:38 blake@deploy1003: blake: Backport for [[gerrit:1300082{{!}}ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:37 moritzm: failover Ganeti master in eqsin to ganeti5007 [[phab:T428229|T428229]] * 10:35 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300082{{!}}ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] * 10:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 10:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 10:33 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet * 10:29 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet * 10:29 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300072{{!}}ProductionServices: reboot poolcounter1007 (T426736)]] (duration: 07m 45s) * 10:27 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 10:27 jmm@cumin2002: DONE (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for sretest2009.codfw.wmnet: Renew puppet certificate - jmm@cumin2002 * 10:24 blake@deploy1003: blake: Continuing with deployment * 10:23 blake@deploy1003: blake: Backport for [[gerrit:1300072{{!}}ProductionServices: reboot poolcounter1007 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:22 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 10:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 10:21 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:21 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300072{{!}}ProductionServices: reboot poolcounter1007 (T426736)]] * 10:21 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:21 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:21 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:21 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:20 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet * 10:14 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1152: Security updates * 10:14 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:14 root@cumin1003: START - Cookbook sre.mysql.parsercache * 10:14 root@cumin1003: START - Cookbook sre.mysql.depool depool db1152: Security updates * 10:13 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet * 10:12 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300064{{!}}ProductionServices: reboot poolcounter1006.eqiad (T426736)]] (duration: 07m 46s) * 10:07 blake@deploy1003: blake: Continuing with deployment * 10:06 blake@deploy1003: blake: Backport for [[gerrit:1300064{{!}}ProductionServices: reboot poolcounter1006.eqiad (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:04 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300064{{!}}ProductionServices: reboot poolcounter1006.eqiad (T426736)]] * 09:57 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300058{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] (duration: 09m 32s) * 09:52 kharlan@deploy1003: kharlan: Continuing with deployment * 09:49 kharlan@deploy1003: kharlan: Backport for [[gerrit:1300058{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:47 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1300058{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] * 09:35 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox * 09:34 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 09:32 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 09:32 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 09:26 moritzm: upgrade routinator in eqiad to 0.15.2 [[phab:T428456|T428456]] * 09:23 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 09:23 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 09:22 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 09:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus5003.eqsin.wmnet to plain * 09:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus5003.eqsin.wmnet to plain * 09:15 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:04 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:54 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:20 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:09 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:09 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:07 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 08:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:04 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:01 fceratto@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host db1215.eqiad.wmnet with OS trixie * 07:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:53 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:48 javiermonton@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply * 07:48 javiermonton@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply * 07:44 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1215.eqiad.wmnet with reason: host reimage * 07:41 javiermonton@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply * 07:40 javiermonton@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply * 07:40 moritzm: installing openssl security updates * 07:39 fceratto@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1215.eqiad.wmnet with reason: host reimage * 07:38 javiermonton@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply * 07:37 javiermonton@deploy1003: helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply * 07:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:29 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299556{{!}}ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561{{!}}ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529{{!}}translate: adding separate read/write endpoints (T425377)]] (duration: 14m 03s) * 07:25 atsuko@deploy1003: atsuko: Continuing with deployment * 07:23 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db1215.eqiad.wmnet with OS trixie * 07:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1215.eqiad.wmnet with reason: Reimage * 07:21 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:20 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:20 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:17 atsuko@deploy1003: atsuko: Backport for [[gerrit:1299556{{!}}ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561{{!}}ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529{{!}}translate: adding separate read/write endpoints (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be veri * 07:16 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:15 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1299556{{!}}ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561{{!}}ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529{{!}}translate: adding separate read/write endpoints (T425377)]] * 07:14 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:12 atsukoito: backporting extensions/Translate to wmf/1.47.0-wmf.5 and applying the config * 07:12 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 06:45 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 05:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 05:43 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 05:42 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 05:41 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 47s) * 02:07 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1008.eqiad.wmnet with OS trixie * 02:03 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync * 02:02 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:52 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:51 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 01:51 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:50 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:50 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:49 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1008.eqiad.wmnet with reason: host reimage * 01:49 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 01:49 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:49 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:49 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 01:48 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 01:48 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 01:47 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 01:47 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 01:46 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 01:46 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 01:44 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 01:44 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 01:43 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 01:43 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1008.eqiad.wmnet with reason: host reimage * 01:25 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main1008 * 01:24 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main1008 * 01:24 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main1008 * 01:24 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main1008.eqiad.wmnet 45.32.64.10.in-addr.arpa 5.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 01:23 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main1008.eqiad.wmnet 45.32.64.10.in-addr.arpa 5.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 01:23 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 01:23 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1008 - jasmine@cumin2002" * 01:23 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1008 - jasmine@cumin2002" * 01:19 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 01:12 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main1008 * 01:11 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1008.eqiad.wmnet with OS trixie * 01:00 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2009.codfw.wmnet with OS trixie * 00:54 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync * 00:53 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync * 00:43 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2009.codfw.wmnet with reason: host reimage * 00:40 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:38 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2009.codfw.wmnet with reason: host reimage * 00:38 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 00:38 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:37 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:37 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 00:36 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 00:36 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 00:34 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 00:34 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 00:33 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 00:33 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 00:32 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 00:32 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 00:32 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2009 * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2009 * 00:15 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2009 * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2009.codfw.wmnet 33.48.192.10.in-addr.arpa 3.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:15 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2009.codfw.wmnet 33.48.192.10.in-addr.arpa 3.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2009 - jasmine@cumin2002" * 00:15 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2009 - jasmine@cumin2002" * 00:10 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 00:03 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2009 * 00:03 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2009.codfw.wmnet with OS trixie == 2026-06-09 == * 22:50 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299640{{!}}HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] (duration: 08m 59s) * 22:45 cscott@deploy1003: cscott: Continuing with deployment * 22:43 cscott@deploy1003: cscott: Backport for [[gerrit:1299640{{!}}HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:41 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299640{{!}}HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] * 22:15 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299639{{!}}[Bug] Donor Badge: Remove client prefs for control group (T428501)]] (duration: 20m 57s) * 22:11 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 22:07 mutante: gerrit - apache httpd log file location moved to /srv/gerrit/site_path/review_site/logs/ [[phab:T425667|T425667]] * 22:06 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on gerrit2003.wikimedia.org with reason: debug * 21:56 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1299639{{!}}[Bug] Donor Badge: Remove client prefs for control group (T428501)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:54 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299639{{!}}[Bug] Donor Badge: Remove client prefs for control group (T428501)]] * 21:52 ryankemper: [[phab:T428241|T428241]] removed retired wdqs2009 full-graph journal dump (446G x2, ~892G) from clouddumps100[1-2]:/srv/dumps/xmldatadumps/public/other/wdqs * 21:49 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299602{{!}}Revert "Create VectorComponentPageToolbar component" (T428649)]] (duration: 08m 16s) * 21:48 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 21:45 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 21:43 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1299602{{!}}Revert "Create VectorComponentPageToolbar component" (T428649)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:41 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299602{{!}}Revert "Create VectorComponentPageToolbar component" (T428649)]] * 21:34 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gerrit1003.wikimedia.org with reason: debug * 21:27 maryum: Deployed security fix for [[phab:T428324|T428324]] * 21:24 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 21:15 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart * 21:06 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart * 20:50 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs2002.codfw.wmnet with OS trixie * 20:50 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299588{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] (duration: 11m 13s) * 20:46 cscott@deploy1003: cscott: Continuing with deployment * 20:43 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs2002.codfw.wmnet with OS trixie * 20:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:42 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:41 cscott@deploy1003: cscott: Backport for [[gerrit:1299588{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:39 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299588{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] * 20:38 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:38 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:33 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:33 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:32 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299454{{!}}wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] (duration: 22m 08s) * 20:28 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:28 cscott@deploy1003: cscott, gkyziridis: Continuing with deployment * 20:24 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2004 * 20:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2004 * 20:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2003 * 20:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2003 * 20:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2002 * 20:13 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2002 * 20:13 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2001 * 20:13 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2001 * 20:12 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:12 cscott@deploy1003: cscott, gkyziridis: Backport for [[gerrit:1299454{{!}}wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:10 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299454{{!}}wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] * 20:09 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:04 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:59 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:54 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:53 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:48 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:47 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:47 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:46 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:46 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:45 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:45 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:28 ryankemper@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts wdqs1015.eqiad.wmnet * 19:28 ryankemper@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:28 ryankemper@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002" * 19:27 ryankemper@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002" * 19:20 ryankemper@cumin2002: START - Cookbook sre.dns.netbox * 19:15 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2008.codfw.wmnet with OS trixie * 19:15 ryankemper@cumin2002: START - Cookbook sre.hosts.decommission for hosts wdqs1015.eqiad.wmnet * 19:12 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync * 19:12 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync * 19:00 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:58 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 18:58 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2008.codfw.wmnet with reason: host reimage * 18:58 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 18:58 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 18:57 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 18:57 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 18:56 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 18:56 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 18:55 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 18:55 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 18:55 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 18:54 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 18:54 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:54 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 18:53 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 18:53 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 18:53 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 18:52 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:52 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2003 to codfw - jhancock@cumin2002" * 18:52 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2003 to codfw - jhancock@cumin2002" * 18:52 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 18:52 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 18:51 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2008.codfw.wmnet with reason: host reimage * 18:51 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 18:51 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 18:51 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 18:50 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 18:50 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 18:47 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:47 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:47 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:46 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:46 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:42 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:42 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:31 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 18:29 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2008.codfw.wmnet with OS trixie * 18:26 jasmine@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2008.codfw.wmnet with OS trixie * 17:48 mutante: https://releases.wikimedia.org {{!}} https://releases-jenkins.wikimedia.org - down for maintenance [[phab:T418299|T418299]] * 17:48 cmooney@dns2005: END - running authdns-update * 17:47 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet with reason: reimage * 17:47 cmooney@dns2005: START - running authdns-update * 17:46 sukhe: sudo cumin 'A:hcaptcha-proxy' 'run-puppet-agent': rolling out CR {{Gerrit|1299427}} [[phab:T428539|T428539]] * 17:43 jayme: kafka-main2008 is down due to hardware failure [[phab:T428654|T428654]] * 17:32 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf1002.eqiad.wmnet with OS trixie * 17:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf1002.eqiad.wmnet with reason: host reimage * 17:06 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf1002.eqiad.wmnet with reason: host reimage * 17:05 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2008 * 17:05 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2008 * 17:04 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 17:04 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2008 * 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2008.codfw.wmnet 4.32.192.10.in-addr.arpa 4.0.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:04 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 17:04 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2008.codfw.wmnet 4.32.192.10.in-addr.arpa 4.0.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2008 - jasmine@cumin2002" * 17:04 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5018 * 17:04 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2008 - jasmine@cumin2002" * 17:03 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5018.eqsin.wmnet with OS trixie * 16:58 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 16:58 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 16:57 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 16:57 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 16:57 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 16:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 16:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 16:50 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf1002.eqiad.wmnet with OS trixie * 16:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 16:47 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf1001.eqiad.wmnet with OS trixie * 16:47 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply * 16:47 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply * 16:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 16:41 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 16:41 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 16:35 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2008 * 16:34 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2008.codfw.wmnet with OS trixie * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:31 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply * 16:30 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply * 16:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf1001.eqiad.wmnet with reason: host reimage * 16:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:26 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf1001.eqiad.wmnet with reason: host reimage * 16:23 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply * 16:22 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: apply * 16:20 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:16 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:15 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:13 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:13 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:12 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf1001.eqiad.wmnet with OS trixie * 16:10 jiji@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. * 16:09 jiji@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. * 16:07 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf2002.codfw.wmnet with OS trixie * 16:02 jiji@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 16:02 jiji@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 16:00 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. * 15:59 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 15:59 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. * 15:59 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 15:59 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. * 15:59 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 15:58 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] DONE helmfile.d/services/termbox: apply * 15:58 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] START helmfile.d/services/termbox: apply * 15:57 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 15:57 jiji@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 15:57 lucaswerkmeister-wmde@deploy1003: helmfile [staging] DONE helmfile.d/services/termbox: apply * 15:56 lucaswerkmeister-wmde@deploy1003: helmfile [staging] START helmfile.d/services/termbox: apply * 15:54 jiji@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 15:53 jiji@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 15:51 jiji@deploy1003: Finished scap sync-world: redeploy {{Gerrit|1299468}} (duration: 07m 23s) * 15:49 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf2002.codfw.wmnet with reason: host reimage * 15:47 jiji@deploy1003: jiji: Continuing with deployment * 15:46 jiji@deploy1003: jiji: redeploy {{Gerrit|1299468}} synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:46 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf2002.codfw.wmnet with reason: host reimage * 15:45 jiji@deploy1003: Started scap sync-world: redeploy {{Gerrit|1299468}} * 15:43 brouberol@cumin1003: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-eqiad * 15:34 brennen@deploy1003: Finished deploy [phabricator/deployment@73e57ce]: deploy phab1004 for [[phab:T410849|T410849]] (followup for robots.txt) (duration: 00m 40s) * 15:33 brennen@deploy1003: Started deploy [phabricator/deployment@73e57ce]: deploy phab1004 for [[phab:T410849|T410849]] (followup for robots.txt) * 15:33 brennen@deploy1003: Finished deploy [phabricator/deployment@73e57ce]: deploy phab2002 for [[phab:T410849|T410849]] (followup for robots.txt) (duration: 00m 45s) * 15:32 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299468{{!}}ProductionServices.php: switch filebackend.php to rdb2015:6381 #2 (T418918 T291916)]] (duration: 07m 21s) * 15:32 brennen@deploy1003: Started deploy [phabricator/deployment@73e57ce]: deploy phab2002 for [[phab:T410849|T410849]] (followup for robots.txt) * 15:28 jiji@deploy1003: Rolling back deployment * 15:27 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf2002.codfw.wmnet with OS trixie * 15:27 jiji@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 15:26 jiji@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 15:25 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1299468{{!}}ProductionServices.php: switch filebackend.php to rdb2015:6381 #2 (T418918 T291916)]] * 15:22 urbanecm: Remove `migrateMentorStatusAwayToCommunityConfiguration` from updatelog on all wikis ([[phab:T409170|T409170]]; the script was only ever run as a dry-run) * 15:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 15:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 15:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf2001.codfw.wmnet with OS trixie * 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@d244a3e]: deploy phab1004 for [[phab:T410849|T410849]] (duration: 00m 42s) * 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@d244a3e]: deploy phab1004 for [[phab:T410849|T410849]] * 15:02 brennen@deploy1003: Finished deploy [phabricator/deployment@d244a3e]: deploy phab2002 for [[phab:T410849|T410849]] (duration: 00m 45s) * 15:01 brennen@deploy1003: Started deploy [phabricator/deployment@d244a3e]: deploy phab2002 for [[phab:T410849|T410849]] * 14:58 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf2001.codfw.wmnet with reason: host reimage * 14:52 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf2001.codfw.wmnet with reason: host reimage * 14:52 arnaudb@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phab[2002-2003].codfw.wmnet,phab[1004-1006].eqiad.wmnet with reason: [[phab:T410849|T410849]] * 14:47 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 14:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 14:40 moritzm: upgrade routinator in codfw to 0.15.2 [[phab:T428456|T428456]] * 14:35 brouberol@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad * 14:33 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf2001.codfw.wmnet with OS trixie * 14:26 brouberol@cumin1003: END (ERROR) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=97) rolling reboot on A:cephosd-eqiad * 14:26 brouberol@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad * 14:20 btullis@cumin1003: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-codfw * 14:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host parsoidtest1001.eqiad.wmnet * 14:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2153: Migration of db2153.codfw.wmnet completed * 14:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of rpki2003.codfw.wmnet to drbd * 14:14 moritzm: imported routinator 0.15.2-1bookworm to thirdparty/routinator for bookworm-wikimedia [[phab:T428456|T428456]] * 14:12 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:12 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1184: Migration of db1184.eqiad.wmnet completed * 14:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host parsoidtest1001.eqiad.wmnet * 14:07 Dreamy_Jazz: Afternoon UTC backport window done * 14:07 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 14:06 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299495{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502{{!}}SecurePollLogPager: Cast user IDs to ints before use (T428599)]] (duration: 06m 53s) * 14:06 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 14:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: rack depool * 14:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of rpki2003.codfw.wmnet to drbd * 14:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow2004.codfw.wmnet to drbd * 14:02 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:02 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1299495{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502{{!}}SecurePollLogPager: Cast user IDs to ints before use (T428599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:59 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1299495{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502{{!}}SecurePollLogPager: Cast user IDs to ints before use (T428599)]] * 13:58 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:58 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:56 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 13:56 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 13:55 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:55 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * {{safesubst:SAL entry|1=13:55 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298929{{!}}Simplify fragment processing (T423700)]], [[gerrit:1298926{{!}}Move ::getFragmentsToTransform() to Content<nowiki>{</nowiki>Text,DOM<nowiki>}</nowiki>TransformStage]], [[gerrit:1298927{{!}}OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925{{!}}Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497}} * 13:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:51 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow2004.codfw.wmnet to drbd * 13:50 cscott@deploy1003: cscott: Continuing with deployment * 13:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2045.codfw.wmnet to cluster codfw and group A * 13:48 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2045.codfw.wmnet to cluster codfw and group A * 13:48 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2027.codfw.wmnet to cluster codfw and group A * 13:47 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2027.codfw.wmnet to cluster codfw and group A * 13:46 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 13:45 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 13:44 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * {{safesubst:SAL entry|1=13:42 cscott@deploy1003: cscott: Backport for [[gerrit:1298929{{!}}Simplify fragment processing (T423700)]], [[gerrit:1298926{{!}}Move ::getFragmentsToTransform() to Content<nowiki>{</nowiki>Text,DOM<nowiki>}</nowiki>TransformStage]], [[gerrit:1298927{{!}}OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925{{!}}Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497{{!}}Store indicators}} * 13:41 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * {{safesubst:SAL entry|1=13:40 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1298929{{!}}Simplify fragment processing (T423700)]], [[gerrit:1298926{{!}}Move ::getFragmentsToTransform() to Content<nowiki>{</nowiki>Text,DOM<nowiki>}</nowiki>TransformStage]], [[gerrit:1298927{{!}}OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925{{!}}Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497{{!}}}} * 13:40 btullis@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-codfw * 13:39 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 13:37 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 13:35 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 13:33 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 13:32 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 13:32 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298834{{!}}config: Disable EmailConfirmationBanner on all wikis (T428291)]] (duration: 07m 01s) * 13:30 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2153: Migration of db2153.codfw.wmnet completed * 13:28 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 lucaswerkmeister-wmde@deploy1003: mmartorana, lucaswerkmeister-wmde: Continuing with deployment * 13:27 lucaswerkmeister-wmde@deploy1003: mmartorana, lucaswerkmeister-wmde: Backport for [[gerrit:1298834{{!}}config: Disable EmailConfirmationBanner on all wikis (T428291)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:26 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1184: Migration of db1184.eqiad.wmnet completed * 13:25 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298834{{!}}config: Disable EmailConfirmationBanner on all wikis (T428291)]] * 13:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 13:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 13:23 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:21 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 13:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2153.codfw.wmnet with OS trixie * 13:20 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2241: rack depool * 13:20 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1237: repool after maintenance db1237 * 13:19 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298654{{!}}Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] (duration: 09m 40s) * 13:17 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2006.codfw.wmnet * 13:17 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2006.codfw.wmnet * 13:16 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2251-2253].codfw.wmnet * 13:16 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2251-2253].codfw.wmnet * 13:16 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2005.codfw.wmnet * 13:16 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2005.codfw.wmnet * 13:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1184.eqiad.wmnet with OS trixie * 13:14 lucaswerkmeister-wmde@deploy1003: neriah, lucaswerkmeister-wmde: Continuing with deployment * 13:11 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 * 13:11 lucaswerkmeister-wmde@deploy1003: neriah, lucaswerkmeister-wmde: Backport for [[gerrit:1298654{{!}}Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:09 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298654{{!}}Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] * 13:04 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:04 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:04 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2153.codfw.wmnet with reason: host reimage * 13:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:03 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1015.eqiad.wmnet with OS trixie * 12:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage * 12:58 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2153.codfw.wmnet with reason: host reimage * 12:57 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1016.eqiad.wmnet with OS trixie * 12:57 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 * 12:56 XioNoX: lsw1-a4-codfw> request system reboot - [[phab:T427357|T427357]] * 12:55 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:53 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage * 12:50 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299477{{!}}hCaptcha: Roll out to all wikis for api account creation. (T426050)]] (duration: 07m 21s) * 12:46 kharlan@deploy1003: kharlan, dbrant: Continuing with deployment * 12:46 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 * 12:45 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1015.eqiad.wmnet with reason: host reimage * 12:45 kharlan@deploy1003: kharlan, dbrant: Backport for [[gerrit:1299477{{!}}hCaptcha: Roll out to all wikis for api account creation. (T426050)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:45 topranks: shut sub-interfaces for row A/B legacy vlans on cr1-codfw [[phab:T427357|T427357]] * 12:45 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 * 12:43 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1299477{{!}}hCaptcha: Roll out to all wikis for api account creation. (T426050)]] * 12:42 topranks: increase OSPF cost on ssw1-a1-codfw link to lsw1-a4-codfw to force traffic via alternate spine [[phab:T427357|T427357]] * 12:41 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299478{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]] (duration: 07m 02s) * 12:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1016.eqiad.wmnet with reason: host reimage * 12:40 moritzm: installing wireshark security updates * 12:40 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2153.codfw.wmnet with OS trixie * 12:38 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1184.eqiad.wmnet with OS trixie * 12:37 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 12:36 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1299478{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:34 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2153: Upgrading db2153.codfw.wmnet * 12:34 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1237: repool after maintenance db1237 * 12:34 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1299478{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]] * 12:34 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2153: Upgrading db2153.codfw.wmnet * 12:34 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1184: Upgrading db1184.eqiad.wmnet * 12:33 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1184: Upgrading db1184.eqiad.wmnet * 12:33 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:32 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1237.eqiad.wmnet with OS trixie * 12:32 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1015.eqiad.wmnet with reason: host reimage * 12:32 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1016.eqiad.wmnet with reason: host reimage * 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 12:27 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2005.codfw.wmnet * 12:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2046: repool after maintenance * 12:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2006.codfw.wmnet * 12:23 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298829{{!}}wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] (duration: 16m 04s) * 12:23 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2006.codfw.wmnet * 12:22 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2251-2253].codfw.wmnet * 12:22 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2005.codfw.wmnet * 12:20 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2251-2253].codfw.wmnet * 12:20 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 12:20 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: rack depool * 12:20 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 12:20 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2241: rack depool * 12:19 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1016 * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1016 * 12:19 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1015 * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1015 * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1016.eqiad.wmnet with OS trixie * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1015.eqiad.wmnet with OS trixie * 12:17 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 * 12:17 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 24 hosts with reason: Rack A4 depool * 12:16 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Continuing with deployment * 12:15 topranks: drain traffic on ssw1-a1-codfw - add gshut community in evpn underlay - [[phab:T427357|T427357]] * 12:14 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 * 12:13 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Backport for [[gerrit:1298829{{!}}wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:10 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1237.eqiad.wmnet with reason: host reimage * 12:07 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1298829{{!}}wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] * 12:05 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1237.eqiad.wmnet with reason: host reimage * 12:00 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Dmaza out of all services on: 2435 hosts * 11:51 atsuko@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 11:51 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie * 11:49 atsuko@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 11:48 atsuko@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 11:47 atsuko@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 11:45 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 11:44 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 11:43 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:43 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2046: repool after maintenance * 11:38 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 11:36 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:36 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2046.codfw.wmnet with OS trixie * 11:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2185.codfw.wmnet with reason: Reimage * 11:31 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging HMonroy out of all services on: 2435 hosts * 11:28 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging KSiebert out of all services on: 2435 hosts * 11:26 slyngs: CAS-SSO upgrade to version 7.3.7.2 * 11:26 slyngshede@dns1004: END - running authdns-update * 11:24 slyngshede@dns1004: START - running authdns-update * 11:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2046.codfw.wmnet with reason: host reimage * 11:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1043: repool after upgrade * 11:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2046.codfw.wmnet with reason: host reimage * 10:55 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2046.codfw.wmnet with OS trixie * 10:53 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2046: Upgrading es2046.codfw.wmnet * 10:53 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2046: Upgrading es2046.codfw.wmnet * 10:52 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply * 10:52 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:52 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply * 10:52 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply * 10:52 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:52 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply * 10:51 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:32 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1043: repool after upgrade * 10:31 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:28 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1160: Repooling * 10:26 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1043.eqiad.wmnet with OS trixie * 10:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:17 elukey: complete rollout of apache2 upgrades * 10:16 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:15 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:13 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:12 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:12 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:08 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1043.eqiad.wmnet with reason: host reimage * 10:04 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:04 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1043.eqiad.wmnet with reason: host reimage * 10:04 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:04 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:04 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:04 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:04 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:57 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1160: Repooling * 09:51 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 09:51 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 09:50 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 09:50 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 09:49 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1043.eqiad.wmnet with OS trixie * 09:48 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool es1043: Upgrading es1043.eqiad.wmnet * 09:48 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 09:47 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:45 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 09:41 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 09:36 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=5 --verbose --last-checked="20260603"` (after stopping previous scan run) * 09:34 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=5 --verbose` (after stopping previous scan run) * 09:27 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 09:26 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 09:17 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 09:17 fceratto@cumin1003: MariaDB change: Setting sections s5 as read-write * 09:17 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 09:14 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1043: Upgrading es1043.eqiad.wmnet * 09:14 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:12 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1042 to es4 eqiad primary [[phab:T428386|T428386]]', diff saved to https://phabricator.wikimedia.org/P93943 and previous config saved to /var/cache/conftool/dbconfig/20260609-091215-marostegui.json * 09:11 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1043 to es4 eqiad primary [[phab:T428386|T428386]]', diff saved to https://phabricator.wikimedia.org/P93942 and previous config saved to /var/cache/conftool/dbconfig/20260609-091147-marostegui.json * 09:03 jiji@cumin1003: conftool action : set/pooled=yes; selector: service=docker-registry,name=registry2005.codfw.wmnet * 08:59 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:59 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:57 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1237.eqiad.wmnet with OS trixie * 08:55 jiji@cumin1003: conftool action : set/pooled=no; selector: service=docker-registry,name=registry2005.codfw.wmnet * 08:55 jiji@cumin1003: conftool action : set/pooled=yes; selector: service=docker-registry,name=registry2004.codfw.wmnet * 08:50 jiji@cumin1003: conftool action : set/pooled=no; selector: service=docker-registry,name=registry2004.codfw.wmnet * 08:22 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=docker-registry,name=codfw * 08:22 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=docker-registry,name=eqiad * 08:08 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=docker-registry,name=eqiad * 08:08 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=docker-registry,name=codfw * 07:59 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:59 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix typoes - ayounsi@cumin1003" * 07:59 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix typoes - ayounsi@cumin1003" * 07:52 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 07:47 brouberol@dns1004: END - running authdns-update * 07:46 brouberol@dns1004: START - running authdns-update * 07:44 brouberol@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/kafka-ui: apply * 07:43 brouberol@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/kafka-ui: apply * 07:43 brouberol@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/kafka-ui: apply * 07:42 brouberol@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/kafka-ui: apply * 07:41 brouberol@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/kafka-ui: apply * 07:39 brouberol@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/kafka-ui: apply * 07:38 brouberol@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:37 brouberol@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 07:37 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie * 07:36 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.major-upgrade (exit_code=97) * 07:36 brouberol@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:36 brouberol@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:26 fceratto@dns1004: END - running authdns-update * 07:24 fceratto@dns1004: START - running authdns-update * 07:22 marostegui@dns1004: END - running authdns-update * 07:21 marostegui@dns1004: START - running authdns-update * 07:19 elukey@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:19 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Fix dse-k8s-wdqs2002 duplicate ipv6 address - elukey@cumin1003" * 07:19 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Fix dse-k8s-wdqs2002 duplicate ipv6 address - elukey@cumin1003" * 07:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1160.eqiad.wmnet with reason: Maintenance * 07:12 elukey@cumin1003: START - Cookbook sre.dns.netbox * 07:11 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1160: Repooling * 07:11 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1160: Repooling * 07:11 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1160: Repooling * 07:11 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1160: Repooling * 07:00 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:00 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1237.eqiad.wmnet with OS trixie * 06:24 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1160 [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93940 and previous config saved to /var/cache/conftool/dbconfig/20260609-062412-fceratto.json * 06:17 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 06:16 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 06:16 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 06:16 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 06:15 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 06:15 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 06:15 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 06:14 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 06:12 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1244 to s4 primary and set section read-write [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93939 and previous config saved to /var/cache/conftool/dbconfig/20260609-061222-fceratto.json * 06:11 fceratto@cumin1003: dbctl commit (dc=all): 'Set s4 eqiad as read-only for maintenance - [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93938 and previous config saved to /var/cache/conftool/dbconfig/20260609-061131-fceratto.json * 06:10 federico3: Starting s4 eqiad failover from db1160 to db1244 - [[phab:T426086|T426086]] * 06:01 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1244 with weight 0 [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93937 and previous config saved to /var/cache/conftool/dbconfig/20260609-060121-fceratto.json * 06:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 40 hosts with reason: Primary switchover s4 [[phab:T426086|T426086]] * 05:40 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie * 05:37 marostegui@dns1004: START - running authdns-update * 05:27 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1237: Upgrading db1237.eqiad.wmnet * 05:27 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1237: Upgrading db1237.eqiad.wmnet * 05:27 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:24 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1237 [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93935 and previous config saved to /var/cache/conftool/dbconfig/20260609-052420-marostegui.json * 05:23 marostegui@dns1004: START - running authdns-update * 05:23 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93934 and previous config saved to /var/cache/conftool/dbconfig/20260609-052311-marostegui.json * 05:22 marostegui@cumin1003: dbctl commit (dc=all): 'Set x1 eqiad as read-only for maintenance - [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93933 and previous config saved to /var/cache/conftool/dbconfig/20260609-052253-marostegui.json * 05:22 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T428158|T428158]] * 05:19 marostegui@cumin1003: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93932 and previous config saved to /var/cache/conftool/dbconfig/20260609-051859-marostegui.json * 05:18 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: Primary switchover x1 [[phab:T428158|T428158]] * 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.3 (duration: 02m 43s) * 03:40 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] (duration: 37m 16s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 38s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-06-08 == * 22:00 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298915{{!}}CommonSettings: Set $wgScoreSafeMode = false (T428484)]] (duration: 07m 42s) * 21:56 reedy@deploy1003: reedy: Continuing with deployment * 21:54 reedy@deploy1003: reedy: Backport for [[gerrit:1298915{{!}}CommonSettings: Set $wgScoreSafeMode = false (T428484)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:53 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1298915{{!}}CommonSettings: Set $wgScoreSafeMode = false (T428484)]] * 21:12 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298891{{!}}OOUIHTMLForm: Avoid treating form header as a clickable label (T428359)]] (duration: 08m 10s) * 21:07 mlitn@deploy1003: mlitn, neriah: Continuing with deployment * 21:05 mlitn@deploy1003: mlitn, neriah: Backport for [[gerrit:1298891{{!}}OOUIHTMLForm: Avoid treating form header as a clickable label (T428359)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:03 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1298891{{!}}OOUIHTMLForm: Avoid treating form header as a clickable label (T428359)]] * 20:43 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297162{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias]], [[gerrit:1298841{{!}}Squashed diff to master]] (duration: 07m 05s) * 20:39 mlitn@deploy1003: mlitn: Continuing with deployment * 20:38 mlitn@deploy1003: mlitn: Backport for [[gerrit:1297162{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias]], [[gerrit:1298841{{!}}Squashed diff to master]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:36 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1297162{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias]], [[gerrit:1298841{{!}}Squashed diff to master]] * 20:29 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298390{{!}}English Wikibooks: update FlaggedRevs configuration (T428329)]], [[gerrit:1298328{{!}}English Wikiversity: Add new user group "autopatrolled" (T428269)]] (duration: 08m 58s) * 20:25 mlitn@deploy1003: mlitn, vadymts1: Continuing with deployment * 20:22 mlitn@deploy1003: mlitn, vadymts1: Backport for [[gerrit:1298390{{!}}English Wikibooks: update FlaggedRevs configuration (T428329)]], [[gerrit:1298328{{!}}English Wikiversity: Add new user group "autopatrolled" (T428269)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1298390{{!}}English Wikibooks: update FlaggedRevs configuration (T428329)]], [[gerrit:1298328{{!}}English Wikiversity: Add new user group "autopatrolled" (T428269)]] * 20:03 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298879{{!}}SimpleCaptcha: Re-render captcha when edit form is redisplayed (T428437)]] (duration: 37m 43s) * 19:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:31 kharlan@deploy1003: kharlan: Continuing with deployment * 19:30 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:30 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:29 kharlan@deploy1003: kharlan: Backport for [[gerrit:1298879{{!}}SimpleCaptcha: Re-render captcha when edit form is redisplayed (T428437)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 19:28 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:27 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:25 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1298879{{!}}SimpleCaptcha: Re-render captcha when edit form is redisplayed (T428437)]] * 19:24 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab (duration: 01m 32s) * 19:23 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:22 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab * 19:20 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab (duration: 01m 40s) * 19:19 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab * 19:16 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:14 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:06 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:59 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2004 * 18:52 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2004 * 18:52 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2003 * 18:52 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2003 * 18:51 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:51 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2004 to codfw - jhancock@cumin2002" * 18:51 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2004 to codfw - jhancock@cumin2002" * 18:44 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:42 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:42 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2030 to codfw - jhancock@cumin2002" * 18:42 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2030 to codfw - jhancock@cumin2002" * 18:37 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:33 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2002 * 18:32 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2002 * 18:31 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:31 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2002 to codfw - jhancock@cumin2002" * 18:31 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2002 to codfw - jhancock@cumin2002" * 18:25 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:22 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2001 * 18:22 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2001 * 18:21 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:21 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: updating dse-k8s-wdqs2001 to codfw - jhancock@cumin2002" * 18:21 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: updating dse-k8s-wdqs2001 to codfw - jhancock@cumin2002" * 18:17 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:02 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T427286|T427286]] (duration: 00m 12s) * 18:02 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T427286|T427286]] * 17:37 jnuche@deploy1003: Installation of scap version "4.268.0" completed for 2 hosts * 17:35 jnuche@deploy1003: Installing scap version "4.268.0" for 2 host(s) * 17:21 claime: restarting varnish-frontend service on cp6012 * 17:21 claime: restarting varnish-frontend service on cp6011 * 17:21 claime: restarted varnish-frontend service on cp6009 * 17:13 taavi: bounce sirenbot to get it to re-join a channel * 17:05 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 17:05 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:58 urbanecm@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 16:57 urbanecm@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 16:55 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 16:53 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 16:53 urbanecm@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 16:52 urbanecm@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 16:30 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 16:29 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 16:29 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 16:28 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 16:28 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 16:28 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 16:28 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 16:27 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 16:27 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 16:26 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 16:26 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 16:25 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 16:18 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 16:17 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 16:17 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 16:16 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 16:16 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 16:16 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 16:16 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 16:15 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 16:14 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 16:14 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 16:14 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 16:14 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 16:13 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 16:13 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 16:13 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 16:12 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 16:12 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 16:10 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 16:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 16:10 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 16:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 16:10 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 16:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 16:09 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 16:08 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 16:08 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 16:07 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 16:06 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 15:57 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2042: repool after upgrade * 15:45 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db[2183-2184].codfw.wmnet * 15:45 jynus@cumin2002: START - Cookbook sre.hosts.remove-downtime for db[2183-2184].codfw.wmnet * 15:18 jynus: dbmaint on backup1-codfw@codfw ([[phab:T428467|T428467]]) * 15:12 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2042: repool after upgrade * 15:12 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 15:09 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 15:09 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 15:09 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 15:07 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2042.codfw.wmnet with OS trixie * 15:04 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 15:04 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 15:03 jynus@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db[2183-2184].codfw.wmnet with reason: Switchover db * 15:03 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 15:03 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 15:02 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 15:01 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply * 15:00 eevans@deploy1003: helmfile [staging] START helmfile.d/services/data-gateway: apply * 14:59 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:55 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:55 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:54 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:50 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:50 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:50 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:49 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:49 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2042.codfw.wmnet with reason: host reimage * 14:42 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2042.codfw.wmnet with reason: host reimage * 14:32 Lucas_WMDE: UTC afternoon backport+config window done * 14:32 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298709{{!}}Add translatable messages for WikiProject names (T427804)]], [[gerrit:1298710{{!}}Use translatable messages for WikiProject links (T427804)]], [[gerrit:1297644{{!}}WikiProject links - remove 'text' config (T427804)]] (duration: 31m 57s) * 14:27 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 14:26 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2042.codfw.wmnet with OS trixie * 14:26 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 14:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2042: Upgrading es2042.codfw.wmnet * 14:25 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2042: Upgrading es2042.codfw.wmnet * 14:25 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:24 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2043 to es4 codfw primary [[phab:T428386|T428386]]', diff saved to https://phabricator.wikimedia.org/P93926 and previous config saved to /var/cache/conftool/dbconfig/20260608-142423-marostegui.json * 14:23 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 14:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1041: repool after maintenance * 14:19 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Continuing with deployment * 14:18 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Backport for [[gerrit:1298709{{!}}Add translatable messages for WikiProject names (T427804)]], [[gerrit:1298710{{!}}Use translatable messages for WikiProject links (T427804)]], [[gerrit:1297644{{!}}WikiProject links - remove 'text' config (T427804)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:11 cgoubert@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=liftwing-openapi-server.* * 14:10 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp6013.* * 14:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:05 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 14:05 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 13:54 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 13:52 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 13:50 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 13:50 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 13:50 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296550{{!}}hCaptcha: Don't show AbuseFilter CAPTCHA for wbsetclaim API (T427608)]] (duration: 08m 31s) * 13:48 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 13:46 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 13:43 cgoubert@dns1004: END - running authdns-update * 13:43 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296550{{!}}hCaptcha: Don't show AbuseFilter CAPTCHA for wbsetclaim API (T427608)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:41 cgoubert@dns1004: START - running authdns-update * 13:41 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296550{{!}}hCaptcha: Don't show AbuseFilter CAPTCHA for wbsetclaim API (T427608)]] * 13:39 urbanecm@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * {{safesubst:SAL entry|1=13:38 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298758{{!}}feat(V2): toggle experiment features based on custom url override (T424646)]], [[gerrit:1298762{{!}}specialCreateAccount: use GECreateAccountExperimentV2 instead of hook (T424646)]], [[gerrit:1298764{{!}}fix: correctly read experiments param on Special:UserLogin]], [[gerrit:1298765{{!}}signup.js: use JS var instead of TestKitchen to show exp}} * 13:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1041: repool after maintenance * 13:38 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 13:38 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:37 urbanecm@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 13:36 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 13:35 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1041.eqiad.wmnet with OS trixie * 13:34 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 13:34 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2041: repool after upgrade * 13:34 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Continuing with deployment * 13:34 urbanecm@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 13:32 urbanecm@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * {{safesubst:SAL entry|1=13:30 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Backport for [[gerrit:1298758{{!}}feat(V2): toggle experiment features based on custom url override (T424646)]], [[gerrit:1298762{{!}}specialCreateAccount: use GECreateAccountExperimentV2 instead of hook (T424646)]], [[gerrit:1298764{{!}}fix: correctly read experiments param on Special:UserLogin]], [[gerrit:1298765{{!}}signup.js: use JS var instead of TestKitchen to show}} * {{safesubst:SAL entry|1=13:29 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298758{{!}}feat(V2): toggle experiment features based on custom url override (T424646)]], [[gerrit:1298762{{!}}specialCreateAccount: use GECreateAccountExperimentV2 instead of hook (T424646)]], [[gerrit:1298764{{!}}fix: correctly read experiments param on Special:UserLogin]], [[gerrit:1298765{{!}}signup.js: use JS var instead of TestKitchen to show expe}} * 13:21 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298418{{!}}NewUserMessage: Add $wgNewUserMessageOnAutoCreateFirstEdit (T426206)]], [[gerrit:1298717{{!}}Replace NewUserMessageOnAutoCreateFirstEdit with wgNewUserMessageOnFirstEdit (T426206)]], [[gerrit:1298734{{!}}Enable wgNewUserMessageOnFirstEdit on incubatorwiki (T426206)]] (duration: 11m 06s) * 13:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1041.eqiad.wmnet with reason: host reimage * 13:17 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Continuing with deployment * 13:12 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 13:12 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Backport for [[gerrit:1298418{{!}}NewUserMessage: Add $wgNewUserMessageOnAutoCreateFirstEdit (T426206)]], [[gerrit:1298717{{!}}Replace NewUserMessageOnAutoCreateFirstEdit with wgNewUserMessageOnFirstEdit (T426206)]], [[gerrit:1298734{{!}}Enable wgNewUserMessageOnFirstEdit on incubatorwiki (T426206)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki * 13:12 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 13:12 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1041.eqiad.wmnet with reason: host reimage * 13:11 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 13:11 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 13:10 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298418{{!}}NewUserMessage: Add $wgNewUserMessageOnAutoCreateFirstEdit (T426206)]], [[gerrit:1298717{{!}}Replace NewUserMessageOnAutoCreateFirstEdit with wgNewUserMessageOnFirstEdit (T426206)]], [[gerrit:1298734{{!}}Enable wgNewUserMessageOnFirstEdit on incubatorwiki (T426206)]] * 12:57 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298767{{!}}Follow-up: Allow CaptchaConsequence to be skipped via hook (T427608)]] (duration: 06m 20s) * 12:57 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1041.eqiad.wmnet with OS trixie * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:56 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1041: Upgrading es1041.eqiad.wmnet * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:55 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1041: Upgrading es1041.eqiad.wmnet * 12:55 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:54 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:53 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 12:53 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1298767{{!}}Follow-up: Allow CaptchaConsequence to be skipped via hook (T427608)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:51 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:51 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1298767{{!}}Follow-up: Allow CaptchaConsequence to be skipped via hook (T427608)]] * 12:49 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:49 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2041: repool after upgrade * 12:49 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:47 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:46 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 12:44 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 12:43 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 12:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 12:41 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 12:40 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2063.codfw.wmnet with OS bullseye * 12:32 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2062.codfw.wmnet with OS bullseye * 12:27 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2041.codfw.wmnet with OS trixie * 12:21 joal@deploy1003: Finished deploy [analytics/refinery@d67c584] (thin): Regular analytics weekly train THIN [analytics/refinery@d67c584f] (duration: 02m 00s) * 12:19 joal@deploy1003: Started deploy [analytics/refinery@d67c584] (thin): Regular analytics weekly train THIN [analytics/refinery@d67c584f] * 12:19 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2063.codfw.wmnet with reason: host reimage * 12:18 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 12:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 12:16 joal@deploy1003: Finished deploy [analytics/refinery@d67c584]: Regular analytics weekly train [analytics/refinery@d67c584f] (duration: 07m 52s) * 12:15 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2063.codfw.wmnet with reason: host reimage * 12:13 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2062.codfw.wmnet with reason: host reimage * 12:09 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2041.codfw.wmnet with reason: host reimage * 12:08 joal@deploy1003: Started deploy [analytics/refinery@d67c584]: Regular analytics weekly train [analytics/refinery@d67c584f] * 12:08 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2062.codfw.wmnet with reason: host reimage * 12:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add eqiad e8 public vlans - ayounsi@cumin1003" * 12:06 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add eqiad e8 public vlans - ayounsi@cumin1003" * 12:03 joal@deploy1003: Finished deploy [analytics/refinery@d67c584] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@d67c584f] (duration: 02m 00s) * 12:03 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2041.codfw.wmnet with reason: host reimage * 12:01 joal@deploy1003: Started deploy [analytics/refinery@d67c584] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@d67c584f] * 12:01 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 12:00 ayounsi@cumin1003: END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) * 12:00 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 12:00 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 12:00 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:57 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2063 * 11:57 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2063 * 11:57 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2063 * 11:57 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2063.codfw.wmnet 52.16.192.10.in-addr.arpa 2.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:56 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2063.codfw.wmnet 52.16.192.10.in-addr.arpa 2.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:56 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:56 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2063 - mvernon@cumin2002" * 11:56 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2063 - mvernon@cumin2002" * 11:51 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:51 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2063 * 11:50 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2063.codfw.wmnet with OS bullseye * 11:50 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2062 * 11:50 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2062 * 11:49 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2062 * 11:49 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2062.codfw.wmnet 123.0.192.10.in-addr.arpa 3.2.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:49 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2062.codfw.wmnet 123.0.192.10.in-addr.arpa 3.2.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:49 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:49 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2062 - mvernon@cumin2002" * 11:49 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2062 - mvernon@cumin2002" * 11:47 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2041.codfw.wmnet with OS trixie * 11:45 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2041: Upgrading es2041.codfw.wmnet * 11:45 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2041: Upgrading es2041.codfw.wmnet * 11:44 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:44 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.major-upgrade (exit_code=97) * 11:44 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:44 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1042: repool after maintenance * 11:43 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:43 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2062 * 11:42 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2062.codfw.wmnet with OS bullseye * 11:30 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298728{{!}}SpecialMediaSearch: Prefer thumb steps over thumb limits (T424032)]] (duration: 17m 39s) * 11:25 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 11:18 Raine: progressively switching shellbox to bookworm (start) * 11:15 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 11:14 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 11:14 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1298728{{!}}SpecialMediaSearch: Prefer thumb steps over thumb limits (T424032)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:13 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 11:12 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 11:12 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1298728{{!}}SpecialMediaSearch: Prefer thumb steps over thumb limits (T424032)]] * 11:02 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be2062 * 11:02 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be2063 * 10:58 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1042: repool after maintenance * 10:58 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:56 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1042.eqiad.wmnet with OS trixie * 10:47 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298721{{!}}GuessedThumbnailInfo: Also allow showing webp originals (T428202)]] (duration: 16m 41s) * 10:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1042.eqiad.wmnet with reason: host reimage * 10:39 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 10:39 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 10:38 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 10:36 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2160.codfw.wmnet * 10:36 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2160.codfw.wmnet * 10:35 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2043: repool after upgrade * 10:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2160.codfw.wmnet with reason: Reboot * 10:34 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1298721{{!}}GuessedThumbnailInfo: Also allow showing webp originals (T428202)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:34 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1042.eqiad.wmnet with reason: host reimage * 10:30 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1298721{{!}}GuessedThumbnailInfo: Also allow showing webp originals (T428202)]] * 10:18 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1042.eqiad.wmnet with OS trixie * 10:18 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:18 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:18 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:18 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:15 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1042: Upgrading es1042.eqiad.wmnet * 10:14 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:14 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:14 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:14 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:13 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1042: Upgrading es1042.eqiad.wmnet * 10:13 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:12 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be2063 * 10:09 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be2062 * 10:07 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:07 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:07 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:06 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 09:52 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply * 09:52 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply * 09:50 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply * 09:49 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply * 09:49 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2043: repool after upgrade * 09:49 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2043.codfw.wmnet with OS trixie * 09:44 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 09:44 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 09:42 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: sync * 09:42 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: sync * 09:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2043.codfw.wmnet with reason: host reimage * 09:27 jelto@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org * 09:23 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2043.codfw.wmnet with reason: host reimage * 09:17 jelto@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org * 09:15 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: sync * 09:15 ozge@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: sync * 09:07 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2043.codfw.wmnet with OS trixie * 09:06 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2043: Upgrading es2043.codfw.wmnet * 09:06 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2043: Upgrading es2043.codfw.wmnet * 09:05 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:41 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1217.eqiad.wmnet with OS trixie * 08:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1217.eqiad.wmnet with reason: host reimage * 08:15 taavi@cumin1003: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database urwikisource ([[phab:T415977|T415977]]) * 08:14 taavi@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database urwikisource ([[phab:T415977|T415977]]) * 08:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1217.eqiad.wmnet with reason: host reimage * 08:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2052: repool after upgrade * 08:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1051: repool after maintenance * 08:03 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Managing sanitization for wikis urwikisource in section s5 * 07:55 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1217.eqiad.wmnet with OS trixie * 07:53 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1217.eqiad.wmnet with reason: reimage * 07:53 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis urwikisource in section s5 * 07:52 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Checking sanitization for wikis urwikisource in section s5 * 07:50 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis urwikisource in section s5 * 07:50 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.sanitize-wiki (exit_code=97) Managing sanitization for wikis urwikisource in section s5 * 07:50 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis urwikisource in section s5 * 07:44 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297681{{!}}Global rollout - Sub-ref deployments to Group 0, Group 1 and frwiki (T425662)]] (duration: 32m 51s) * 07:32 wmde-fisch@deploy1003: wmde-fisch, lilients: Continuing with deployment * 07:29 wmde-fisch@deploy1003: wmde-fisch, lilients: Backport for [[gerrit:1297681{{!}}Global rollout - Sub-ref deployments to Group 0, Group 1 and frwiki (T425662)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:21 elukey: upgrade sudo package on an-* hosts for [[phab:T428384|T428384]] * 07:18 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2052: repool after upgrade * 07:18 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1051: repool after maintenance * 07:17 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:17 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:12 taavi@cumin1003: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database urwikisource ([[phab:T415977|T415977]]) * 07:12 elukey: upgrade exim4 packages on seaborgium for security upgrades * 07:11 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1297681{{!}}Global rollout - Sub-ref deployments to Group 0, Group 1 and frwiki (T425662)]] * 06:36 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1051.eqiad.wmnet with OS trixie * 06:20 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1051.eqiad.wmnet with reason: host reimage * 06:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1051.eqiad.wmnet with reason: host reimage * 06:15 taavi@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database urwikisource ([[phab:T415977|T415977]]) * 05:58 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1051.eqiad.wmnet with OS trixie * 05:54 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2052.codfw.wmnet with OS trixie * 05:44 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool es1051: Upgrading es1051.eqiad.wmnet * 05:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2052.codfw.wmnet with reason: host reimage * 05:35 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2052.codfw.wmnet with reason: host reimage * 05:35 marostegui@dns1004: END - running authdns-update * 05:34 marostegui@dns1004: START - running authdns-update * 05:33 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1051: Upgrading es1051.eqiad.wmnet * 05:33 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:31 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1054 to es3 eqiad primary [[phab:T428050|T428050]]', diff saved to https://phabricator.wikimedia.org/P93895 and previous config saved to /var/cache/conftool/dbconfig/20260608-053156-marostegui.json * 05:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2052.codfw.wmnet with OS trixie * 05:18 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2052: Upgrading es2052.codfw.wmnet * 05:18 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2052: Upgrading es2052.codfw.wmnet * 05:18 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade == 2026-06-07 == * 16:32 elukey: `elukey@cumin1003:~$ sudo cumin 'cp6* and not cp6014* and not cp6010*' "varnish-frontend-restart" -b 1` * 16:29 elukey: restart varnish-frontend on cp6014 == 2026-06-06 == * 09:07 ammarpad@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=hewiki --logwiki=metawiki W.Mechelke Tungsten_Mechelke # [[phab:T428182|T428182]] == 2026-06-05 == * 22:16 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 21:01 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=10 --verbose` (after stopping the other commons scan) * 20:56 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=30 --verbose` (after stopping the other commons scan) * 20:20 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290093{{!}}Enable wmgUseUrlShortenerLegacy on test2wiki (T107188)]] (duration: 10m 02s) * 20:16 krinkle@deploy1003: krinkle: Continuing with deployment * 20:12 krinkle@deploy1003: krinkle: Backport for [[gerrit:1290093{{!}}Enable wmgUseUrlShortenerLegacy on test2wiki (T107188)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:10 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1290093{{!}}Enable wmgUseUrlShortenerLegacy on test2wiki (T107188)]] * 16:45 jgreen@dns1004: END - running authdns-update * 16:44 jgreen@dns1004: START - running authdns-update * 16:17 dzahn@dns1005: END - running authdns-update * 16:17 mutante: DNS - adding new project language "mag" - Magahi - a language spoken in India and Nepal by about 12 million native speakers ([[phab:T428266|T428266]]) * 16:16 dzahn@dns1005: START - running authdns-update * 14:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:38 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:37 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 12:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:30 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:30 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 12:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2202.codfw.wmnet with reason: Reboot * 12:28 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:28 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 12:08 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:07 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 12:07 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:06 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 11:29 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 11:28 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:55 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:54 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:31 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1054: repool after upgrade * 08:08 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 08:07 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 08:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 08:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/kafka-ui: apply * 07:39 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1054: repool after upgrade * 07:38 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 07:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 07:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/kafka-ui: apply * 07:07 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 06:01 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1054.eqiad.wmnet with OS trixie * 05:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1054.eqiad.wmnet with reason: host reimage * 05:37 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1054.eqiad.wmnet with reason: host reimage * 05:22 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1054.eqiad.wmnet with OS trixie * 05:21 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1054: Upgrading es1054.eqiad.wmnet * 05:21 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1054: Upgrading es1054.eqiad.wmnet * 05:20 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 01:55 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1010.eqiad.wmnet with OS trixie * 01:39 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: host reimage * 01:32 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: host reimage * 01:16 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1010.eqiad.wmnet with OS trixie * 00:56 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1007.eqiad.wmnet with OS trixie * 00:40 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1007.eqiad.wmnet with reason: host reimage * 00:33 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1007.eqiad.wmnet with reason: host reimage * 00:17 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1007.eqiad.wmnet with OS trixie * 00:02 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297268{{!}}Redirect unknown wikinews languages to portal (T427126)]] (duration: 07m 02s) == 2026-06-04 == * 23:57 ladsgroup@deploy1003: ladsgroup, pppery: Continuing with deployment * 23:57 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1006.eqiad.wmnet with OS trixie * 23:57 ladsgroup@deploy1003: ladsgroup, pppery: Backport for [[gerrit:1297268{{!}}Redirect unknown wikinews languages to portal (T427126)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:55 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1297268{{!}}Redirect unknown wikinews languages to portal (T427126)]] * 23:40 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1006.eqiad.wmnet with reason: host reimage * 23:36 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1006.eqiad.wmnet with reason: host reimage * 23:20 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1006.eqiad.wmnet with OS trixie * 21:28 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host releases1003.eqiad.wmnet with OS trixie * 21:04 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases1003.eqiad.wmnet with reason: host reimage * 20:58 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on releases1003.eqiad.wmnet with reason: host reimage * 20:50 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5030.* * 20:42 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host releases1003.eqiad.wmnet with OS trixie * 20:27 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1100.eqiad.wmnet,service=(cdn{{!}}ats-be) * 20:26 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6013.drmrs.wmnet,service=(cdn{{!}}ats-be) * 20:20 brett@dns1006: END - running authdns-update * 20:19 brett@dns1006: START - running authdns-update * 20:18 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5030.eqsin.wmnet with OS trixie * 20:10 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296015{{!}}Deploy PRV to 6 wikis (T427851)]] (duration: 07m 39s) * 20:08 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist group2.dblist extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` * 20:06 arlolra@deploy1003: arlolra: Continuing with deployment * 20:04 arlolra@deploy1003: arlolra: Backport for [[gerrit:1296015{{!}}Deploy PRV to 6 wikis (T427851)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:02 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1296015{{!}}Deploy PRV to 6 wikis (T427851)]] * 19:49 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5030.eqsin.wmnet with reason: host reimage * 19:43 cmooney@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5030.eqsin.wmnet with reason: host reimage * 19:15 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5030 * 19:15 cmooney@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5030 * 19:14 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cp5030 * 19:14 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5030.eqsin.wmnet 27.0.132.10.in-addr.arpa 7.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:14 cmooney@cumin1003: START - Cookbook sre.dns.wipe-cache cp5030.eqsin.wmnet 27.0.132.10.in-addr.arpa 7.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:14 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:14 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5030 - cmooney@cumin1003" * 19:13 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5030 - cmooney@cumin1003" * 19:09 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 19:08 cmooney@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5030 * 19:08 cmooney@cumin1003: START - Cookbook sre.hosts.reimage for host cp5030.eqsin.wmnet with OS trixie * 18:51 cmooney@dns2005: END - running authdns-update * 18:50 cmooney@dns2005: START - running authdns-update * 18:43 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:42 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove IPs that had been used for eqsin cr links - cmooney@cumin1003" * 18:40 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove IPs that had been used for eqsin cr links - cmooney@cumin1003" * 18:37 sukhe: sukhe@cp6013:~$ sudo traffic_server -C clear_cache * 18:36 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 18:08 dancy@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.5 refs [[phab:T423914|T423914]] * 17:17 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297751{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297752{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] (duration: 06m 40s) * 17:13 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 17:13 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297751{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297752{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:11 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297751{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297752{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] * 16:55 topranks: shift traffic off cr1-esams et-1/0/1 link to asw1-by27-esams [[phab:T427056|T427056]] * 16:45 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297741{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297742{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] (duration: 13m 58s) * 16:41 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 16:33 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297741{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297742{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:31 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297741{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297742{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] * 16:17 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 16:03 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297740{{!}}hCaptcha: Move ConfirmEditCaptchaClass hook inside hCaptcha block (T428183)]] (duration: 10m 21s) * 16:03 elukey: uploaded spicerack_12.7.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia * 15:59 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 15:55 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297740{{!}}hCaptcha: Move ConfirmEditCaptchaClass hook inside hCaptcha block (T428183)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:53 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297740{{!}}hCaptcha: Move ConfirmEditCaptchaClass hook inside hCaptcha block (T428183)]] * 15:44 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5030.* * 15:41 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2007.codfw.wmnet with OS trixie * 15:39 ladsgroup@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0) * 15:28 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views * 15:24 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297730{{!}}ptwiki: Disable Article Guidance experiment (T426871)]] (duration: 07m 26s) * 15:24 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2007.codfw.wmnet with reason: host reimage * 15:20 sbisson@deploy1003: sbisson: Continuing with deployment * 15:19 sbisson@deploy1003: sbisson: Backport for [[gerrit:1297730{{!}}ptwiki: Disable Article Guidance experiment (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:19 jayme@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2007.codfw.wmnet with reason: host reimage * 15:17 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1297730{{!}}ptwiki: Disable Article Guidance experiment (T426871)]] * 15:13 ladsgroup@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0) * 15:06 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297724{{!}}Revert "Start reading from new file tables on commons"]] (duration: 07m 00s) * 15:05 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views * 15:02 zabe@deploy1003: zabe: Continuing with deployment * 15:01 zabe@deploy1003: zabe: Backport for [[gerrit:1297724{{!}}Revert "Start reading from new file tables on commons"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:59 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1297724{{!}}Revert "Start reading from new file tables on commons"]] * 14:57 zabe@deploy1003: Finished scap sync-world: [[phab:T416548|T416548]] (duration: 05m 10s) * 14:56 jayme@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-main2007.codfw.wmnet with OS trixie * 14:52 zabe@deploy1003: Started scap sync-world: [[phab:T416548|T416548]] * 14:50 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:49 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:43 zabe@deploy1003: sync-world aborted: Backport for [[gerrit:1270513{{!}}Start reading from new file tables on commons (T416548)]] (duration: 03m 58s) * 14:43 zabe@deploy1003: zabe: Continuing with deployment * 14:41 zabe@deploy1003: zabe: Backport for [[gerrit:1270513{{!}}Start reading from new file tables on commons (T416548)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:40 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f1-codfw * 14:40 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device lsw1-f1-codfw * 14:39 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1270513{{!}}Start reading from new file tables on commons (T416548)]] * 14:36 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297711{{!}}hCaptcha: Enable for MobileFrontend in some Group 2 wikis (T425940)]] (duration: 08m 20s) * 14:32 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:30 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297711{{!}}hCaptcha: Enable for MobileFrontend in some Group 2 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:29 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1057: repool after upgrade * 14:28 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297711{{!}}hCaptcha: Enable for MobileFrontend in some Group 2 wikis (T425940)]] * 14:20 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 14:16 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:15 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply * 14:15 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:15 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply * 14:13 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297704{{!}}Use the globalblock-local-status right over globalblock-whitelist (T277942)]], [[gerrit:1296620{{!}}core-Permissions: Stop assigning unused globalblock-whitelist right (T277942)]] (duration: 06m 46s) * 14:10 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 14:08 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:08 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297704{{!}}Use the globalblock-local-status right over globalblock-whitelist (T277942)]], [[gerrit:1296620{{!}}core-Permissions: Stop assigning unused globalblock-whitelist right (T277942)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:07 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:06 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply * 14:06 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297704{{!}}Use the globalblock-local-status right over globalblock-whitelist (T277942)]], [[gerrit:1296620{{!}}core-Permissions: Stop assigning unused globalblock-whitelist right (T277942)]] * 14:06 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:06 tappof: bump space for prometheus k8s-aux in eqiad * 14:05 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply * 14:05 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:04 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply * 13:56 _joe_: transferred requestctl api tokens for all ops to the db ([[phab:T428119|T428119]]) * 13:56 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2050 to es3 codfw primary [[phab:T428050|T428050]]', diff saved to https://phabricator.wikimedia.org/P93878 and previous config saved to /var/cache/conftool/dbconfig/20260604-135631-marostegui.json * 13:56 Dreamy_Jazz: Afternoon UTC backport window done * 13:54 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297700{{!}}Revert "hCaptcha: Provide always challenge sitekey for account creation"]] (duration: 13m 38s) * 13:51 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 13:50 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 13:47 sukhe: sukhe@cp6011:~$ sudo -i varnish-frontend-restart * 13:44 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1057: repool after upgrade * 13:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:43 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297700{{!}}Revert "hCaptcha: Provide always challenge sitekey for account creation"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:41 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1057.eqiad.wmnet with OS trixie * 13:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297700{{!}}Revert "hCaptcha: Provide always challenge sitekey for account creation"]] * 13:38 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297692{{!}}hCaptcha: Provide always challenge sitekey for account creation (T421041)]] (duration: 05m 27s) * 13:38 dreamyjazz@deploy1003: dreamyjazz: Rolling back deployment * 13:36 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: down * 13:35 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297692{{!}}hCaptcha: Provide always challenge sitekey for account creation (T421041)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:33 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297692{{!}}hCaptcha: Provide always challenge sitekey for account creation (T421041)]] * 13:31 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295978{{!}}Update config for WikiProjects linking prototype (T427804)]] (duration: 17m 13s) * 13:26 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Continuing with deployment * 13:25 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1057.eqiad.wmnet with reason: host reimage * 13:17 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1057.eqiad.wmnet with reason: host reimage * 13:16 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Backport for [[gerrit:1295978{{!}}Update config for WikiProjects linking prototype (T427804)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:14 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1295978{{!}}Update config for WikiProjects linking prototype (T427804)]] * 13:13 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:13 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1220: Migration of db1220.eqiad.wmnet completed * 13:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: down * 13:12 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1224', diff saved to https://phabricator.wikimedia.org/P93875 and previous config saved to /var/cache/conftool/dbconfig/20260604-131219-marostegui.json * 13:00 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1057.eqiad.wmnet with OS trixie * 13:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1057: Upgrading es1057.eqiad.wmnet * 12:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1057: Upgrading es1057.eqiad.wmnet * 12:59 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:56 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296557{{!}}wmf-config: Skip CAPTCHA for action=mcrundo (T427612)]] (duration: 08m 30s) * 12:52 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Continuing with deployment * 12:50 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Backport for [[gerrit:1296557{{!}}wmf-config: Skip CAPTCHA for action=mcrundo (T427612)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2050: repool after upgrade * 12:48 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296557{{!}}wmf-config: Skip CAPTCHA for action=mcrundo (T427612)]] * 12:37 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 12:37 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 12:28 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1220: Migration of db1220.eqiad.wmnet completed * 12:20 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1220.eqiad.wmnet with OS trixie * 12:04 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2050: repool after upgrade * 12:04 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 12:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1220.eqiad.wmnet with reason: host reimage * 11:59 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1220.eqiad.wmnet with reason: host reimage * 11:42 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1220.eqiad.wmnet with OS trixie * 11:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2050.codfw.wmnet with OS trixie * 11:40 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1220: Upgrading db1220.eqiad.wmnet * 11:37 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1220: Upgrading db1220.eqiad.wmnet * 11:36 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1179: Migration of db1179.eqiad.wmnet completed * 11:23 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2050.codfw.wmnet with reason: host reimage * 11:16 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2050.codfw.wmnet with reason: host reimage * 11:00 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2050.codfw.wmnet with OS trixie * 11:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2050: Upgrading es2050.codfw.wmnet * 10:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2050: Upgrading es2050.codfw.wmnet * 10:59 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:59 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2057: repool after upgrade * 10:58 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:55 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 10:46 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1179: Migration of db1179.eqiad.wmnet completed * 10:38 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1179.eqiad.wmnet with OS trixie * 10:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1179.eqiad.wmnet with reason: host reimage * 10:16 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply * 10:15 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply * 10:15 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply * 10:15 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/kartotherian: apply * 10:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1179.eqiad.wmnet with reason: host reimage * 10:13 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2057: repool after upgrade * 10:13 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2057.codfw.wmnet with OS trixie * 09:59 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1179.eqiad.wmnet with OS trixie * 09:58 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1179: Upgrading db1179.eqiad.wmnet * 09:58 jynus: redoing m2 backups after grant change [[phab:T411111|T411111]] * 09:57 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1179: Upgrading db1179.eqiad.wmnet * 09:56 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:54 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2057.codfw.wmnet with reason: host reimage * 09:53 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 09:49 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2057.codfw.wmnet with reason: host reimage * 09:39 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:39 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Migration of db1224.eqiad.wmnet completed * 09:38 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 09:37 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 09:36 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 09:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/kafka-ui: apply * 09:33 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2057.codfw.wmnet with OS trixie * 09:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2057: Upgrading es2057.codfw.wmnet * 09:32 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2057: Upgrading es2057.codfw.wmnet * 09:31 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:26 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=30 --sleep=60 --verbose` * 09:25 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist "group0.dblist + group1.dblist - mediamoderation-continuous-scan.dblist" extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` * 08:54 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Introduce pluggable authentication - oblivian@cumin1003" * 08:54 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Introduce pluggable authentication - oblivian@cumin1003 * 08:53 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Migration of db1224.eqiad.wmnet completed * 08:53 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Introduce pluggable authentication - oblivian@cumin1003 * 08:53 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Introduce pluggable authentication - oblivian@cumin1003" * 08:29 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 08:29 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 08:24 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 08:24 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 08:21 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 08:21 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1224.eqiad.wmnet with OS trixie * 08:21 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 08:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1224.eqiad.wmnet with reason: host reimage * 08:02 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2249.codfw.wmnet with reason: upgrade * 08:00 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1224.eqiad.wmnet with reason: host reimage * 07:53 marostegui: Install mariadb 10.11.17 on db2249 [[phab:T427345|T427345]] * 07:43 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1224.eqiad.wmnet with OS trixie * 07:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1224: Upgrading db1224.eqiad.wmnet * 07:41 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1224: Upgrading db1224.eqiad.wmnet * 07:41 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:39 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:39 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1255: Migration of db1255.eqiad.wmnet completed * 07:34 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297536{{!}}hCaptcha risk scores: VE plugin to collect risk scores for block notices (T426943)]], [[gerrit:1297200{{!}}hCaptcha: Render a fresh mobile widget for each captcha attempt (T425929)]], [[gerrit:1297173{{!}}hCaptcha: Enable risk-score collection for users blocked by IP blocks (T424629)]] (duration: 08m 56s) * 07:29 kharlan@deploy1003: kharlan, harroyo-wmf: Continuing with deployment * 07:27 kharlan@deploy1003: kharlan, harroyo-wmf: Backport for [[gerrit:1297536{{!}}hCaptcha risk scores: VE plugin to collect risk scores for block notices (T426943)]], [[gerrit:1297200{{!}}hCaptcha: Render a fresh mobile widget for each captcha attempt (T425929)]], [[gerrit:1297173{{!}}hCaptcha: Enable risk-score collection for users blocked by IP blocks (T424629)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwd * 07:25 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297536{{!}}hCaptcha risk scores: VE plugin to collect risk scores for block notices (T426943)]], [[gerrit:1297200{{!}}hCaptcha: Render a fresh mobile widget for each captcha attempt (T425929)]], [[gerrit:1297173{{!}}hCaptcha: Enable risk-score collection for users blocked by IP blocks (T424629)]] * 07:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2191: Migration of db2191.codfw.wmnet completed * 07:12 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297550{{!}}Revert "EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion"]] (duration: 06m 45s) * 07:08 kharlan@deploy1003: kharlan: Continuing with deployment * 07:08 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297550{{!}}Revert "EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:06 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297550{{!}}Revert "EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion"]] * 07:04 otto@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297260{{!}}EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] (duration: 399m 30s) * 07:03 otto@deploy1003: otto: Rolling back deployment * 06:53 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1255: Migration of db1255.eqiad.wmnet completed * 06:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1255.eqiad.wmnet with OS trixie * 06:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2191: Migration of db2191.codfw.wmnet completed * 06:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1255.eqiad.wmnet with reason: host reimage * 06:32 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2191.codfw.wmnet with OS trixie * 06:31 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1255.eqiad.wmnet with reason: host reimage * 06:16 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1255.eqiad.wmnet with OS trixie * 06:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2191.codfw.wmnet with reason: host reimage * 06:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1255: Upgrading db1255.eqiad.wmnet * 06:12 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1255: Upgrading db1255.eqiad.wmnet * 06:12 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2191.codfw.wmnet with reason: host reimage * 06:04 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db1255 [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93836 and previous config saved to /var/cache/conftool/dbconfig/20260604-060428-cwilliams.json * 06:03 cwilliams@dns1004: END - running authdns-update * 06:02 cwilliams@dns1004: START - running authdns-update * 05:54 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db1258 to x3 primary and set section read-write [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93835 and previous config saved to /var/cache/conftool/dbconfig/20260604-055429-cwilliams.json * 05:53 cwilliams@cumin1003: dbctl commit (dc=all): 'Set x3 eqiad as read-only for maintenance - [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93834 and previous config saved to /var/cache/conftool/dbconfig/20260604-055346-cwilliams.json * 05:53 cezmunsta: Starting x3 eqiad failover from db1255 to db1258 - [[phab:T427895|T427895]] * 05:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2191.codfw.wmnet with OS trixie * 05:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2191: Upgrading db2191.codfw.wmnet * 05:50 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2191: Upgrading db2191.codfw.wmnet * 05:50 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db1258 with weight 0 [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93833 and previous config saved to /var/cache/conftool/dbconfig/20260604-055021-cwilliams.json * 05:50 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:50 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Primary switchover x3 [[phab:T427895|T427895]] * 05:48 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 05:46 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2191 [[phab:T428120|T428120]]', diff saved to https://phabricator.wikimedia.org/P93832 and previous config saved to /var/cache/conftool/dbconfig/20260604-054614-marostegui.json * 05:45 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db2215 to x1 primary [[phab:T428120|T428120]]', diff saved to https://phabricator.wikimedia.org/P93831 and previous config saved to /var/cache/conftool/dbconfig/20260604-054528-marostegui.json * 05:44 marostegui: Starting x1 codfw failover from db2191 to db2215 - [[phab:T428120|T428120]] * 05:27 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: Primary switchover x1 [[phab:T428120|T428120]] * 05:27 marostegui@cumin1003: dbctl commit (dc=all): 'Set db2215 with weight 0 [[phab:T428120|T428120]]', diff saved to https://phabricator.wikimedia.org/P93830 and previous config saved to /var/cache/conftool/dbconfig/20260604-052722-marostegui.json * 05:19 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 03:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93829 and previous config saved to /var/cache/conftool/dbconfig/20260604-034546-fceratto.json * 03:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P93828 and previous config saved to /var/cache/conftool/dbconfig/20260604-033538-fceratto.json * 03:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P93827 and previous config saved to /var/cache/conftool/dbconfig/20260604-032531-fceratto.json * 03:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93826 and previous config saved to /var/cache/conftool/dbconfig/20260604-031523-fceratto.json * 03:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1263 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93825 and previous config saved to /var/cache/conftool/dbconfig/20260604-030710-fceratto.json * 03:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1263.eqiad.wmnet with reason: Maintenance * 03:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93824 and previous config saved to /var/cache/conftool/dbconfig/20260604-030642-fceratto.json * 02:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P93823 and previous config saved to /var/cache/conftool/dbconfig/20260604-025634-fceratto.json * 02:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P93822 and previous config saved to /var/cache/conftool/dbconfig/20260604-024627-fceratto.json * 02:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93821 and previous config saved to /var/cache/conftool/dbconfig/20260604-023619-fceratto.json * 02:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1262 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93820 and previous config saved to /var/cache/conftool/dbconfig/20260604-022809-fceratto.json * 02:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Maintenance * 02:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93819 and previous config saved to /var/cache/conftool/dbconfig/20260604-022742-fceratto.json * 02:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P93818 and previous config saved to /var/cache/conftool/dbconfig/20260604-021734-fceratto.json * 02:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P93817 and previous config saved to /var/cache/conftool/dbconfig/20260604-020726-fceratto.json * 01:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93816 and previous config saved to /var/cache/conftool/dbconfig/20260604-015718-fceratto.json * 01:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1261 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93815 and previous config saved to /var/cache/conftool/dbconfig/20260604-014909-fceratto.json * 01:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1261.eqiad.wmnet with reason: Maintenance * 01:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93814 and previous config saved to /var/cache/conftool/dbconfig/20260604-014841-fceratto.json * 01:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P93813 and previous config saved to /var/cache/conftool/dbconfig/20260604-013833-fceratto.json * 01:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P93812 and previous config saved to /var/cache/conftool/dbconfig/20260604-012826-fceratto.json * 01:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93811 and previous config saved to /var/cache/conftool/dbconfig/20260604-011818-fceratto.json * 01:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1260 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93810 and previous config saved to /var/cache/conftool/dbconfig/20260604-011005-fceratto.json * 01:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1260.eqiad.wmnet with reason: Maintenance * 01:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93809 and previous config saved to /var/cache/conftool/dbconfig/20260604-010937-fceratto.json * 00:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P93808 and previous config saved to /var/cache/conftool/dbconfig/20260604-005929-fceratto.json * 00:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P93807 and previous config saved to /var/cache/conftool/dbconfig/20260604-004922-fceratto.json * 00:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93806 and previous config saved to /var/cache/conftool/dbconfig/20260604-003914-fceratto.json * 00:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1252 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93805 and previous config saved to /var/cache/conftool/dbconfig/20260604-002851-fceratto.json * 00:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1252.eqiad.wmnet with reason: Maintenance * 00:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93804 and previous config saved to /var/cache/conftool/dbconfig/20260604-002821-fceratto.json * 00:26 otto@deploy1003: otto: Backport for [[gerrit:1297260{{!}}EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 00:24 otto@deploy1003: Started scap sync-world: Backport for [[gerrit:1297260{{!}}EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] * 00:18 Amir1: mwscript-k8s --follow --dblist=all -- extensions/timeline/maintenance/DeleteOldTimelineFiles.php --date {{Gerrit|20210101000000}} * 00:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P93803 and previous config saved to /var/cache/conftool/dbconfig/20260604-001813-fceratto.json * 00:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P93802 and previous config saved to /var/cache/conftool/dbconfig/20260604-000805-fceratto.json == 2026-06-03 == * 23:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93801 and previous config saved to /var/cache/conftool/dbconfig/20260603-235758-fceratto.json * 23:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93800 and previous config saved to /var/cache/conftool/dbconfig/20260603-234935-fceratto.json * 23:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1248.eqiad.wmnet with reason: Maintenance * 23:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93799 and previous config saved to /var/cache/conftool/dbconfig/20260603-234907-fceratto.json * 23:42 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296561{{!}}Add a maintenance script to delete old files]], [[gerrit:1296560{{!}}Add a maintenance script to delete old files]] (duration: 07m 09s) * 23:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P93798 and previous config saved to /var/cache/conftool/dbconfig/20260603-233859-fceratto.json * 23:37 ladsgroup@deploy1003: ladsgroup, reedy: Continuing with deployment * 23:36 ladsgroup@deploy1003: ladsgroup, reedy: Backport for [[gerrit:1296561{{!}}Add a maintenance script to delete old files]], [[gerrit:1296560{{!}}Add a maintenance script to delete old files]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:34 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1296561{{!}}Add a maintenance script to delete old files]], [[gerrit:1296560{{!}}Add a maintenance script to delete old files]] * 23:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P93797 and previous config saved to /var/cache/conftool/dbconfig/20260603-232852-fceratto.json * 23:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 23:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 23:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93796 and previous config saved to /var/cache/conftool/dbconfig/20260603-231844-fceratto.json * 23:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93795 and previous config saved to /var/cache/conftool/dbconfig/20260603-231031-fceratto.json * 23:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1247.eqiad.wmnet with reason: Maintenance * 23:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93794 and previous config saved to /var/cache/conftool/dbconfig/20260603-231001-fceratto.json * 22:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P93793 and previous config saved to /var/cache/conftool/dbconfig/20260603-225953-fceratto.json * 22:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P93792 and previous config saved to /var/cache/conftool/dbconfig/20260603-224945-fceratto.json * 22:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93791 and previous config saved to /var/cache/conftool/dbconfig/20260603-223937-fceratto.json * 22:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1244 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93790 and previous config saved to /var/cache/conftool/dbconfig/20260603-223116-fceratto.json * 22:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1244.eqiad.wmnet with reason: Maintenance * 22:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93789 and previous config saved to /var/cache/conftool/dbconfig/20260603-223048-fceratto.json * 22:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P93788 and previous config saved to /var/cache/conftool/dbconfig/20260603-222041-fceratto.json * 22:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P93787 and previous config saved to /var/cache/conftool/dbconfig/20260603-221034-fceratto.json * 22:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93786 and previous config saved to /var/cache/conftool/dbconfig/20260603-220026-fceratto.json * 21:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1243 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93785 and previous config saved to /var/cache/conftool/dbconfig/20260603-215110-fceratto.json * 21:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1243.eqiad.wmnet with reason: Maintenance * 21:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93784 and previous config saved to /var/cache/conftool/dbconfig/20260603-215053-fceratto.json * 21:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P93783 and previous config saved to /var/cache/conftool/dbconfig/20260603-214046-fceratto.json * 21:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P93782 and previous config saved to /var/cache/conftool/dbconfig/20260603-213038-fceratto.json * 21:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93781 and previous config saved to /var/cache/conftool/dbconfig/20260603-212030-fceratto.json * 21:12 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1242 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93779 and previous config saved to /var/cache/conftool/dbconfig/20260603-211206-fceratto.json * 21:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1242.eqiad.wmnet with reason: Maintenance * 21:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93778 and previous config saved to /var/cache/conftool/dbconfig/20260603-211138-fceratto.json * 21:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P93774 and previous config saved to /var/cache/conftool/dbconfig/20260603-210130-fceratto.json * 20:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P93773 and previous config saved to /var/cache/conftool/dbconfig/20260603-205122-fceratto.json * 20:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93772 and previous config saved to /var/cache/conftool/dbconfig/20260603-204115-fceratto.json * 20:33 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297228{{!}}Attribution research don't use testKitchen compatibility layer (T417050)]] (duration: 06m 41s) * 20:32 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1241 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93771 and previous config saved to /var/cache/conftool/dbconfig/20260603-203254-fceratto.json * 20:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1241.eqiad.wmnet with reason: Maintenance * 20:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93770 and previous config saved to /var/cache/conftool/dbconfig/20260603-203227-fceratto.json * 20:29 cjming@deploy1003: cjming: Continuing with deployment * 20:29 cjming@deploy1003: cjming: Backport for [[gerrit:1297228{{!}}Attribution research don't use testKitchen compatibility layer (T417050)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:26 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1297228{{!}}Attribution research don't use testKitchen compatibility layer (T417050)]] * 20:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P93769 and previous config saved to /var/cache/conftool/dbconfig/20260603-202219-fceratto.json * 20:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P93766 and previous config saved to /var/cache/conftool/dbconfig/20260603-201211-fceratto.json * 20:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93765 and previous config saved to /var/cache/conftool/dbconfig/20260603-200203-fceratto.json * 19:59 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 19:59 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 19:59 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 19:59 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 19:53 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93764 and previous config saved to /var/cache/conftool/dbconfig/20260603-195341-fceratto.json * 19:53 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1238.eqiad.wmnet with reason: Maintenance * 19:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93763 and previous config saved to /var/cache/conftool/dbconfig/20260603-195313-fceratto.json * 19:47 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5032.* * 19:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P93762 and previous config saved to /var/cache/conftool/dbconfig/20260603-194306-fceratto.json * 19:39 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5032.* * 19:37 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5032.* * 19:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P93761 and previous config saved to /var/cache/conftool/dbconfig/20260603-193258-fceratto.json * 19:26 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 19:25 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 19:25 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 19:25 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 19:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93760 and previous config saved to /var/cache/conftool/dbconfig/20260603-192250-fceratto.json * 19:22 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 19:22 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 19:14 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93759 and previous config saved to /var/cache/conftool/dbconfig/20260603-191437-fceratto.json * 19:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1024-1025].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 19:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1221.eqiad.wmnet with reason: Maintenance * 19:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93758 and previous config saved to /var/cache/conftool/dbconfig/20260603-191348-fceratto.json * 19:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P93757 and previous config saved to /var/cache/conftool/dbconfig/20260603-190340-fceratto.json * 18:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P93756 and previous config saved to /var/cache/conftool/dbconfig/20260603-185331-fceratto.json * 18:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93755 and previous config saved to /var/cache/conftool/dbconfig/20260603-184324-fceratto.json * 18:34 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1199 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93754 and previous config saved to /var/cache/conftool/dbconfig/20260603-183455-fceratto.json * 18:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance * 18:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93753 and previous config saved to /var/cache/conftool/dbconfig/20260603-183427-fceratto.json * 18:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P93752 and previous config saved to /var/cache/conftool/dbconfig/20260603-182420-fceratto.json * 18:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P93751 and previous config saved to /var/cache/conftool/dbconfig/20260603-181412-fceratto.json * 18:10 dancy@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.5 refs [[phab:T423914|T423914]] * 18:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93750 and previous config saved to /var/cache/conftool/dbconfig/20260603-180404-fceratto.json * 17:57 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5032.* * 17:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93749 and previous config saved to /var/cache/conftool/dbconfig/20260603-175544-fceratto.json * 17:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance * 17:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93748 and previous config saved to /var/cache/conftool/dbconfig/20260603-175342-fceratto.json * 17:52 hashar: contint1003: sudo puppet agent --disable "Prevent Jenkins from coming back" * 17:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P93747 and previous config saved to /var/cache/conftool/dbconfig/20260603-174334-fceratto.json * 17:38 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:37 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 17:37 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:36 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:36 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:35 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:35 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:35 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:34 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:34 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:33 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P93746 and previous config saved to /var/cache/conftool/dbconfig/20260603-173327-fceratto.json * 17:33 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:32 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:29 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5032.* * 17:26 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host sretest2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 17:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93745 and previous config saved to /var/cache/conftool/dbconfig/20260603-172319-fceratto.json * 17:18 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: Stopping before sync operations * 17:17 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: Started scap sync-world: No-deploy scap run to verify scap config change * 17:17 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:15 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:15 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1253 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93744 and previous config saved to /var/cache/conftool/dbconfig/20260603-171521-fceratto.json * 17:15 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:15 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance * 17:14 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93743 and previous config saved to /var/cache/conftool/dbconfig/20260603-171452-fceratto.json * 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:13 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:12 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:10 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:09 ayounsi@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2012.wikimedia.org with OS trixie * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P93742 and previous config saved to /var/cache/conftool/dbconfig/20260603-170444-fceratto.json * 17:04 swfrench@deploy1003: Stopping before sync operations * 17:03 swfrench@deploy1003: Started scap sync-world: No-deploy scap run to verify clean state before config change * 16:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P93741 and previous config saved to /var/cache/conftool/dbconfig/20260603-165436-fceratto.json * 16:53 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:53 hashar: Restarting CI Jenkins one last time # [[phab:T418521|T418521]] * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:44 btullis@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295922{{!}}Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] (duration: 07m 16s) * 16:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93740 and previous config saved to /var/cache/conftool/dbconfig/20260603-164428-fceratto.json * 16:43 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:43 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:42 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:41 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:40 btullis@deploy1003: btullis: Continuing with deployment * 16:39 btullis@deploy1003: btullis: Backport for [[gerrit:1295922{{!}}Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93739 and previous config saved to /var/cache/conftool/dbconfig/20260603-163726-fceratto.json * 16:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance * 16:37 btullis@deploy1003: Started scap sync-world: Backport for [[gerrit:1295922{{!}}Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] * 16:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93738 and previous config saved to /var/cache/conftool/dbconfig/20260603-163658-fceratto.json * 16:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P93737 and previous config saved to /var/cache/conftool/dbconfig/20260603-162650-fceratto.json * 16:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P93736 and previous config saved to /var/cache/conftool/dbconfig/20260603-161643-fceratto.json * 16:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93735 and previous config saved to /var/cache/conftool/dbconfig/20260603-160635-fceratto.json * 16:04 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:59 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93734 and previous config saved to /var/cache/conftool/dbconfig/20260603-155928-fceratto.json * 15:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance * 15:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93733 and previous config saved to /var/cache/conftool/dbconfig/20260603-155859-fceratto.json * 15:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 15:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P93732 and previous config saved to /var/cache/conftool/dbconfig/20260603-154852-fceratto.json * 15:46 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:46 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2012.wikimedia.org with OS trixie * 15:40 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:40 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 15:40 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 15:40 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 15:39 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 15:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P93731 and previous config saved to /var/cache/conftool/dbconfig/20260603-153844-fceratto.json * 15:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93729 and previous config saved to /var/cache/conftool/dbconfig/20260603-152836-fceratto.json * 15:25 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host sretest2012 * 15:25 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2012 * 15:25 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host sretest2012 * 15:25 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2012 * 15:24 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:23 mutante: disabling jenkins on CI servers for maintenance * 15:23 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host sretest2012 * 15:23 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2012 * 15:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 15:21 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1202 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93728 and previous config saved to /var/cache/conftool/dbconfig/20260603-152129-fceratto.json * 15:21 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance * 15:21 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:21 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding sretest2012 to codfw - jhancock@cumin2002" * 15:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 15:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93727 and previous config saved to /var/cache/conftool/dbconfig/20260603-152102-fceratto.json * 15:20 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding sretest2012 to codfw - jhancock@cumin2002" * 15:18 brouberol@dns1004: END - running authdns-update * 15:18 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:16 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 15:16 brouberol@dns1004: START - running authdns-update * 15:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P93726 and previous config saved to /var/cache/conftool/dbconfig/20260603-151055-fceratto.json * 15:01 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P93725 and previous config saved to /var/cache/conftool/dbconfig/20260603-150047-fceratto.json * 14:57 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 14:52 cmooney@cumin1003: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox * 14:51 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93723 and previous config saved to /var/cache/conftool/dbconfig/20260603-145039-fceratto.json * 14:48 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297137{{!}}Revert "MultimediaViewer: enable image carousel as a beta feature on Wikipedias"]] (duration: 06m 46s) * 14:47 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 14:46 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:46 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:43 mlitn@deploy1003: mlitn: Continuing with deployment * 14:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93722 and previous config saved to /var/cache/conftool/dbconfig/20260603-144334-fceratto.json * 14:43 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:43 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance * 14:43 mlitn@deploy1003: mlitn: Backport for [[gerrit:1297137{{!}}Revert "MultimediaViewer: enable image carousel as a beta feature on Wikipedias"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93721 and previous config saved to /var/cache/conftool/dbconfig/20260603-144306-fceratto.json * 14:41 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:41 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:41 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1297137{{!}}Revert "MultimediaViewer: enable image carousel as a beta feature on Wikipedias"]] * 14:39 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:39 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:39 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:39 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:38 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:35 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 14:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 14:34 sgimeno@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297130{{!}}editor: make redesigned anon warning the default experience (T424595)]] (duration: 10m 45s) * 14:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P93719 and previous config saved to /var/cache/conftool/dbconfig/20260603-143259-fceratto.json * 14:30 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:28 sgimeno@deploy1003: sgimeno: Continuing with deployment * 14:25 sgimeno@deploy1003: sgimeno: Backport for [[gerrit:1297130{{!}}editor: make redesigned anon warning the default experience (T424595)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:24 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:24 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:23 sgimeno@deploy1003: Started scap sync-world: Backport for [[gerrit:1297130{{!}}editor: make redesigned anon warning the default experience (T424595)]] * 14:23 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P93717 and previous config saved to /var/cache/conftool/dbconfig/20260603-142251-fceratto.json * 14:22 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:22 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:21 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:21 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:21 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:20 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:20 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:20 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:20 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:19 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:19 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:16 vriley@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:16 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:16 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:13 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:12 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93716 and previous config saved to /var/cache/conftool/dbconfig/20260603-141242-fceratto.json * 14:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:11 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:11 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:10 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mc2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:10 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host mc2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:10 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:09 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:08 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:07 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:05 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296631{{!}}translate: adding separate read/write endpoints (T425377)]] (duration: 13m 06s) * 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1191 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93715 and previous config saved to /var/cache/conftool/dbconfig/20260603-140537-fceratto.json * 14:05 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance * 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93714 and previous config saved to /var/cache/conftool/dbconfig/20260603-140507-fceratto.json * 14:01 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:58 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:58 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:58 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 13:56 dcausse@deploy1003: atsuko, dcausse: Rolling back deployment * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T426633|T426633]])', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20260603-133440-fceratto.json * 13:29 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:29 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2186: Migration of db2186.codfw.wmnet completed * 13:28 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295910{{!}}hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]] (duration: 07m 36s) * 13:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1174 ([[phab:T426633|T426633]])', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20260603-132638-fceratto.json * 13:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance * 13:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93710 and previous config saved to /var/cache/conftool/dbconfig/20260603-132605-fceratto.json * 13:25 sukhe: sudo cumin 'A:lvs or A:liberica' 'disable-puppet "merging CR 1282764"' * 13:23 kharlan@deploy1003: kharlan: Continuing with deployment * 13:22 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295910{{!}}hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:20 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295910{{!}}hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]] * 13:18 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296649{{!}}hCaptcha: Roll out to all except enwiki for mobile apps. (T426048)]] (duration: 07m 46s) * 13:16 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20260603-131556-fceratto.json * 13:15 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:13 kharlan@deploy1003: dbrant, kharlan: Continuing with deployment * 13:12 kharlan@deploy1003: dbrant, kharlan: Backport for [[gerrit:1296649{{!}}hCaptcha: Roll out to all except enwiki for mobile apps. (T426048)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296649{{!}}hCaptcha: Roll out to all except enwiki for mobile apps. (T426048)]] * 13:09 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:09 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add codfw d3 and e5 public vlans - ayounsi@cumin1003" * 13:09 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add codfw d3 and e5 public vlans - ayounsi@cumin1003" * 13:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P93708 and previous config saved to /var/cache/conftool/dbconfig/20260603-130548-fceratto.json * 13:05 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 12:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93706 and previous config saved to /var/cache/conftool/dbconfig/20260603-125540-fceratto.json * 12:51 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297110{{!}}ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] (duration: 07m 44s) * 12:49 jgreen@dns1004: END - running authdns-update * 12:47 jgreen@dns1004: START - running authdns-update * 12:46 jiji@deploy1003: jiji: Continuing with deployment * 12:46 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93705 and previous config saved to /var/cache/conftool/dbconfig/20260603-124624-fceratto.json * 12:46 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance * 12:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93704 and previous config saved to /var/cache/conftool/dbconfig/20260603-124556-fceratto.json * 12:45 jiji@deploy1003: jiji: Backport for [[gerrit:1297110{{!}}ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:43 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2186: Migration of db2186.codfw.wmnet completed * 12:43 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1297110{{!}}ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] * 12:41 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1067.eqiad.wmnet with OS bullseye * 12:38 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1292364{{!}}Update hCaptcha checks to retrieve API parameters from $_REQUEST (T427105)]] (duration: 11m 15s) * 12:36 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2186.codfw.wmnet with OS trixie * 12:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P93702 and previous config saved to /var/cache/conftool/dbconfig/20260603-123548-fceratto.json * 12:34 dreamyjazz@deploy1003: somerandomdeveloper, dreamyjazz: Continuing with deployment * 12:31 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1066.eqiad.wmnet with OS bullseye * 12:29 dreamyjazz@deploy1003: somerandomdeveloper, dreamyjazz: Backport for [[gerrit:1292364{{!}}Update hCaptcha checks to retrieve API parameters from $_REQUEST (T427105)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:27 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1292364{{!}}Update hCaptcha checks to retrieve API parameters from $_REQUEST (T427105)]] * 12:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P93701 and previous config saved to /var/cache/conftool/dbconfig/20260603-122541-fceratto.json * 12:22 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1067.eqiad.wmnet with reason: host reimage * 12:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2186.codfw.wmnet with reason: host reimage * 12:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93700 and previous config saved to /var/cache/conftool/dbconfig/20260603-121533-fceratto.json * 12:13 mvernon@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ms-be1066.eqiad.wmnet with reason: host reimage * 12:13 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2186.codfw.wmnet with reason: host reimage * 12:11 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1067.eqiad.wmnet with reason: host reimage * 12:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93699 and previous config saved to /var/cache/conftool/dbconfig/20260603-120732-fceratto.json * 12:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 12:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance * 12:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93698 and previous config saved to /var/cache/conftool/dbconfig/20260603-120634-fceratto.json * 12:03 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1066.eqiad.wmnet with reason: host reimage * 11:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P93697 and previous config saved to /var/cache/conftool/dbconfig/20260603-115626-fceratto.json * 11:54 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2186.codfw.wmnet with OS trixie * 11:54 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be1067 * 11:54 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1067 * 11:52 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1067 * 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be1067.eqiad.wmnet 96.48.64.10.in-addr.arpa 6.9.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:52 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be1067.eqiad.wmnet 96.48.64.10.in-addr.arpa 6.9.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1067 - mvernon@cumin2002" * 11:52 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1067 - mvernon@cumin2002" * 11:48 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2186: Upgrading db2186.codfw.wmnet * 11:48 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2186: Upgrading db2186.codfw.wmnet * 11:48 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:47 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:46 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be1067 * 11:46 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be1067.eqiad.wmnet with OS bullseye * 11:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P93695 and previous config saved to /var/cache/conftool/dbconfig/20260603-114618-fceratto.json * 11:46 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be1066 * 11:46 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1066 * 11:45 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1066 * 11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be1066.eqiad.wmnet 117.32.64.10.in-addr.arpa 7.1.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:45 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be1066.eqiad.wmnet 117.32.64.10.in-addr.arpa 7.1.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1066 - mvernon@cumin2002" * 11:45 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1066 - mvernon@cumin2002" * 11:43 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply * 11:41 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:40 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be1066 * 11:40 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be1066.eqiad.wmnet with OS bullseye * 11:39 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be1067 * 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93693 and previous config saved to /var/cache/conftool/dbconfig/20260603-113611-fceratto.json * 11:33 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:33 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2196: Migration of db2196.codfw.wmnet completed * 11:29 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93691 and previous config saved to /var/cache/conftool/dbconfig/20260603-112909-fceratto.json * 11:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance * 11:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance * 11:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93690 and previous config saved to /var/cache/conftool/dbconfig/20260603-112838-fceratto.json * 11:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:20 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:20 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:20 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P93689 and previous config saved to /var/cache/conftool/dbconfig/20260603-111831-fceratto.json * 11:14 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:09 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply * 11:09 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply * 11:08 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P93687 and previous config saved to /var/cache/conftool/dbconfig/20260603-110823-fceratto.json * 11:07 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be1066 * 11:07 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply * 11:06 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply * 11:05 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply * 11:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:01 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:01 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:00 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289895{{!}}Update UserInfoCard to be enabled by default for certain user groups (T426021)]] (duration: 07m 37s) * 11:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:59 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply * 10:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:59 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply * 10:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:58 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply * 10:58 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93685 and previous config saved to /var/cache/conftool/dbconfig/20260603-105815-fceratto.json * 10:58 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:56 mszwarc@deploy1003: mszwarc: Continuing with deployment * 10:55 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1289895{{!}}Update UserInfoCard to be enabled by default for certain user groups (T426021)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:54 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply * 10:54 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply * 10:53 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply * 10:53 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1289895{{!}}Update UserInfoCard to be enabled by default for certain user groups (T426021)]] * 10:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:50 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1198 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93684 and previous config saved to /var/cache/conftool/dbconfig/20260603-105006-fceratto.json * 10:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance * 10:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93683 and previous config saved to /var/cache/conftool/dbconfig/20260603-104939-fceratto.json * 10:45 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:45 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:44 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2196: Migration of db2196.codfw.wmnet completed * 10:44 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:41 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 10:40 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 10:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P93681 and previous config saved to /var/cache/conftool/dbconfig/20260603-103931-fceratto.json * 10:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1053: repool after upgrade * 10:37 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2196.codfw.wmnet with OS trixie * 10:36 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297090{{!}}hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] (duration: 12m 03s) * 10:32 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P93679 and previous config saved to /var/cache/conftool/dbconfig/20260603-102924-fceratto.json * 10:26 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297090{{!}}hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:24 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297090{{!}}hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] * 10:22 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be1067 * 10:21 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be1066 * 10:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2196.codfw.wmnet with reason: host reimage * 10:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93677 and previous config saved to /var/cache/conftool/dbconfig/20260603-101916-fceratto.json * 10:15 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2013.codfw.wmnet * 10:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2196.codfw.wmnet with reason: host reimage * 10:11 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93676 and previous config saved to /var/cache/conftool/dbconfig/20260603-101105-fceratto.json * 10:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1189.eqiad.wmnet with reason: Maintenance * 10:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93675 and previous config saved to /var/cache/conftool/dbconfig/20260603-101037-fceratto.json * 10:10 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2013.codfw.wmnet * 10:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P93673 and previous config saved to /var/cache/conftool/dbconfig/20260603-100029-fceratto.json * 09:59 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2196.codfw.wmnet with OS trixie * 09:57 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2196: Upgrading db2196.codfw.wmnet * 09:57 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2196: Upgrading db2196.codfw.wmnet * 09:57 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:52 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1053: repool after upgrade * 09:52 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:52 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:52 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:52 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:51 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:51 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:51 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P93670 and previous config saved to /var/cache/conftool/dbconfig/20260603-095022-fceratto.json * 09:49 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:49 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:48 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1053.eqiad.wmnet with OS trixie * 09:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:43 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2013.codfw.wmnet * 09:41 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on es1053.eqiad.wmnet with reason: host reimage * 09:41 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1053.eqiad.wmnet with reason: host reimage * 09:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93669 and previous config saved to /var/cache/conftool/dbconfig/20260603-094014-fceratto.json * 09:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2215: Migration of db2215.codfw.wmnet completed * 09:38 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2013.codfw.wmnet * 09:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93667 and previous config saved to /var/cache/conftool/dbconfig/20260603-093146-fceratto.json * 09:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance * 09:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93666 and previous config saved to /var/cache/conftool/dbconfig/20260603-093119-fceratto.json * 09:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1211: Migration of db1211.eqiad.wmnet completed * 09:27 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297069{{!}}hCaptcha: Collect risk score for blocked account creations (T427784)]] (duration: 07m 26s) * 09:25 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1053.eqiad.wmnet with OS trixie * 09:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add public1-b3-codfw gateway IPs - ayounsi@cumin1003" * 09:24 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add public1-b3-codfw gateway IPs - ayounsi@cumin1003" * 09:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1053: Upgrading es1053.eqiad.wmnet * 09:23 kharlan@deploy1003: kharlan: Continuing with deployment * 09:22 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1053: Upgrading es1053.eqiad.wmnet * 09:22 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:21 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297069{{!}}hCaptcha: Collect risk score for blocked account creations (T427784)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:21 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply * 09:21 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2054: repool after upgrade * 09:21 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply * 09:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P93661 and previous config saved to /var/cache/conftool/dbconfig/20260603-092111-fceratto.json * 09:20 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 09:20 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297069{{!}}hCaptcha: Collect risk score for blocked account creations (T427784)]] * 09:14 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297065{{!}}Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] (duration: 07m 06s) * 09:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P93659 and previous config saved to /var/cache/conftool/dbconfig/20260603-091104-fceratto.json * 09:10 kharlan@deploy1003: kharlan: Continuing with deployment * 09:09 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297065{{!}}Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:07 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297065{{!}}Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] * 09:06 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 09:06 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297064{{!}}Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 10m 54s) * 09:05 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 09:04 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 09:01 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003 - [[phab:T422043|T422043]]" * 09:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93656 and previous config saved to /var/cache/conftool/dbconfig/20260603-090056-fceratto.json * 09:00 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003 - [[phab:T422043|T422043]]" * 09:00 ayounsi@cumin1003: END (ERROR) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=97) generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003" * 09:00 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003" * 08:59 kharlan@deploy1003: kharlan: Continuing with deployment * 08:59 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297064{{!}}Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:55 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297064{{!}}Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] * 08:53 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296635{{!}}Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 11m 43s) * 08:52 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2215: Migration of db2215.codfw.wmnet completed * 08:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet * 08:52 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet * 08:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb[1022-1023].eqiad.wmnet * 08:51 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb[1022-1023].eqiad.wmnet * 08:50 kharlan@deploy1003: kharlan: Rolling back deployment * 08:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93652 and previous config saved to /var/cache/conftool/dbconfig/20260603-084846-fceratto.json * 08:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance * 08:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93651 and previous config saved to /var/cache/conftool/dbconfig/20260603-084819-fceratto.json * 08:47 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296635{{!}}Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2215.codfw.wmnet with OS trixie * 08:45 jiji@cumin1003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check docker-registry: maintenance * 08:45 jiji@cumin1003: START - Cookbook sre.discovery.service-route check docker-registry: maintenance * 08:43 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1211: Migration of db1211.eqiad.wmnet completed * 08:41 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296635{{!}}Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] * 08:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1211.eqiad.wmnet with OS trixie * 08:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93649 and previous config saved to /var/cache/conftool/dbconfig/20260603-083811-fceratto.json * 08:37 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296632{{!}}Image Browsing: add accessible labels to carousel elements (T407793)]] (duration: 32m 11s) * 08:36 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2054: repool after upgrade * 08:35 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool es2054.codfw.wmnet: After reimage * 08:35 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2054.codfw.wmnet: After reimage * 08:35 jiji@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:34 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 08:34 jiji@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:33 jiji@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:33 jiji@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:31 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:31 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:31 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2054.codfw.wmnet with OS trixie * 08:30 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:29 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 08:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2215.codfw.wmnet with reason: host reimage * 08:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93647 and previous config saved to /var/cache/conftool/dbconfig/20260603-082804-fceratto.json * 08:25 mszwarc@deploy1003: mlitn, mszwarc: Continuing with deployment * 08:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1211.eqiad.wmnet with reason: host reimage * 08:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1049: repool after upgrade * 08:22 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2215.codfw.wmnet with reason: host reimage * 08:22 mszwarc@deploy1003: mlitn, mszwarc: Backport for [[gerrit:1296632{{!}}Image Browsing: add accessible labels to carousel elements (T407793)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:18 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1211.eqiad.wmnet with reason: host reimage * 08:18 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 08:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93645 and previous config saved to /var/cache/conftool/dbconfig/20260603-081756-fceratto.json * 08:17 jiji@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 08:17 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 08:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 08:14 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2054.codfw.wmnet with reason: host reimage * 08:08 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2054.codfw.wmnet with reason: host reimage * 08:05 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296632{{!}}Image Browsing: add accessible labels to carousel elements (T407793)]] * {{safesubst:SAL entry|1=08:04 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296580{{!}}Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703{{!}}jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713{{!}}conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627{{!}}Add missing lazy img to carousel (T427821)]], [[gerrit:1295968{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T426799)]}} * 08:03 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93643 and previous config saved to /var/cache/conftool/dbconfig/20260603-080346-fceratto.json * 08:03 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1211.eqiad.wmnet with OS trixie * 08:03 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance * 08:03 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2215.codfw.wmnet with OS trixie * 08:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1211: Upgrading db1211.eqiad.wmnet * 08:02 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2215: Upgrading db2215.codfw.wmnet * 08:01 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:01 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1211: Upgrading db1211.eqiad.wmnet * 08:01 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2215: Upgrading db2215.codfw.wmnet * 08:01 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:01 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:01 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1157: Repooling * 08:01 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1157: Repooling * 08:00 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 07:57 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1022-1023].eqiad.wmnet with reason: Reimaging upstream server * 07:57 mszwarc@deploy1003: anzx, mlitn, mfossati, mszwarc: Continuing with deployment * 07:56 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Reimaging upstream server * {{safesubst:SAL entry|1=07:54 mszwarc@deploy1003: anzx, mlitn, mfossati, mszwarc: Backport for [[gerrit:1296580{{!}}Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703{{!}}jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713{{!}}conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627{{!}}Add missing lazy img to carousel (T427821)]], [[gerrit:1295968{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T42}} * 07:52 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2231: repool after maintenance * 07:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2054.codfw.wmnet with OS trixie * 07:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2054: Upgrading es2054.codfw.wmnet * 07:50 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2054: Upgrading es2054.codfw.wmnet * 07:50 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:50 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296580{{!}}Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703{{!}}jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713{{!}}conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627{{!}}Add missing lazy img to carousel (T427821)]], [[gerrit:1295968{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T426799)]] * 07:48 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296516{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] (duration: 32m 13s) * 07:44 marostegui@dns1004: END - running authdns-update * 07:43 marostegui@dns1004: START - running authdns-update * 07:42 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1056 to es2 eqiad primary [[phab:T427875|T427875]]', diff saved to https://phabricator.wikimedia.org/P93637 and previous config saved to /var/cache/conftool/dbconfig/20260603-074250-marostegui.json * 07:37 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1049: repool after upgrade * 07:37 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:35 mszwarc@deploy1003: mszwarc, stran: Continuing with deployment * 07:35 mszwarc@deploy1003: mszwarc, stran: Backport for [[gerrit:1296516{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:32 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1049.eqiad.wmnet with OS trixie * 07:16 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296516{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] * 07:14 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1049.eqiad.wmnet with reason: host reimage * 07:07 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1049.eqiad.wmnet with reason: host reimage * 07:07 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2231: repool after maintenance * 07:04 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 06:57 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2231.codfw.wmnet with OS trixie * 06:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1049.eqiad.wmnet with OS trixie * 06:46 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1049: Upgrading es1049.eqiad.wmnet * 06:46 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2056 to es2 codfw primary [[phab:T427875|T427875]]', diff saved to https://phabricator.wikimedia.org/P93632 and previous config saved to /var/cache/conftool/dbconfig/20260603-064623-marostegui.json * 06:45 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1049: Upgrading es1049.eqiad.wmnet * 06:45 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:44 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1056: repool after upgrade * 06:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2231.codfw.wmnet with reason: host reimage * 06:36 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2231.codfw.wmnet with reason: host reimage * 06:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2231.codfw.wmnet with OS trixie * 06:09 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2231: Upgrading db2231.codfw.wmnet * 06:09 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2231: Upgrading db2231.codfw.wmnet * 06:09 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:59 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1056: repool after upgrade * 05:59 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 05:55 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1056.eqiad.wmnet with OS trixie * 05:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1056.eqiad.wmnet with reason: host reimage * 05:33 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1056.eqiad.wmnet with reason: host reimage * 05:18 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1056.eqiad.wmnet with OS trixie * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1056: Upgrading es1056.eqiad.wmnet * 05:17 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1056: Upgrading es1056.eqiad.wmnet * 05:16 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade == 2026-06-02 == * 22:21 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296689{{!}}hCaptcha: Correct inaccurate comment]] (duration: 06m 27s) * 22:18 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 22:18 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 22:17 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 22:17 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296689{{!}}hCaptcha: Correct inaccurate comment]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:15 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296689{{!}}hCaptcha: Correct inaccurate comment]] * 22:13 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296551{{!}}hCaptcha: Enable for badlogin on group0 wikis (T426875)]] (duration: 08m 31s) * 22:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 22:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 22:09 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 22:07 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296551{{!}}hCaptcha: Enable for badlogin on group0 wikis (T426875)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:05 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296551{{!}}hCaptcha: Enable for badlogin on group0 wikis (T426875)]] * 20:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93621 and previous config saved to /var/cache/conftool/dbconfig/20260602-203945-fceratto.json * 20:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93620 and previous config saved to /var/cache/conftool/dbconfig/20260602-202937-fceratto.json * 20:27 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1054.eqiad.wmnet * 20:27 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:27 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1054.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:26 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1054.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:20 jiji@cumin1003: START - Cookbook sre.dns.netbox * 20:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93619 and previous config saved to /var/cache/conftool/dbconfig/20260602-201929-fceratto.json * 20:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93618 and previous config saved to /var/cache/conftool/dbconfig/20260602-200922-fceratto.json * 20:03 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1054.eqiad.wmnet * 19:48 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1053.eqiad.wmnet * 19:48 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:48 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1053.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 19:37 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1053.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 19:09 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93617 and previous config saved to /var/cache/conftool/dbconfig/20260602-190907-fceratto.json * 19:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance * 19:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93616 and previous config saved to /var/cache/conftool/dbconfig/20260602-190811-fceratto.json * 19:05 dancy@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.5 refs [[phab:T423914|T423914]] * 18:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P93615 and previous config saved to /var/cache/conftool/dbconfig/20260602-185804-fceratto.json * 18:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P93614 and previous config saved to /var/cache/conftool/dbconfig/20260602-184757-fceratto.json * 18:38 jiji@cumin1003: START - Cookbook sre.dns.netbox * 18:38 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:38 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93612 and previous config saved to /var/cache/conftool/dbconfig/20260602-183749-fceratto.json * 18:37 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:37 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:33 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1053.eqiad.wmnet * 18:30 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1259 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93611 and previous config saved to /var/cache/conftool/dbconfig/20260602-183023-fceratto.json * 18:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance * 18:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93610 and previous config saved to /var/cache/conftool/dbconfig/20260602-182956-fceratto.json * 18:27 mutante: gerrit delete unused plugin projects: barricade, WikimediaBlocks and WikimediaWebSessions * 18:26 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1052.eqiad.wmnet * 18:26 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:26 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1052.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 18:25 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1052.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 18:25 dancy: Train is blocked at testwikis on https://phabricator.wikimedia.org/T427935 * 18:21 Daimona: Running query from [[phab:T427962|T427962]]#11978299 in x1.wikishared * 18:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P93609 and previous config saved to /var/cache/conftool/dbconfig/20260602-181949-fceratto.json * 18:16 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296615{{!}}feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614{{!}}feat(cleanMentorList): Add a feature flag (T427386)]] (duration: 34m 09s) * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 18:12 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 18:12 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 18:12 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 18:10 jiji@cumin1003: START - Cookbook sre.dns.netbox * 18:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P93608 and previous config saved to /var/cache/conftool/dbconfig/20260602-180941-fceratto.json * 18:08 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 18:07 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 18:06 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 18:06 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 18:05 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:05 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:05 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 18:05 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 18:04 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 18:02 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 18:02 swfrench-wmf: reverting shellbox to 2026-05-20-192555 due to errors in shellbox-syntaxhighlight * 18:02 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 18:01 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 18:01 urbanecm@deploy1003: urbanecm: Continuing with deployment * 18:01 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1296615{{!}}feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614{{!}}feat(cleanMentorList): Add a feature flag (T427386)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:00 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1052.eqiad.wmnet * 17:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93607 and previous config saved to /var/cache/conftool/dbconfig/20260602-175933-fceratto.json * 17:58 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:57 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:56 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1051.eqiad.wmnet * 17:56 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:56 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1051.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:55 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1051.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:53 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93605 and previous config saved to /var/cache/conftool/dbconfig/20260602-175227-fceratto.json * 17:52 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance * 17:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93604 and previous config saved to /var/cache/conftool/dbconfig/20260602-175157-fceratto.json * 17:51 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:51 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:50 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:50 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:50 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:49 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:49 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:48 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:48 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:47 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:44 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:42 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:42 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:42 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P93603 and previous config saved to /var/cache/conftool/dbconfig/20260602-174150-fceratto.json * 17:41 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1296615{{!}}feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614{{!}}feat(cleanMentorList): Add a feature flag (T427386)]] * 17:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P93602 and previous config saved to /var/cache/conftool/dbconfig/20260602-173143-fceratto.json * 17:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93601 and previous config saved to /var/cache/conftool/dbconfig/20260602-172135-fceratto.json * 17:14 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93600 and previous config saved to /var/cache/conftool/dbconfig/20260602-171422-fceratto.json * 17:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance * 17:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93599 and previous config saved to /var/cache/conftool/dbconfig/20260602-171354-fceratto.json * 17:04 jiji@cumin1003: START - Cookbook sre.dns.netbox * 17:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P93598 and previous config saved to /var/cache/conftool/dbconfig/20260602-170344-fceratto.json * 16:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P93597 and previous config saved to /var/cache/conftool/dbconfig/20260602-165336-fceratto.json * 16:49 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1051.eqiad.wmnet * 16:48 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1050.eqiad.wmnet * 16:48 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:48 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1050.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:47 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1050.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93596 and previous config saved to /var/cache/conftool/dbconfig/20260602-164328-fceratto.json * 16:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93595 and previous config saved to /var/cache/conftool/dbconfig/20260602-163622-fceratto.json * 16:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance * 16:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93594 and previous config saved to /var/cache/conftool/dbconfig/20260602-163550-fceratto.json * 16:34 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:34 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1072.eqiad.wmnet with OS trixie * 16:30 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:29 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:27 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2006.codfw.wmnet with OS trixie * 16:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P93593 and previous config saved to /var/cache/conftool/dbconfig/20260602-162542-fceratto.json * 16:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P93591 and previous config saved to /var/cache/conftool/dbconfig/20260602-161534-fceratto.json * 16:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1072.eqiad.wmnet with reason: host reimage * 16:10 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1071.eqiad.wmnet with OS trixie * 16:10 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296624{{!}}Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 06m 40s) * 16:09 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2006.codfw.wmnet with reason: host reimage * 16:05 kharlan@deploy1003: kharlan: Continuing with deployment * 16:05 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1072.eqiad.wmnet with reason: host reimage * 16:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93590 and previous config saved to /var/cache/conftool/dbconfig/20260602-160527-fceratto.json * 16:05 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2006.codfw.wmnet with reason: host reimage * 16:05 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296624{{!}}Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:03 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296624{{!}}Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] * 15:59 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295909{{!}}hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] (duration: 09m 48s) * 15:59 kharlan@deploy1003: kharlan: Rolling back deployment * 15:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93589 and previous config saved to /var/cache/conftool/dbconfig/20260602-155817-fceratto.json * 15:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance * 15:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93588 and previous config saved to /var/cache/conftool/dbconfig/20260602-155749-fceratto.json * 15:54 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1071.eqiad.wmnet with reason: host reimage * 15:53 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1072.eqiad.wmnet with OS trixie * 15:51 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1070.eqiad.wmnet with OS trixie * 15:51 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295909{{!}}hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:50 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1071.eqiad.wmnet with reason: host reimage * 15:49 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295909{{!}}hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] * 15:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P93587 and previous config saved to /var/cache/conftool/dbconfig/20260602-154742-fceratto.json * 15:47 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296558{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] (duration: 07m 24s) * 15:43 kharlan@deploy1003: kharlan: Continuing with deployment * 15:42 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296558{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:40 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296558{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] * 15:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P93586 and previous config saved to /var/cache/conftool/dbconfig/20260602-153734-fceratto.json * 15:37 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1071.eqiad.wmnet with OS trixie * 15:36 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1069.eqiad.wmnet with OS trixie * 15:35 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1070.eqiad.wmnet with reason: host reimage * 15:32 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:32 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:31 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1070.eqiad.wmnet with reason: host reimage * 15:30 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:29 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93585 and previous config saved to /var/cache/conftool/dbconfig/20260602-152726-fceratto.json * 15:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2158: Repooling * {{safesubst:SAL entry|1=15:22 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295502{{!}}Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106{{!}}Remove unused 'writeapi' right]], [[gerrit:1296566{{!}}Clean up bot password configuration]], [[gerrit:1296563{{!}}Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574{{!}}cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582{{!}}U}} * 15:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1069.eqiad.wmnet with reason: host reimage * 15:20 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93583 and previous config saved to /var/cache/conftool/dbconfig/20260602-152026-fceratto.json * 15:20 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance * 15:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93582 and previous config saved to /var/cache/conftool/dbconfig/20260602-151958-fceratto.json * 15:19 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:19 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:18 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1070.eqiad.wmnet with OS trixie * 15:18 dreamyjazz@deploy1003: matmarex, anzx, dreamyjazz: Continuing with deployment * 15:18 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 15:17 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:17 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:15 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1069.eqiad.wmnet with reason: host reimage * {{safesubst:SAL entry|1=15:15 dreamyjazz@deploy1003: matmarex, anzx, dreamyjazz: Backport for [[gerrit:1295502{{!}}Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106{{!}}Remove unused 'writeapi' right]], [[gerrit:1296566{{!}}Clean up bot password configuration]], [[gerrit:1296563{{!}}Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574{{!}}cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582}} * 15:14 jiji@cumin1003: START - Cookbook sre.dns.netbox * {{safesubst:SAL entry|1=15:13 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1295502{{!}}Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106{{!}}Remove unused 'writeapi' right]], [[gerrit:1296566{{!}}Clean up bot password configuration]], [[gerrit:1296563{{!}}Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574{{!}}cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582{{!}}Us}} * 15:12 jayme@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-main2006.codfw.wmnet with OS trixie * 15:12 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1068.eqiad.wmnet with OS trixie * 15:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P93580 and previous config saved to /var/cache/conftool/dbconfig/20260602-150951-fceratto.json * 15:09 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296514{{!}}[Growth] Set wgGEMentorshipCleanupEnabled to false on all wikis (T427386)]] (duration: 06m 22s) * 15:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1167: Repooling after Icing wait-for-green timeout * 15:06 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1050.eqiad.wmnet * 15:06 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1049.eqiad.wmnet * 15:06 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:06 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1049.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:05 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1049.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:02 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1296514{{!}}[Growth] Set wgGEMentorshipCleanupEnabled to false on all wikis (T427386)]] * 15:02 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1069.eqiad.wmnet with OS trixie * 15:01 jiji@cumin1003: START - Cookbook sre.dns.netbox * 14:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P93578 and previous config saved to /var/cache/conftool/dbconfig/20260602-145943-fceratto.json * 14:54 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage * 14:52 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:52 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:52 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1049.eqiad.wmnet * 14:51 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1067.eqiad.wmnet with OS trixie * 14:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:50 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage * 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93575 and previous config saved to /var/cache/conftool/dbconfig/20260602-144935-fceratto.json * 14:42 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for pc2021.codfw.wmnet * 14:42 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for pc2021.codfw.wmnet * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2250.codfw.wmnet * 14:41 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2250.codfw.wmnet * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2158.codfw.wmnet * 14:41 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2158.codfw.wmnet * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc2021: Repooling * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 14:41 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache * 14:41 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc2021: Repooling * 14:41 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93573 and previous config saved to /var/cache/conftool/dbconfig/20260602-144110-fceratto.json * 14:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance * 14:41 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2158: Repooling * 14:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93571 and previous config saved to /var/cache/conftool/dbconfig/20260602-144043-fceratto.json * 14:38 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:38 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:38 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:37 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1048.eqiad.wmnet * 14:37 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:37 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1048.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 14:37 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1068.eqiad.wmnet with OS trixie * 14:36 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1066.eqiad.wmnet with OS trixie * 14:34 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage * 14:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P93569 and previous config saved to /var/cache/conftool/dbconfig/20260602-143035-fceratto.json * 14:30 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage * 14:25 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1048.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 14:21 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1167: Repooling after Icing wait-for-green timeout * 14:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage * 14:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P93566 and previous config saved to /var/cache/conftool/dbconfig/20260602-142027-fceratto.json * 14:17 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1067.eqiad.wmnet with OS trixie * 14:17 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 14:17 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1167.eqiad.wmnet * 14:17 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1167.eqiad.wmnet * 14:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1065.eqiad.wmnet with OS trixie * 14:15 jayme@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2006.codfw.wmnet with OS trixie * 14:14 jiji@cumin1003: START - Cookbook sre.dns.netbox * 14:13 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage * 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93564 and previous config saved to /var/cache/conftool/dbconfig/20260602-141019-fceratto.json * 14:09 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments userOptions.php --delete --nowarn growthexperiments-homepage-variant # [[phab:T417621|T417621]] * 14:09 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1048.eqiad.wmnet * 14:08 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments userOptions.php --delete growthexperiments-homepage-variant # [[phab:T417621|T417621]] * 14:05 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 14:01 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93563 and previous config saved to /var/cache/conftool/dbconfig/20260602-140140-fceratto.json * 14:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 14:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1156.eqiad.wmnet with reason: Maintenance * 14:01 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1066.eqiad.wmnet with OS trixie * 14:00 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage * 14:00 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 14:00 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 14:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93562 and previous config saved to /var/cache/conftool/dbconfig/20260602-140022-fceratto.json * 14:00 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1064.eqiad.wmnet with OS trixie * 13:56 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage * 13:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1167.eqiad.wmnet with OS trixie * 13:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P93561 and previous config saved to /var/cache/conftool/dbconfig/20260602-135015-fceratto.json * 13:47 topranks: revert all config to normal on cr1-codfw and ssw1-a1-codfw * 13:43 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1065.eqiad.wmnet with OS trixie * 13:42 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1064.eqiad.wmnet with reason: host reimage * 13:40 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1063.eqiad.wmnet with OS trixie * 13:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P93560 and previous config saved to /var/cache/conftool/dbconfig/20260602-134007-fceratto.json * 13:38 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1167.eqiad.wmnet with reason: host reimage * 13:35 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs1002.eqiad.wmnet with OS trixie * 13:35 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs1003.eqiad.wmnet with OS trixie * 13:34 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:34 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:32 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1064.eqiad.wmnet with reason: host reimage * 13:31 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1167.eqiad.wmnet with reason: host reimage * 13:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93559 and previous config saved to /var/cache/conftool/dbconfig/20260602-132959-fceratto.json * 13:27 slyngshede@dns1004: END - running authdns-update * 13:25 slyngshede@dns1004: START - running authdns-update * 13:24 topranks: increase OSPF cost on ssw1-a1-codfw et-0/0/4 towards lsw1-a5-codfw [[phab:T427301|T427301]] * 13:23 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1063.eqiad.wmnet with reason: host reimage * 13:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93558 and previous config saved to /var/cache/conftool/dbconfig/20260602-132314-fceratto.json * 13:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1210.eqiad.wmnet with reason: Maintenance * 13:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93557 and previous config saved to /var/cache/conftool/dbconfig/20260602-132246-fceratto.json * 13:20 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1064.eqiad.wmnet with OS trixie * 13:19 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 13:19 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1062.eqiad.wmnet with OS trixie * 13:18 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1063.eqiad.wmnet with reason: host reimage * 13:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2049: repool after upgrade * 13:17 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 13:16 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1167.eqiad.wmnet with OS trixie * 13:15 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 13:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1167: Upgrading db1167.eqiad.wmnet * 13:13 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1167: Upgrading db1167.eqiad.wmnet * 13:13 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:12 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 13:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P93554 and previous config saved to /var/cache/conftool/dbconfig/20260602-131238-fceratto.json * 13:12 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 13:12 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 13:11 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 13:07 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1003.eqiad.wmnet with OS trixie * 13:07 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1002.eqiad.wmnet with OS trixie * 13:06 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1063.eqiad.wmnet with OS trixie * 13:04 jayme@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-main2006.codfw.wmnet with OS trixie * 13:04 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:04 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:03 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1022-1023].eqiad.wmnet with reason: Reimaging upstream servers * 13:03 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1001.eqiad.wmnet with OS trixie * 13:03 topranks: increase OSPF cost on ssw1-a1-codfw et-0/0/2 towards lsw1-a3-codfw [[phab:T427301|T427301]] * 13:03 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1062.eqiad.wmnet with reason: host reimage * 13:02 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Reimaging upstream servers * 13:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P93553 and previous config saved to /var/cache/conftool/dbconfig/20260602-130230-fceratto.json * 12:59 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1062.eqiad.wmnet with reason: host reimage * 12:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:57 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:57 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2161: Migration of db2161.codfw.wmnet completed * 12:54 topranks: shutdown sub-interfaces on cr1-codfw et-1/1/5 for row A/B vlans [[phab:T427301|T427301]] * 12:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 12:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93550 and previous config saved to /var/cache/conftool/dbconfig/20260602-125223-fceratto.json * 12:50 topranks: enable bgp graceful-shutdown in overlay on ssw1-a1-codfw [[phab:T427301|T427301]] * 12:49 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc1061.eqiad.wmnet with OS trixie * 12:48 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt * 12:48 ayounsi@cumin1003: START - Cookbook sre.hosts.remove-downtime for lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt * 12:47 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1062.eqiad.wmnet with OS trixie * 12:45 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93548 and previous config saved to /var/cache/conftool/dbconfig/20260602-124541-fceratto.json * 12:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance * 12:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93547 and previous config saved to /var/cache/conftool/dbconfig/20260602-124512-fceratto.json * 12:43 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc1060.eqiad.wmnet with OS trixie * 12:42 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 12:42 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mc1061.eqiad.wmnet with reason: host reimage * 12:42 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1061.eqiad.wmnet with reason: host reimage * 12:41 topranks: enable bgp graceful-shutdown in underlay on ssw1-a1-codfw [[phab:T427301|T427301]] * 12:35 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mc1060.eqiad.wmnet with reason: host reimage * 12:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P93545 and previous config saved to /var/cache/conftool/dbconfig/20260602-123505-fceratto.json * 12:33 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 12:33 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1060.eqiad.wmnet with reason: host reimage * 12:31 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2049: repool after upgrade * 12:31 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 12:29 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1061.eqiad.wmnet with OS trixie * 12:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2049.codfw.wmnet with OS trixie * 12:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P93542 and previous config saved to /var/cache/conftool/dbconfig/20260602-122459-fceratto.json * 12:24 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1059.eqiad.wmnet with OS trixie * 12:21 XioNoX: reboot lsw1-a3-codfw for software upgrade - [[phab:T427301|T427301]] * 12:20 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1060.eqiad.wmnet with OS trixie * 12:20 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 12:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1058.eqiad.wmnet with OS trixie * 12:17 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 12:16 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296532{{!}}hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533{{!}}hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] (duration: 09m 02s) * 12:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93539 and previous config saved to /var/cache/conftool/dbconfig/20260602-121451-fceratto.json * 12:11 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 12:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2049.codfw.wmnet with reason: host reimage * 12:11 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt with reason: Switch maintenance * 12:10 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2161: Migration of db2161.codfw.wmnet completed * 12:09 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Switch maintenance * 12:09 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296532{{!}}hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533{{!}}hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:08 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1200 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93537 and previous config saved to /var/cache/conftool/dbconfig/20260602-120755-fceratto.json * 12:07 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1059.eqiad.wmnet with reason: host reimage * 12:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance * 12:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93536 and previous config saved to /var/cache/conftool/dbconfig/20260602-120728-fceratto.json * 12:07 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 12:07 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296532{{!}}hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533{{!}}hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] * 12:05 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2049.codfw.wmnet with reason: host reimage * 12:04 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1058.eqiad.wmnet with reason: host reimage * 12:02 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1059.eqiad.wmnet with reason: host reimage * 12:01 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2161.codfw.wmnet with OS trixie * 12:00 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1058.eqiad.wmnet with reason: host reimage * 11:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P93535 and previous config saved to /var/cache/conftool/dbconfig/20260602-115721-fceratto.json * 11:55 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 11:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:55 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 11:53 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 11:53 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 11:53 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:50 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1059.eqiad.wmnet with OS trixie * 11:49 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1057.eqiad.wmnet with OS trixie * 11:49 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2049.codfw.wmnet with OS trixie * 11:48 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2049: Upgrading es2049.codfw.wmnet * 11:48 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2049: Upgrading es2049.codfw.wmnet * 11:47 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:47 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1058.eqiad.wmnet with OS trixie * 11:47 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2056: repool after upgrade * 11:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P93532 and previous config saved to /var/cache/conftool/dbconfig/20260602-114713-fceratto.json * 11:45 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1056.eqiad.wmnet with OS trixie * 11:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2161.codfw.wmnet with reason: host reimage * 11:40 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2161.codfw.wmnet with reason: host reimage * 11:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93531 and previous config saved to /var/cache/conftool/dbconfig/20260602-113705-fceratto.json * 11:33 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1057.eqiad.wmnet with reason: host reimage * 11:30 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1185 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93529 and previous config saved to /var/cache/conftool/dbconfig/20260602-113019-fceratto.json * 11:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance * 11:29 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1056.eqiad.wmnet with reason: host reimage * 11:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1161: Repooling * 11:26 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1161: Repooling * 11:23 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2161.codfw.wmnet with OS trixie * 11:22 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1057.eqiad.wmnet with reason: host reimage * 11:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2161: Upgrading db2161.codfw.wmnet * 11:21 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2161: Upgrading db2161.codfw.wmnet * 11:21 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1056.eqiad.wmnet with reason: host reimage * 11:21 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P93527 and previous config saved to /var/cache/conftool/dbconfig/20260602-111954-fceratto.json * 11:15 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2161 [[phab:T427892|T427892]]', diff saved to https://phabricator.wikimedia.org/P93525 and previous config saved to /var/cache/conftool/dbconfig/20260602-111511-cwilliams.json * 11:12 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2165 to s8 primary [[phab:T427892|T427892]]', diff saved to https://phabricator.wikimedia.org/P93524 and previous config saved to /var/cache/conftool/dbconfig/20260602-111200-cwilliams.json * 11:10 cezmunsta: Starting s8 codfw failover from db2161 to db2165 - [[phab:T427892|T427892]] * 11:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P93523 and previous config saved to /var/cache/conftool/dbconfig/20260602-110947-fceratto.json * 11:09 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1057.eqiad.wmnet with OS trixie * 11:09 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1056.eqiad.wmnet with OS trixie * 11:04 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2165 with weight 0 [[phab:T427892|T427892]]', diff saved to https://phabricator.wikimedia.org/P93522 and previous config saved to /var/cache/conftool/dbconfig/20260602-110420-cwilliams.json * 11:03 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s8 [[phab:T427892|T427892]] * 11:02 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2056: repool after upgrade * 11:01 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93520 and previous config saved to /var/cache/conftool/dbconfig/20260602-105939-fceratto.json * 10:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93519 and previous config saved to /var/cache/conftool/dbconfig/20260602-105239-fceratto.json * 10:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 10:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance * 10:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93518 and previous config saved to /var/cache/conftool/dbconfig/20260602-105202-fceratto.json * 10:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2056.codfw.wmnet with OS trixie * 10:42 moritzm: installing busybox security updates * 10:42 claime: Enabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 10:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P93517 and previous config saved to /var/cache/conftool/dbconfig/20260602-104154-fceratto.json * 10:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P93516 and previous config saved to /var/cache/conftool/dbconfig/20260602-103146-fceratto.json * 10:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2056.codfw.wmnet with reason: host reimage * 10:27 claime: Disabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 10:25 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2056.codfw.wmnet with reason: host reimage * 10:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93515 and previous config saved to /var/cache/conftool/dbconfig/20260602-102139-fceratto.json * 10:09 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2056.codfw.wmnet with OS trixie * 10:08 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2056: Upgrading es2056.codfw.wmnet * 10:08 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2056: Upgrading es2056.codfw.wmnet * 10:08 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:06 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply * 10:06 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply * 09:56 claime: Enabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 09:46 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on cumin2003.codfw.wmnet with reason: in setup * 09:45 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1187: Pooling * 09:37 claime: Running puppet on cp6010 and cp6011 - [[phab:T422937|T422937]] * 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow2004.codfw.wmnet to plain * 09:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93511 and previous config saved to /var/cache/conftool/dbconfig/20260602-093716-fceratto.json * 09:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance * 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow2004.codfw.wmnet to plain * 09:34 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of rpki2003.codfw.wmnet to plain * 09:34 claime: Disabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 09:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of rpki2003.codfw.wmnet to plain * 09:32 moritzm: temporarily remove ganeti2045 from the codfw cluster [[phab:T427357|T427357]] * 09:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1055.eqiad.wmnet with OS trixie * 09:15 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1187: Pooling * 09:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1055.eqiad.wmnet with reason: host reimage * 09:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1187 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93508 and previous config saved to /var/cache/conftool/dbconfig/20260602-091126-fceratto.json * 09:09 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1055.eqiad.wmnet with reason: host reimage * 09:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1187 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93506 and previous config saved to /var/cache/conftool/dbconfig/20260602-090432-fceratto.json * 09:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1187.eqiad.wmnet with reason: Maintenance * 08:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2250.codfw.wmnet with reason: rack A3 maintenance * 08:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:56 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1055.eqiad.wmnet with OS trixie * 08:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:54 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:54 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:53 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 08:52 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 08:51 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 08:50 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 08:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 08:41 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:39 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:37 urbanecm: Reset user email of Barras@votewiki to the one of Barras@SUL * 08:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance * 08:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93505 and previous config saved to /var/cache/conftool/dbconfig/20260602-083033-fceratto.json * 08:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:29 slyngs: IDP, new configuration in preparation for webauthn * 08:20 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P93504 and previous config saved to /var/cache/conftool/dbconfig/20260602-082026-fceratto.json * 08:19 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 08:18 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 08:18 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:17 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296488{{!}}Revert "translate: adding separate read/write endpoints" (T425377)]] (duration: 03m 33s) * 08:16 atsuko@deploy1003: atsuko: Rolling back deployment * 08:16 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2053: repool after upgrade * 08:15 atsuko@deploy1003: atsuko: Backport for [[gerrit:1296488{{!}}Revert "translate: adding separate read/write endpoints" (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:13 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1296488{{!}}Revert "translate: adding separate read/write endpoints" (T425377)]] * 08:11 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:10 marostegui: Install mariadb 10.11.17 on es2053 [[phab:T427345|T427345]] * 08:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P93502 and previous config saved to /var/cache/conftool/dbconfig/20260602-081018-fceratto.json * 08:09 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:09 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: Depool for rack maintenance * 08:03 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296262{{!}}translate: fixing missed variable in credentials formatting closure (T425377)]] (duration: 14m 47s) * 08:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93499 and previous config saved to /var/cache/conftool/dbconfig/20260602-080011-fceratto.json * 07:59 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 07:59 atsuko@deploy1003: atsuko: Rolling back deployment * 07:58 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 07:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1181 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93498 and previous config saved to /var/cache/conftool/dbconfig/20260602-075759-fceratto.json * 07:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1181.eqiad.wmnet with reason: Maintenance * 07:57 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling * 07:50 atsuko@deploy1003: atsuko: Backport for [[gerrit:1296262{{!}}translate: fixing missed variable in credentials formatting closure (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:49 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1296262{{!}}translate: fixing missed variable in credentials formatting closure (T425377)]] * 07:48 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1181: Pooling * 07:47 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1181: Pooling * 07:44 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1181: Reboot * 07:43 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1181: Reboot * 07:42 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1181.eqiad.wmnet with reason: Reboot * 07:41 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 07:41 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:41 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1181: Migration of db1181.eqiad.wmnet completed * 07:40 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294949{{!}}translate: adding separate read/write endpoints (T425377)]] (duration: 21m 01s) * 07:39 atsuko@deploy1003: atsuko: Rolling back deployment * 07:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93490 and previous config saved to /var/cache/conftool/dbconfig/20260602-073904-fceratto.json * 07:32 XioNoX: pfw1-eqiad# delete protocols bgp group Production family inet6 - [[phab:T423384|T423384]] * 07:30 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2053: repool after upgrade * 07:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2158.codfw.wmnet with reason: rack A3 maintenance * 07:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93487 and previous config saved to /var/cache/conftool/dbconfig/20260602-072856-fceratto.json * 07:28 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2158: rack A3 maintenance * 07:28 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2158: rack A3 maintenance * 07:27 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on pc2021.codfw.wmnet with reason: rack A3 maintenance * 07:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc2021: rack A3 maintenance * 07:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 07:25 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache * 07:25 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool pc2021: rack A3 maintenance * 07:23 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2241: Depool for rack maintenance * 07:23 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2241.codfw.wmnet * 07:23 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2241.codfw.wmnet * 07:21 atsuko@deploy1003: atsuko: Backport for [[gerrit:1294949{{!}}translate: adding separate read/write endpoints (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:20 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2053.codfw.wmnet with OS trixie * 07:19 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1294949{{!}}translate: adding separate read/write endpoints (T425377)]] * 07:15 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2241.codfw.wmnet with reason: Depool for rack maintenance * 07:14 marostegui: Install mariadb 10.11.17 on db2186 [[phab:T427345|T427345]] * 07:12 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: Depool for rack maintenance * 07:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2186.codfw.wmnet with reason: upgrade * 07:12 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2241: Depool for rack maintenance * 07:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2053.codfw.wmnet with reason: host reimage * 06:59 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2053.codfw.wmnet with reason: host reimage * 06:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93478 and previous config saved to /var/cache/conftool/dbconfig/20260602-065533-fceratto.json * 06:55 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1181: Migration of db1181.eqiad.wmnet completed * 06:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 06:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1181.eqiad.wmnet with OS trixie * 06:43 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2053.codfw.wmnet with OS trixie * 06:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2053: Upgrading es2053.codfw.wmnet * 06:41 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2053: Upgrading es2053.codfw.wmnet * 06:41 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:37 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 06:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 06:36 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 06:36 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1052: repool after upgrade * 06:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1181.eqiad.wmnet with reason: host reimage * 06:24 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1181.eqiad.wmnet with reason: host reimage * 06:22 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 06:21 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 06:16 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 06:15 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 06:08 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1181.eqiad.wmnet with OS trixie * 06:05 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1181: Upgrading db1181.eqiad.wmnet * 06:05 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1181: Upgrading db1181.eqiad.wmnet * 06:04 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:02 marostegui@dns1004: END - running authdns-update * 06:01 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1181 [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93473 and previous config saved to /var/cache/conftool/dbconfig/20260602-060157-marostegui.json * 06:01 marostegui@dns1004: START - running authdns-update * 06:00 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db1236 to s7 primary and set section read-write [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93472 and previous config saved to /var/cache/conftool/dbconfig/20260602-060041-marostegui.json * 06:00 marostegui@cumin1003: dbctl commit (dc=all): 'Set s7 eqiad as read-only for maintenance - [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93471 and previous config saved to /var/cache/conftool/dbconfig/20260602-060018-marostegui.json * 06:00 marostegui: Starting s7 eqiad failover from db1181 to db1236 - [[phab:T426088|T426088]] * 05:51 marostegui@cumin1003: dbctl commit (dc=all): 'Set db1236 with weight 0 [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93470 and previous config saved to /var/cache/conftool/dbconfig/20260602-055153-marostegui.json * 05:51 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 [[phab:T426088|T426088]] * 05:50 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1052: repool after upgrade * 05:50 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 05:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1052.eqiad.wmnet with OS trixie * 05:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1052.eqiad.wmnet with reason: host reimage * 05:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:22 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1052.eqiad.wmnet with reason: host reimage * 05:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:07 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1052.eqiad.wmnet with OS trixie * 05:06 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1052: Upgrading es1052.eqiad.wmnet * 05:06 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1052: Upgrading es1052.eqiad.wmnet * 05:05 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 04:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 04:49 ryankemper: [[phab:T425007|T425007]] (k8s) created 4 wdqs namespaces on `dse-k8s-codfw`'s `admin_ng` ns: `wdqs-[internal,external]` & `wdqs-[internal,external]-next`; certs issued * 04:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 04:40 ryankemper@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 04:36 ryankemper@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 04:05 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.2 (duration: 05m 33s) == 2026-06-01 == * 23:27 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295963{{!}}Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962{{!}}Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] (duration: 07m 17s) * 23:23 jdlrobson@deploy1003: mfossati, jdlrobson: Continuing with deployment * 23:22 jdlrobson@deploy1003: mfossati, jdlrobson: Backport for [[gerrit:1295963{{!}}Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962{{!}}Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:20 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1295963{{!}}Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962{{!}}Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] * 23:15 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296022{{!}}Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028{{!}}styles: Limit selector to badge client pref (T427407)]] (duration: 09m 33s) * 23:11 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 23:07 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1296022{{!}}Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028{{!}}styles: Limit selector to badge client pref (T427407)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:06 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1296022{{!}}Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028{{!}}styles: Limit selector to badge client pref (T427407)]] * 23:04 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6015.* * 22:36 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296024{{!}}Add maintenance script to scrape SVG render files]] (duration: 06m 22s) * 22:32 reedy@deploy1003: reedy: Continuing with deployment * 22:31 reedy@deploy1003: reedy: Backport for [[gerrit:1296024{{!}}Add maintenance script to scrape SVG render files]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:30 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1296024{{!}}Add maintenance script to scrape SVG render files]] * 22:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 22:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 22:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 21:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 21:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 21:51 sbassett: Deployed updated mitigation for [[phab:T326691|T326691]] * 21:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 21:35 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 21:35 maryum: Deployed security fix for [[phab:T427611|T427611]] * 21:35 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 21:33 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 21:32 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 21:27 maryum: Deployed security fix for [[phab:T427235|T427235]] * 21:13 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296002{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009{{!}}Redirect Special:AccountRecovery to the shared domain (T427692)]] (duration: 09m 20s) * 21:09 catrope@deploy1003: catrope, arlolra: Continuing with deployment * 21:09 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 21:09 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 21:08 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 21:07 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 21:07 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 21:06 catrope@deploy1003: catrope, arlolra: Backport for [[gerrit:1296002{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009{{!}}Redirect Special:AccountRecovery to the shared domain (T427692)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:04 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1296002{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009{{!}}Redirect Special:AccountRecovery to the shared domain (T427692)]] * 20:53 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 20:37 ryankemper@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wdqs1015.eqiad.wmnet with reason: [[phab:T427852|T427852]] hw failure * 20:26 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1285412{{!}}Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531{{!}}Enable AbuseFilter block action on nlwiki (T427384)]] (duration: 07m 48s) * 20:22 catrope@deploy1003: sfaci, xxblackburnxx, catrope: Continuing with deployment * 20:20 catrope@deploy1003: sfaci, xxblackburnxx, catrope: Backport for [[gerrit:1285412{{!}}Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531{{!}}Enable AbuseFilter block action on nlwiki (T427384)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:18 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1285412{{!}}Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531{{!}}Enable AbuseFilter block action on nlwiki (T427384)]] * 20:12 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295504{{!}}passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] (duration: 07m 37s) * 20:08 catrope@deploy1003: catrope: Continuing with deployment * 20:07 catrope@deploy1003: catrope: Backport for [[gerrit:1295504{{!}}passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:05 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1295504{{!}}passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] * 19:48 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 19:47 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 19:47 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 19:46 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 19:46 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 19:45 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 19:01 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: sync * 19:00 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: sync * 18:24 otto@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295950{{!}}mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] (duration: 06m 42s) * 18:20 otto@deploy1003: otto: Continuing with deployment * 18:19 otto@deploy1003: otto: Backport for [[gerrit:1295950{{!}}mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:17 otto@deploy1003: Started scap sync-world: Backport for [[gerrit:1295950{{!}}mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] * 18:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 18:05 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 18:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd2001.codfw.wmnet to plain * 18:02 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 18:02 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd2001.codfw.wmnet to plain * 18:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2003.codfw.wmnet to plain * 18:01 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply * 18:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2003.codfw.wmnet to plain * 17:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 17:58 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 17:53 jasmine@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2006.codfw.wmnet with OS trixie * 17:42 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295976{{!}}nlwiki: change to Wikipedia 25 logo (T424519)]] (duration: 07m 29s) * 17:37 samtar@deploy1003: chlod, samtar: Continuing with deployment * 17:36 samtar@deploy1003: chlod, samtar: Backport for [[gerrit:1295976{{!}}nlwiki: change to Wikipedia 25 logo (T424519)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:34 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1295976{{!}}nlwiki: change to Wikipedia 25 logo (T424519)]] * 17:20 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1236: Update * 17:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd2001.codfw.wmnet to drbd * 17:04 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling * 17:04 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 17:04 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1180: Pooling * 17:03 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 17:03 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling * 17:03 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 16:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd2001.codfw.wmnet to drbd * 16:58 Amir1: drop flaggedrevs tables on wikinews wikis ([[phab:T423577|T423577]]) * 16:57 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 16:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93462 and previous config saved to /var/cache/conftool/dbconfig/20260601-165717-fceratto.json * 16:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93460 and previous config saved to /var/cache/conftool/dbconfig/20260601-164709-fceratto.json * 16:42 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling * 16:37 ryankemper@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,cluster=wdqs-main,service=wdqs-main,name=wdqs1015.eqiad.wmnet * 16:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93458 and previous config saved to /var/cache/conftool/dbconfig/20260601-163701-fceratto.json * 16:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:35 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1236.eqiad.wmnet * 16:35 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1236.eqiad.wmnet * 16:35 ryankemper@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,cluster=wdqs,service=wdqs-main,name=wdqs1015.eqiad.wmnet * 16:34 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update * 16:34 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1236: Update * 16:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1236.eqiad.wmnet with reason: Kernel update [[phab:T426633|T426633]] * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:30 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1236.eqiad.wmnet * 16:30 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1236.eqiad.wmnet * 16:30 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update * 16:29 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1236: Update * 16:29 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update * 16:29 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2003.codfw.wmnet to drbd * 16:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93455 and previous config saved to /var/cache/conftool/dbconfig/20260601-162653-fceratto.json * 16:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 16:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1209: Migration of db1209.eqiad.wmnet completed * 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1236.eqiad.wmnet with reason: Kernel update [[phab:T426633|T426633]] * 16:09 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1236: Update * 16:09 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1236: Update * 16:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:06 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 16:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2003.codfw.wmnet to drbd * 16:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 16:03 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 16:02 moritzm: temporarily remove ganeti2027 from the codfw cluster [[phab:T427357|T427357]] * 15:56 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:56 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.depool (exit_code=97) depool db1224: Pooling * 15:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2005.codfw.wmnet with OS bullseye * 15:53 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1224: Pooling * 15:51 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 15:49 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling * 15:49 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:48 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet * 15:45 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling * 15:44 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2005.codfw.wmnet with reason: host reimage * 15:40 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:40 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling * 15:40 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet * 15:40 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet * 15:40 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet * 15:40 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet * 15:39 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 15:39 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 15:39 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1209: Migration of db1209.eqiad.wmnet completed * 15:39 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:38 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:38 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling * 15:37 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2005.codfw.wmnet with reason: host reimage * 15:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 15:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1209.eqiad.wmnet with OS trixie * 15:28 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295802{{!}}hCaptcha: Raise SiteVerify error threshold to 100]] (duration: 06m 15s) * 15:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93446 and previous config saved to /var/cache/conftool/dbconfig/20260601-152638-fceratto.json * 15:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 15:26 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:25 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet * 15:25 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet * 15:25 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling * 15:25 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:24 kharlan@deploy1003: kharlan: Continuing with deployment * 15:24 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295802{{!}}hCaptcha: Raise SiteVerify error threshold to 100]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:22 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host testvm2005.codfw.wmnet with OS bullseye * 15:22 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295802{{!}}hCaptcha: Raise SiteVerify error threshold to 100]] * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:20 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295946{{!}}hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] (duration: 08m 24s) * 15:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:16 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 15:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1209.eqiad.wmnet with reason: host reimage * 15:14 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1295946{{!}}hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:13 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:12 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1295946{{!}}hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] * 15:10 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1209.eqiad.wmnet with reason: host reimage * 15:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93445 and previous config saved to /var/cache/conftool/dbconfig/20260601-151024-fceratto.json * 15:08 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-reboot (exit_code=0) rolling reboot on A:sessionstore * 15:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93443 and previous config saved to /var/cache/conftool/dbconfig/20260601-150017-fceratto.json * 14:55 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1209.eqiad.wmnet with OS trixie * 14:52 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 14:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1209: Upgrading db1209.eqiad.wmnet * 14:52 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 14:52 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1209: Upgrading db1209.eqiad.wmnet * 14:52 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 14:51 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:51 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 14:50 atsuko@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 14:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93441 and previous config saved to /var/cache/conftool/dbconfig/20260601-145010-fceratto.json * 14:49 atsuko@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 14:49 atsuko@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 14:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:42 atsuko@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 14:41 atsuko@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93440 and previous config saved to /var/cache/conftool/dbconfig/20260601-144002-fceratto.json * 14:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:30 ladsgroup@deploy1003: Synchronized portals: Deploy portals ([[phab:T421797|T421797]]) (duration: 02m 43s) * 14:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:27 ladsgroup@deploy1003: Synchronized portals/wikipedia.org/assets: Deploy portals ([[phab:T421797|T421797]]) (duration: 06m 10s) * 14:25 sukhe@dns1004: END - running authdns-update * 14:23 sukhe@dns1004: START - running authdns-update * 14:22 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 14:21 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:16 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:12 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:12 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:11 Lucas_WMDE: UTC afternoon backport+config window done * 14:10 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295918{{!}}Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] (duration: 11m 06s) * 14:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:05 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, codenamenoreste: Continuing with deployment * 14:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, codenamenoreste: Backport for [[gerrit:1295918{{!}}Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:01 eevans@cumin1003: START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:sessionstore * 13:58 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1295918{{!}}Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] * 13:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:52 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1265.eqiad.wmnet with OS trixie * 13:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93439 and previous config saved to /var/cache/conftool/dbconfig/20260601-133947-fceratto.json * 13:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 13:37 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage * 13:35 atsukoito: restarted pybal.service on lvs2013 * 13:31 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage * 13:31 atsukoito: restarted pybal.service on lvs2014 * 13:24 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs-test2001.codfw.wmnet * 13:24 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:22 atsukoito: restarted pybal.service on lvs1019 * 13:22 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in eqiad/ml-serve-eqiad: maintenance * 13:21 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in eqiad/ml-serve-eqiad: maintenance * 13:20 atsukoito: restarted pybal.service on lvs1020 * 13:20 Msz2001: UTC afternoon backpot+config window done * 13:20 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295875{{!}}Add SetGlobalPreference maintenance script (T427476)]] (duration: 06m 22s) * 13:19 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs-test2001.codfw.wmnet * 13:18 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1265.eqiad.wmnet with OS trixie * 13:18 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:16 mszwarc@deploy1003: mszwarc: Continuing with deployment * 13:15 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1295875{{!}}Add SetGlobalPreference maintenance script (T427476)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:14 atsukoito: sudo cumin 'A:lvs-low-traffic-eqiad' 'systemctl restart pybal.service' * 13:14 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1295875{{!}}Add SetGlobalPreference maintenance script (T427476)]] * 13:12 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295536{{!}}swwiki: Enable the Visual Editor on the project namespace (T427117)]] (duration: 10m 06s) * 13:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93438 and previous config saved to /var/cache/conftool/dbconfig/20260601-130949-fceratto.json * 13:08 mszwarc@deploy1003: codenamenoreste, mszwarc: Continuing with deployment * 13:07 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 13:06 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . * 13:05 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 13:04 mszwarc@deploy1003: codenamenoreste, mszwarc: Backport for [[gerrit:1295536{{!}}swwiki: Enable the Visual Editor on the project namespace (T427117)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:04 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 13:04 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 13:03 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 13:02 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1295536{{!}}swwiki: Enable the Visual Editor on the project namespace (T427117)]] * 12:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93437 and previous config saved to /var/cache/conftool/dbconfig/20260601-125941-fceratto.json * 12:56 dpogorzelski@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=inference,name=eqiad * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'readability' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'logo-detection' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'edit-check' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . * 12:52 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:50 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:49 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93436 and previous config saved to /var/cache/conftool/dbconfig/20260601-124934-fceratto.json * 12:48 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:47 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:46 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:44 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:43 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:42 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:41 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93435 and previous config saved to /var/cache/conftool/dbconfig/20260601-123926-fceratto.json * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:29 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:28 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 12:28 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:27 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:27 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2005.codfw.wmnet to plain * 12:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2005.codfw.wmnet to plain * 12:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet * 12:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 12:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2005.codfw.wmnet to drbd * 12:20 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:17 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:15 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in eqiad/ml-serve-eqiad: maintenance * 12:15 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in eqiad/ml-serve-eqiad: maintenance * 12:11 dpogorzelski@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=inference,name=eqiad * 12:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2005.codfw.wmnet to drbd * 12:05 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet * 12:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 12:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2027.codfw.wmnet * 12:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 11:59 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in eqiad/ml-serve-eqiad: maintenance * 11:59 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in eqiad/ml-serve-eqiad: maintenance * 11:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93434 and previous config saved to /var/cache/conftool/dbconfig/20260601-113911-fceratto.json * 11:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 11:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93433 and previous config saved to /var/cache/conftool/dbconfig/20260601-113843-fceratto.json * 11:37 moritzm: installing Exim security updates * 11:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:32 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:32 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:32 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93432 and previous config saved to /var/cache/conftool/dbconfig/20260601-112835-fceratto.json * 11:25 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 11:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:22 moritzm: installing imagemagick security updates * 11:22 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:22 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:22 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 11:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93430 and previous config saved to /var/cache/conftool/dbconfig/20260601-111827-fceratto.json * 11:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:14 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 11:12 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 11:10 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93429 and previous config saved to /var/cache/conftool/dbconfig/20260601-110820-fceratto.json * 11:04 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 11:01 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1055: repool after upgrade * 11:01 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93427 and previous config saved to /var/cache/conftool/dbconfig/20260601-110121-fceratto.json * 11:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance * 10:54 marostegui@dns1004: END - running authdns-update * 10:52 marostegui@dns1004: START - running authdns-update * 10:48 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1050 to es1 eqiad primary [[phab:T427032|T427032]]', diff saved to https://phabricator.wikimedia.org/P93425 and previous config saved to /var/cache/conftool/dbconfig/20260601-104837-marostegui.json * 10:47 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2055 to es1 codfw primary [[phab:T427032|T427032]]', diff saved to https://phabricator.wikimedia.org/P93424 and previous config saved to /var/cache/conftool/dbconfig/20260601-104739-marostegui.json * 10:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1177: Migration of db1177.eqiad.wmnet completed * 10:40 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet * 10:34 kamila@cumin1003: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet * 10:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93421 and previous config saved to /var/cache/conftool/dbconfig/20260601-103316-fceratto.json * 10:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93418 and previous config saved to /var/cache/conftool/dbconfig/20260601-102308-fceratto.json * 10:16 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1055: repool after upgrade * 10:15 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1055.eqiad.wmnet with OS trixie * 10:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93415 and previous config saved to /var/cache/conftool/dbconfig/20260601-101300-fceratto.json * 10:09 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 10:07 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 10:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93414 and previous config saved to /var/cache/conftool/dbconfig/20260601-100252-fceratto.json * 10:00 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1177: Migration of db1177.eqiad.wmnet completed * 09:58 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1055.eqiad.wmnet with reason: host reimage * 09:56 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 09:54 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 09:53 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1055.eqiad.wmnet with reason: host reimage * 09:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1177.eqiad.wmnet with OS trixie * 09:51 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 09:50 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 09:39 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1055.eqiad.wmnet with OS trixie * 09:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1055: Upgrading es1055.eqiad.wmnet * 09:38 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1055: Upgrading es1055.eqiad.wmnet * 09:37 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1177.eqiad.wmnet with reason: host reimage * 09:31 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1177.eqiad.wmnet with reason: host reimage * 09:17 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1177.eqiad.wmnet with OS trixie * 09:15 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 09:14 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 09:13 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 09:12 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 09:12 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1177: Upgrading db1177.eqiad.wmnet * 09:11 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1177: Upgrading db1177.eqiad.wmnet * 09:11 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93410 and previous config saved to /var/cache/conftool/dbconfig/20260601-090237-fceratto.json * 09:02 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93409 and previous config saved to /var/cache/conftool/dbconfig/20260601-090209-fceratto.json * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P93408 and previous config saved to /var/cache/conftool/dbconfig/20260601-085202-fceratto.json * 08:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P93407 and previous config saved to /var/cache/conftool/dbconfig/20260601-084154-fceratto.json * 08:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93406 and previous config saved to /var/cache/conftool/dbconfig/20260601-083146-fceratto.json * 08:24 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93405 and previous config saved to /var/cache/conftool/dbconfig/20260601-082442-fceratto.json * 08:24 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1168.eqiad.wmnet with reason: Maintenance * 07:58 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295454{{!}}Disable the creation of synthetic main refs in production (T427484)]] (duration: 11m 26s) * 07:56 XioNoX: add no_p2p term to pfw1-codfw BGP_fundraising_export - [[phab:T423384|T423384]] * 07:52 wmde-fisch@deploy1003: lilients, wmde-fisch: Continuing with deployment * 07:51 wmde-fisch@deploy1003: lilients, wmde-fisch: Backport for [[gerrit:1295454{{!}}Disable the creation of synthetic main refs in production (T427484)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:47 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1295454{{!}}Disable the creation of synthetic main refs in production (T427484)]] * 07:45 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294826{{!}}Update VE core submodule to master (9cf5524e7) (T424232)]] (duration: 31m 34s) * 07:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 07:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 07:32 wmde-fisch@deploy1003: wmde-fisch: Continuing with deployment * 07:31 wmde-fisch@deploy1003: wmde-fisch: Backport for [[gerrit:1294826{{!}}Update VE core submodule to master (9cf5524e7) (T424232)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet * 07:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet * 07:13 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1294826{{!}}Update VE core submodule to master (9cf5524e7) (T424232)]] * 06:48 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 06:47 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. == 2026-05-31 == * 02:06 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 30s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-30 == * 16:21 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:21 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:21 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:21 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 06:39 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 06:39 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 06:39 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 06:38 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 27s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-29 == * 23:39 aokoth@cumin1003: END (PASS) - Cookbook sre.vrts.upgrade (exit_code=0) on VRTS host vrts1003.eqiad.wmnet * 23:37 aokoth@cumin1003: START - Cookbook sre.vrts.upgrade on VRTS host vrts1003.eqiad.wmnet * 21:42 catrope@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 21:41 catrope@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 17:40 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295487{{!}}Hide experiment if not active and no assigned group]] (duration: 06m 54s) * 17:35 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 17:34 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1295487{{!}}Hide experiment if not active and no assigned group]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:33 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1295487{{!}}Hide experiment if not active and no assigned group]] * 16:30 jgreen@dns1004: END - running authdns-update * 16:28 jgreen@dns1004: START - running authdns-update * 16:13 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:12 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 15:28 dancy@deploy1003: Installation of scap version "4.267.0" completed for 2 hosts * 15:26 dancy@deploy1003: Installing scap version "4.267.0" for 2 host(s) * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:15 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295466{{!}}GlobalPreferencesHandler: Cast auto-reveal expiry to int (T427625)]] (duration: 07m 58s) * 14:11 kharlan@deploy1003: kharlan: Continuing with deployment * 14:09 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295466{{!}}GlobalPreferencesHandler: Cast auto-reveal expiry to int (T427625)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:07 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295466{{!}}GlobalPreferencesHandler: Cast auto-reveal expiry to int (T427625)]] * 13:53 moritzm: imported OpenJDK 21 21.0.11+10-1~deb12u1 to component/jdk21 (backport of latest Java 21 security release for Bookworm) * 12:09 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader1006.wikimedia.org * 12:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1006.wikimedia.org with OS trixie * 11:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1006.wikimedia.org with reason: host reimage * 11:47 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1006.wikimedia.org with reason: host reimage * 11:36 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader1006.wikimedia.org with OS trixie * 11:15 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:15 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:13 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader1006.wikimedia.org on all recursors * 11:12 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader1006.wikimedia.org on all recursors * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:06 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:00 jmm@cumin2002: START - Cookbook sre.dns.netbox * 11:00 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader1006.wikimedia.org * 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader1005.wikimedia.org * 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1005.wikimedia.org with OS trixie * 10:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1005.wikimedia.org with reason: host reimage * 10:40 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2212: Pooling * 10:37 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1005.wikimedia.org with reason: host reimage * 10:27 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader1005.wikimedia.org with OS trixie * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 10:01 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:59 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:55 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 09:50 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 09:49 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:45 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:44 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup2014.codfw.wmnet with OS bookworm * 09:33 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:20 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup2014.codfw.wmnet with reason: host reimage * 09:12 jynus@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on backup2014.codfw.wmnet with reason: host reimage * 09:10 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 09:10 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 09:03 jelto@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM etherpad2002.codfw.wmnet * 08:59 jelto@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM etherpad2002.codfw.wmnet * 08:59 jelto: gnt-instance modify -B memory=4g,vcpus=1 etherpad2002.codfw.wmnet - [[phab:T427588|T427588]] * 08:54 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2014.codfw.wmnet with OS bookworm * 08:51 jelto@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM etherpad1004.eqiad.wmnet * 08:50 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams-internal: apply * 08:50 jynus@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host backup2014.codfw.wmnet with OS bookworm * 08:49 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams-internal: apply * 08:47 jelto@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM etherpad1004.eqiad.wmnet * 08:46 jelto: gnt-instance modify -B memory=4g,vcpus=1 etherpad1004.eqiad.wmnet - [[phab:T427588|T427588]] * 08:42 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2212: Pooling * 08:42 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 08:39 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2212: Pooling * 08:39 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 08:38 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams-internal: apply * 08:37 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams-internal: apply * 08:37 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams-internal: apply * 08:36 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams-internal: apply * 08:33 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2014.codfw.wmnet with OS bookworm * 08:31 jynus@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host backup2014.codfw.wmnet with OS bookworm * 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader1005.wikimedia.org on all recursors * 08:21 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader1005.wikimedia.org on all recursors * 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 08:21 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 08:18 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 08:17 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 08:16 jmm@cumin2002: START - Cookbook sre.dns.netbox * 08:16 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader1005.wikimedia.org * 08:05 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2212: Pooling * 07:59 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 07:59 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 07:54 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 07:54 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2212.codfw.wmnet * 07:54 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2212.codfw.wmnet * 07:22 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2014.codfw.wmnet with OS bookworm * 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader2006.wikimedia.org * 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader2006.wikimedia.org with OS trixie * 06:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader2006.wikimedia.org with reason: host reimage * 06:53 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader2006.wikimedia.org with reason: host reimage * 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader2006.wikimedia.org with OS trixie * 06:32 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:32 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader2006.wikimedia.org on all recursors * 06:31 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader2006.wikimedia.org on all recursors * 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:31 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:27 jmm@cumin2002: START - Cookbook sre.dns.netbox * 06:27 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader2006.wikimedia.org * 03:01 vriley@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts db1224.eqiad.wmnet * 03:00 vriley@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1224.eqiad.wmnet * 03:00 vriley@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1224.eqiad.wmnet * 02:56 vriley@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1224.eqiad.wmnet * 01:47 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5032.eqsin.wmnet with OS trixie * 01:18 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5032.eqsin.wmnet with reason: host reimage * 01:14 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5032.eqsin.wmnet with reason: host reimage * 00:31 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cp5032.eqsin.wmnet with OS trixie * 00:29 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host cp5032.eqsin.wmnet * 00:23 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 00:22 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply * 00:21 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 00:21 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply == 2026-05-28 == * 23:07 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 23:07 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new ae1.522 interface - pt1979@cumin2002" * 23:07 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new ae1.522 interface - pt1979@cumin2002" * 23:02 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 22:34 andrewbogott: reprepro includedeb trixie-wikimedia /home/andrew/magnum-cluster-api_0.36.6-1~wmf13u2_amd64.deb * 22:31 logmsgbot: dreamyjazz Deployed security patch for [[phab:T426388|T426388]] * 21:33 maryum: Deployed security fix for [[phab:T426867|T426867]] * 21:21 alexsanford: Deployed security fix for [[phab:T426889|T426889]] * 21:07 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host cp5032.eqsin.wmnet * 21:04 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "setup new eqsin vlan - pt1979@cumin2002 - [[phab:T427393|T427393]]" * 21:04 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "setup new eqsin vlan - pt1979@cumin2002 - [[phab:T427393|T427393]]" * 20:48 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295066{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T420336 T427098 T427354 T427082)]], [[gerrit:1295067{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T427082)]] (duration: 07m 34s) * 20:44 arlolra@deploy1003: arlolra: Continuing with deployment * 20:43 arlolra@deploy1003: arlolra: Backport for [[gerrit:1295066{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T420336 T427098 T427354 T427082)]], [[gerrit:1295067{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T427082)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:41 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1295066{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T420336 T427098 T427354 T427082)]], [[gerrit:1295067{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T427082)]] * 20:34 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293805{{!}}Deploy PRV to 7 wikis (T427331)]] (duration: 07m 20s) * 20:30 arlolra@deploy1003: arlolra: Continuing with deployment * 20:29 arlolra@deploy1003: arlolra: Backport for [[gerrit:1293805{{!}}Deploy PRV to 7 wikis (T427331)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:27 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1293805{{!}}Deploy PRV to 7 wikis (T427331)]] * 20:22 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291996{{!}}Replace deprecated Hooks::getInstance (T426981)]], [[gerrit:1294393{{!}}Permissions: Create wmf-officeit group on officewiki]], [[gerrit:1294229{{!}}Deploy IRS Direct Reporting feature to enwiki (T427369)]], [[gerrit:1295039{{!}}Add 2FA enforcement demotion config for phase 2 groups (T423119)]] (duration: 09m 07s) * 20:18 stran@deploy1003: alexsanford, stran, catrope, dreamyjazz: Continuing with deployment * 20:14 stran@deploy1003: alexsanford, stran, catrope, dreamyjazz: Backport for [[gerrit:1291996{{!}}Replace deprecated Hooks::getInstance (T426981)]], [[gerrit:1294393{{!}}Permissions: Create wmf-officeit group on officewiki]], [[gerrit:1294229{{!}}Deploy IRS Direct Reporting feature to enwiki (T427369)]], [[gerrit:1295039{{!}}Add 2FA enforcement demotion config for phase 2 groups (T423119)]] synced to the testservers (see https://wikitech. * 20:13 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5032.eqsin.wmnet with OS trixie * 20:13 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1291996{{!}}Replace deprecated Hooks::getInstance (T426981)]], [[gerrit:1294393{{!}}Permissions: Create wmf-officeit group on officewiki]], [[gerrit:1294229{{!}}Deploy IRS Direct Reporting feature to enwiki (T427369)]], [[gerrit:1295039{{!}}Add 2FA enforcement demotion config for phase 2 groups (T423119)]] * 19:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1018.eqiad.wmnet * 19:27 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1018.eqiad.wmnet * 19:09 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1018.eqiad.wmnet with reason: Kernel reboot * 19:09 brett: Stopping pybal/puppet/downtiming lvs1018.eqiad.wmnet for reboot * 19:05 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1019.eqiad.wmnet * 19:05 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1019.eqiad.wmnet * 18:52 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cp5032.eqsin.wmnet with OS trixie * 18:51 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:51 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change cp5032 IP - pt1979@cumin2002" * 18:51 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change cp5032 IP - pt1979@cumin2002" * 18:47 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 18:40 mutante: planet1003/planet2003 - apt-get upgrade - all pending package upgrades * 18:35 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: Kernel reboot * 18:34 brett: Stopping pybal/puppet/downtiming lvs1019.eqiad.wmnet for reboot and BIOS update/memory self-healing - [[phab:T426109|T426109]] * 18:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs2011.codfw.wmnet * 18:25 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs2011.codfw.wmnet * 18:19 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2011.codfw.wmnet with reason: Kernel reboot * 18:19 brett: Stopping pybal/puppet/downtiming lvs2011.codfw.wmnet for reboot * 18:09 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs2013.codfw.wmnet * 18:06 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs2013.codfw.wmnet * 18:00 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2013.codfw.wmnet with reason: Kernel reboot * 17:57 brett: Stopping pybal/puppet/downtiming lvs2013.codfw.wmnet for reboot * 17:19 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 16:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93393 and previous config saved to /var/cache/conftool/dbconfig/20260528-164514-fceratto.json * 16:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P93392 and previous config saved to /var/cache/conftool/dbconfig/20260528-163507-fceratto.json * 16:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P93391 and previous config saved to /var/cache/conftool/dbconfig/20260528-162459-fceratto.json * 16:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db1224.eqiad.wmnet with reason: unreachable [[phab:T427535|T427535]] * 16:17 swfrench-wmf: reprepro include xdebug_3.4.4-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:17 swfrench-wmf: reprepro include wikidiff2_1.14.1-2+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:17 swfrench-wmf: reprepro include php-yaml_2.2.4-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-xhprof_2.3.10-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-wmerrors_2.0.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-uuid_1.3.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-redis_6.2.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 swfrench-wmf: reprepro include php-pcov_1.0.12-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 swfrench-wmf: reprepro include php-memcached_3.3.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 16:15 swfrench-wmf: reprepro include php-luasandbox_4.1.2-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 16:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93390 and previous config saved to /var/cache/conftool/dbconfig/20260528-161452-fceratto.json * 16:14 swfrench-wmf: reprepro include php-imagick_3.7.0-13+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:14 swfrench-wmf: reprepro include php-excimer_1.2.5-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:09 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:09 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:06 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1251 ([[phab:T426633|T426633]])', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20260528-160646-fceratto.json * 16:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1251.eqiad.wmnet with reason: Maintenance * 16:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93388 and previous config saved to /var/cache/conftool/dbconfig/20260528-160613-fceratto.json * 15:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P93387 and previous config saved to /var/cache/conftool/dbconfig/20260528-155605-fceratto.json * 15:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P93386 and previous config saved to /var/cache/conftool/dbconfig/20260528-154557-fceratto.json * 15:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93385 and previous config saved to /var/cache/conftool/dbconfig/20260528-153550-fceratto.json * 15:27 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1235 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93384 and previous config saved to /var/cache/conftool/dbconfig/20260528-152736-fceratto.json * 15:27 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1235.eqiad.wmnet with reason: Maintenance * 15:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93383 and previous config saved to /var/cache/conftool/dbconfig/20260528-152708-fceratto.json * 15:20 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp5032.eqsin.wmnet with reason: Testing reimaging on new subnet * 15:18 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5032.* * 15:17 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P93382 and previous config saved to /var/cache/conftool/dbconfig/20260528-151701-fceratto.json * 15:17 jhathaway: dmarc ingress test on mx-in1001 * 15:14 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:14 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P93381 and previous config saved to /var/cache/conftool/dbconfig/20260528-150653-fceratto.json * 14:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93380 and previous config saved to /var/cache/conftool/dbconfig/20260528-145646-fceratto.json * 14:56 moritzm: installing nginx security updates * 14:49 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1234 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93379 and previous config saved to /var/cache/conftool/dbconfig/20260528-144936-fceratto.json * 14:49 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 14:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1234.eqiad.wmnet with reason: Maintenance * 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93378 and previous config saved to /var/cache/conftool/dbconfig/20260528-144909-fceratto.json * 14:48 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader2005.wikimedia.org * 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader2005.wikimedia.org with OS trixie * 14:47 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 14:39 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2189.codfw.wmnet * 14:39 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2189.codfw.wmnet * 14:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P93377 and previous config saved to /var/cache/conftool/dbconfig/20260528-143901-fceratto.json * 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader2005.wikimedia.org with reason: host reimage * 14:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P93376 and previous config saved to /var/cache/conftool/dbconfig/20260528-142854-fceratto.json * 14:28 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:28 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader2005.wikimedia.org with reason: host reimage * 14:27 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:19 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294998{{!}}ImageContentLookup: Fix issue created by strict types (T427505)]], [[gerrit:1295001{{!}}Enable hCaptcha for VisualEditor in group 1 (T425940)]] (duration: 11m 29s) * 14:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93375 and previous config saved to /var/cache/conftool/dbconfig/20260528-141846-fceratto.json * 14:15 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1232 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93374 and previous config saved to /var/cache/conftool/dbconfig/20260528-141029-fceratto.json * 14:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1232.eqiad.wmnet with reason: Maintenance * 14:10 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader2005.wikimedia.org with OS trixie * 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93373 and previous config saved to /var/cache/conftool/dbconfig/20260528-141001-fceratto.json * 14:09 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1294998{{!}}ImageContentLookup: Fix issue created by strict types (T427505)]], [[gerrit:1295001{{!}}Enable hCaptcha for VisualEditor in group 1 (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:08 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294998{{!}}ImageContentLookup: Fix issue created by strict types (T427505)]], [[gerrit:1295001{{!}}Enable hCaptcha for VisualEditor in group 1 (T425940)]] * 14:00 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down * 13:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P93371 and previous config saved to /var/cache/conftool/dbconfig/20260528-135951-fceratto.json * 13:58 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp6015.drmrs.wmnet,service=(cdn{{!}}ats-be) * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:55 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader2005.wikimedia.org on all recursors * 13:55 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader2005.wikimedia.org on all recursors * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:55 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P93370 and previous config saved to /var/cache/conftool/dbconfig/20260528-134944-fceratto.json * 13:40 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 13:40 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 13:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93369 and previous config saved to /var/cache/conftool/dbconfig/20260528-133936-fceratto.json * 13:39 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 13:38 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 13:36 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294986{{!}}Image Carousel: check candidate pages (T427336)]] (duration: 06m 40s) * 13:34 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 13:33 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 13:32 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93368 and previous config saved to /var/cache/conftool/dbconfig/20260528-133230-fceratto.json * 13:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance * 13:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93367 and previous config saved to /var/cache/conftool/dbconfig/20260528-133202-fceratto.json * 13:31 mlitn@deploy1003: mlitn: Continuing with deployment * 13:31 mlitn@deploy1003: mlitn: Backport for [[gerrit:1294986{{!}}Image Carousel: check candidate pages (T427336)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:29 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1294986{{!}}Image Carousel: check candidate pages (T427336)]] * 13:22 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 13:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P93366 and previous config saved to /var/cache/conftool/dbconfig/20260528-132155-fceratto.json * 13:21 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 13:17 elukey: clean up a lof ot stale Kafka ACLs on Kafka Jumbo - Details in [[phab:T425528|T425528]] * 13:14 jmm@cumin2002: START - Cookbook sre.dns.netbox * 13:14 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader2005.wikimedia.org * 13:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P93365 and previous config saved to /var/cache/conftool/dbconfig/20260528-131147-fceratto.json * 13:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93364 and previous config saved to /var/cache/conftool/dbconfig/20260528-130139-fceratto.json * 12:54 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1218 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93363 and previous config saved to /var/cache/conftool/dbconfig/20260528-125439-fceratto.json * 12:54 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1218.eqiad.wmnet with reason: Maintenance * 12:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93362 and previous config saved to /var/cache/conftool/dbconfig/20260528-125412-fceratto.json * 12:48 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 12:48 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 12:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P93361 and previous config saved to /var/cache/conftool/dbconfig/20260528-124404-fceratto.json * 12:44 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 12:43 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 12:39 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 12:38 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 12:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P93360 and previous config saved to /var/cache/conftool/dbconfig/20260528-123357-fceratto.json * 12:25 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest1006.eqiad.wmnet with OS trixie * 12:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93359 and previous config saved to /var/cache/conftool/dbconfig/20260528-122349-fceratto.json * 12:15 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93358 and previous config saved to /var/cache/conftool/dbconfig/20260528-121551-fceratto.json * 12:15 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance * 12:15 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie * 12:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93357 and previous config saved to /var/cache/conftool/dbconfig/20260528-121523-fceratto.json * 12:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P93356 and previous config saved to /var/cache/conftool/dbconfig/20260528-120515-fceratto.json * 12:02 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest1006.eqiad.wmnet with OS trixie * 12:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 12:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 12:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 12:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 11:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P93355 and previous config saved to /var/cache/conftool/dbconfig/20260528-115508-fceratto.json * 11:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93354 and previous config saved to /var/cache/conftool/dbconfig/20260528-114500-fceratto.json * 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93353 and previous config saved to /var/cache/conftool/dbconfig/20260528-113635-fceratto.json * 11:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 11:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance * 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93352 and previous config saved to /var/cache/conftool/dbconfig/20260528-113559-fceratto.json * 11:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P93351 and previous config saved to /var/cache/conftool/dbconfig/20260528-112551-fceratto.json * 11:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P93350 and previous config saved to /var/cache/conftool/dbconfig/20260528-111543-fceratto.json * 11:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93349 and previous config saved to /var/cache/conftool/dbconfig/20260528-110536-fceratto.json * 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1195 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93348 and previous config saved to /var/cache/conftool/dbconfig/20260528-105820-fceratto.json * 10:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie * 10:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1195.eqiad.wmnet with reason: Maintenance * 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93347 and previous config saved to /var/cache/conftool/dbconfig/20260528-105753-fceratto.json * 10:56 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 10:55 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 10:55 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 10:55 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 10:50 moritzm: update trixie netboot image for 13.5 point release [[phab:T427072|T427072]] * 10:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P93346 and previous config saved to /var/cache/conftool/dbconfig/20260528-104745-fceratto.json * 10:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P93345 and previous config saved to /var/cache/conftool/dbconfig/20260528-103738-fceratto.json * 10:29 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P13724 # [[phab:T406971|T406971]] * 10:28 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P14223 # [[phab:T422264|T422264]] * 10:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93344 and previous config saved to /var/cache/conftool/dbconfig/20260528-102730-fceratto.json * 10:26 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P1748 # [[phab:T422392|T422392]] * 10:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93343 and previous config saved to /var/cache/conftool/dbconfig/20260528-101900-fceratto.json * 10:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance * 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93342 and previous config saved to /var/cache/conftool/dbconfig/20260528-101829-fceratto.json * 10:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P93341 and previous config saved to /var/cache/conftool/dbconfig/20260528-100822-fceratto.json * 09:59 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290687{{!}}stream: webrequest.page_view (T426092 T426091)]] (duration: 06m 41s) * 09:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P93340 and previous config saved to /var/cache/conftool/dbconfig/20260528-095814-fceratto.json * 09:55 javiermonton@deploy1003: javiermonton: Continuing with deployment * 09:54 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1290687{{!}}stream: webrequest.page_view (T426092 T426091)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:52 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1290687{{!}}stream: webrequest.page_view (T426092 T426091)]] * 09:48 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294243{{!}}Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937{{!}}CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] (duration: 07m 37s) * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93339 and previous config saved to /var/cache/conftool/dbconfig/20260528-094807-fceratto.json * 09:44 dreamyjazz@deploy1003: dreamyjazz, stran: Continuing with deployment * 09:44 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:43 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:43 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:43 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:42 dreamyjazz@deploy1003: dreamyjazz, stran: Backport for [[gerrit:1294243{{!}}Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937{{!}}CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294243{{!}}Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937{{!}}CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] * 09:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93338 and previous config saved to /var/cache/conftool/dbconfig/20260528-093920-fceratto.json * 09:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: Maintenance * 09:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93337 and previous config saved to /var/cache/conftool/dbconfig/20260528-093849-fceratto.json * 09:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P93336 and previous config saved to /var/cache/conftool/dbconfig/20260528-092842-fceratto.json * 09:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance * 09:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93335 and previous config saved to /var/cache/conftool/dbconfig/20260528-092239-fceratto.json * 09:22 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pki-root1001.eqiad.wmnet * 09:22 elukey@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:22 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki-root1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1003" * 09:22 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki-root1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1003" * 09:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:18 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 09:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P93334 and previous config saved to /var/cache/conftool/dbconfig/20260528-091834-fceratto.json * 09:18 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 09:18 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 09:17 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1165: Reboot completed * 09:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 09:17 elukey@cumin1003: START - Cookbook sre.dns.netbox * 09:14 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 09:13 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:13 elukey@cumin1003: START - Cookbook sre.hosts.decommission for hosts pki-root1001.eqiad.wmnet * 09:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P93332 and previous config saved to /var/cache/conftool/dbconfig/20260528-091231-fceratto.json * 09:09 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:09 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93331 and previous config saved to /var/cache/conftool/dbconfig/20260528-090826-fceratto.json * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P93329 and previous config saved to /var/cache/conftool/dbconfig/20260528-090224-fceratto.json * 09:02 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Deploying to prod (duration: 02m 31s) * 09:01 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2216 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93328 and previous config saved to /var/cache/conftool/dbconfig/20260528-090114-fceratto.json * 09:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2216.codfw.wmnet with reason: Maintenance * 09:00 joal@deploy1003: Finished deploy [analytics/refinery@878cb24] (thin): Regular analytics weekly train THIN - 2[analytics/refinery@878cb24a] (duration: 02m 08s) * 08:59 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Deploying to prod * 08:58 joal@deploy1003: Started deploy [analytics/refinery@878cb24] (thin): Regular analytics weekly train THIN - 2[analytics/refinery@878cb24a] * 08:57 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Testing on backup host (duration: 00m 53s) * 08:56 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Testing on backup host * 08:56 joal@deploy1003: Finished deploy [analytics/refinery@878cb24]: Regular analytics weekly train - 2 [analytics/refinery@878cb24a] (duration: 06m 54s) * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93327 and previous config saved to /var/cache/conftool/dbconfig/20260528-085216-fceratto.json * 08:50 XioNoX: cr1-codfw# delete protocols bgp group fundraising family inet6 - [[phab:T423384|T423384]] * 08:49 joal@deploy1003: Started deploy [analytics/refinery@878cb24]: Regular analytics weekly train - 2 [analytics/refinery@878cb24a] * 08:49 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294925{{!}}hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] (duration: 09m 20s) * 08:49 joal@deploy1003: Finished deploy [analytics/refinery@878cb24] (hadoop-test): Regular analytics weekly train TEST -2 [analytics/refinery@878cb24a] (duration: 02m 00s) * 08:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93326 and previous config saved to /var/cache/conftool/dbconfig/20260528-084906-fceratto.json * 08:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1209.eqiad.wmnet with reason: Maintenance * 08:48 slyngshede@dns1004: END - running authdns-update * 08:47 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1165: Reboot completed * 08:47 joal@deploy1003: Started deploy [analytics/refinery@878cb24] (hadoop-test): Regular analytics weekly train TEST -2 [analytics/refinery@878cb24a] * 08:47 slyngs: Upgrade IDP to CAS 7.3.7.1 * 08:46 slyngshede@dns1004: START - running authdns-update * 08:45 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 08:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93324 and previous config saved to /var/cache/conftool/dbconfig/20260528-084149-fceratto.json * 08:41 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1294925{{!}}hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294925{{!}}hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] * 08:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet * 08:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet * 08:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93323 and previous config saved to /var/cache/conftool/dbconfig/20260528-083504-fceratto.json * 08:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1025].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 08:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1165.eqiad.wmnet with reason: Maintenance * 08:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93322 and previous config saved to /var/cache/conftool/dbconfig/20260528-083331-fceratto.json * 08:24 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1209: Test * 08:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P93320 and previous config saved to /var/cache/conftool/dbconfig/20260528-082324-fceratto.json * 08:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2189: repool after crash * 08:17 slyngshede@dns1004: END - running authdns-update * 08:16 slyngshede@dns1004: START - running authdns-update * 08:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P93318 and previous config saved to /var/cache/conftool/dbconfig/20260528-081316-fceratto.json * 08:10 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 08:09 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1209: Test * 08:05 hashar@deploy1003: Finished deploy [integration/docroot@2a51016]: build: update dependencies + eslint fix in comment. f021d3f..2a51016 (duration: 00m 13s) * 08:05 hashar@deploy1003: Started deploy [integration/docroot@2a51016]: build: update dependencies + eslint fix in comment. f021d3f..2a51016 * 08:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93315 and previous config saved to /var/cache/conftool/dbconfig/20260528-080309-fceratto.json * 07:56 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93314 and previous config saved to /var/cache/conftool/dbconfig/20260528-075631-fceratto.json * 07:56 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020,1022-1023].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 07:56 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance * 07:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93313 and previous config saved to /var/cache/conftool/dbconfig/20260528-075521-fceratto.json * 07:47 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab replica * 07:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93311 and previous config saved to /var/cache/conftool/dbconfig/20260528-074513-fceratto.json * 07:37 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2189: repool after crash * 07:36 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab replica * 07:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93309 and previous config saved to /var/cache/conftool/dbconfig/20260528-073506-fceratto.json * 07:34 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica * 07:29 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294808{{!}}Don't run the click intent experiment on mobile (T426743)]] (duration: 06m 29s) * 07:25 wmde-fisch@deploy1003: thiemowmde, wmde-fisch: Continuing with deployment * 07:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93308 and previous config saved to /var/cache/conftool/dbconfig/20260528-072458-fceratto.json * 07:24 wmde-fisch@deploy1003: thiemowmde, wmde-fisch: Backport for [[gerrit:1294808{{!}}Don't run the click intent experiment on mobile (T426743)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:24 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica * 07:23 tgr@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwikisource --logwiki=metawiki Ioed Renamed_user_4232d41570b9e8f46ef150e5e360e446 # [[phab:T427459|T427459]] * 07:22 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1294808{{!}}Don't run the click intent experiment on mobile (T426743)]] * 07:20 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1270986{{!}}Update wikimania wordmark for 2026 (T413331)]] (duration: 06m 54s) * 07:18 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93307 and previous config saved to /var/cache/conftool/dbconfig/20260528-071836-fceratto.json * 07:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1264.eqiad.wmnet with reason: Maintenance * 07:16 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1167: Reboot completed * 07:16 wmde-fisch@deploy1003: wmde-fisch, robertsky: Continuing with deployment * 07:15 wmde-fisch@deploy1003: wmde-fisch, robertsky: Backport for [[gerrit:1270986{{!}}Update wikimania wordmark for 2026 (T413331)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:13 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1270986{{!}}Update wikimania wordmark for 2026 (T413331)]] * 07:11 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289898{{!}}Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] (duration: 07m 15s) * 07:07 wmde-fisch@deploy1003: wmde-fisch, arthurtaylor: Continuing with deployment * 07:06 wmde-fisch@deploy1003: wmde-fisch, arthurtaylor: Backport for [[gerrit:1289898{{!}}Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:04 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1289898{{!}}Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] * 06:43 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1167: Reboot completed * 06:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93303 and previous config saved to /var/cache/conftool/dbconfig/20260528-064217-fceratto.json * 06:33 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93302 and previous config saved to /var/cache/conftool/dbconfig/20260528-063357-fceratto.json * 06:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 06:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance * 06:25 hashar: Restarting CI Jenkins for plugins upgrades * 06:16 fceratto@dns1005: END - running authdns-update * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1209 [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93301 and previous config saved to /var/cache/conftool/dbconfig/20260528-061609-fceratto.json * 06:14 fceratto@dns1005: START - running authdns-update * 06:11 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1193 to s8 primary and set section read-write [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93300 and previous config saved to /var/cache/conftool/dbconfig/20260528-061138-fceratto.json * 06:10 fceratto@cumin1003: dbctl commit (dc=all): 'Set s8 eqiad as read-only for maintenance - [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93299 and previous config saved to /var/cache/conftool/dbconfig/20260528-061048-fceratto.json * 06:10 federico3: Starting s8 eqiad failover from db1209 to db1193 - [[phab:T426095|T426095]] * 06:04 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1193 with weight 0 [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93298 and previous config saved to /var/cache/conftool/dbconfig/20260528-060412-fceratto.json * 06:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s8 [[phab:T426095|T426095]] * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 41s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:53 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 00:53 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new subnet in eqsin - pt1979@cumin2002" * 00:53 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new subnet in eqsin - pt1979@cumin2002" * 00:49 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 00:25 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294470{{!}}Activate conductwiki (T426984)]] (duration: 07m 12s) * 00:21 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 00:20 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1294470{{!}}Activate conductwiki (T426984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 00:18 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1294470{{!}}Activate conductwiki (T426984)]] * 00:12 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294438{{!}}Init conductwiki (T426984)]] (duration: 07m 25s) * 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 00:08 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 00:06 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1294438{{!}}Init conductwiki (T426984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 00:04 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1294438{{!}}Init conductwiki (T426984)]] * 00:04 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] == 2026-05-27 == * 23:13 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294432{{!}}Exclude more content from selection (T426308)]], [[gerrit:1285523{{!}}Remove MinervaNightMode config after skin cleanup (T426689)]] (duration: 08m 42s) * 23:09 jdlrobson@deploy1003: jdlrobson, h2o, egardner: Continuing with deployment * 23:06 jdlrobson@deploy1003: jdlrobson, h2o, egardner: Backport for [[gerrit:1294432{{!}}Exclude more content from selection (T426308)]], [[gerrit:1285523{{!}}Remove MinervaNightMode config after skin cleanup (T426689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:04 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294432{{!}}Exclude more content from selection (T426308)]], [[gerrit:1285523{{!}}Remove MinervaNightMode config after skin cleanup (T426689)]] * 22:58 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294435{{!}}passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] (duration: 07m 49s) * 22:55 ladsgroup@cumin1003: END (PASS) - Cookbook sre.mysql.sanitarium_restart (exit_code=0) * 22:54 catrope@deploy1003: catrope: Continuing with deployment * 22:52 catrope@deploy1003: catrope: Backport for [[gerrit:1294435{{!}}passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:50 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1294435{{!}}passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] * 22:46 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294360{{!}}Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322{{!}}Thumbnails are not being optimized in large mode (T427237)]] (duration: 06m 54s) * 22:42 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 22:41 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1294360{{!}}Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322{{!}}Thumbnails are not being optimized in large mode (T427237)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:40 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart * 22:40 ladsgroup@cumin1003: END (FAIL) - Cookbook sre.mysql.sanitarium_restart (exit_code=99) * 22:40 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart * 22:39 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294360{{!}}Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322{{!}}Thumbnails are not being optimized in large mode (T427237)]] * 22:39 ladsgroup@deploy1003: Finished scap sync-world: Add conduct.wikimedia.org ([[phab:T426984|T426984]]) (duration: 07m 16s) * 22:35 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 22:34 ladsgroup@deploy1003: ladsgroup: Add conduct.wikimedia.org ([[phab:T426984|T426984]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:33 ladsgroup@deploy1003: Started scap sync-world: Add conduct.wikimedia.org ([[phab:T426984|T426984]]) * 22:13 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294370{{!}}Carousel only on articles (T427336)]] (duration: 10m 00s) * 22:09 egardner@deploy1003: egardner: Continuing with deployment * 22:05 egardner@deploy1003: egardner: Backport for [[gerrit:1294370{{!}}Carousel only on articles (T427336)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:03 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1294370{{!}}Carousel only on articles (T427336)]] * 21:37 bking@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 15 days, 0:00:00 on relforge[1008-1010].eqiad.wmnet with reason: non-production environment * 21:20 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 21:20 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 21:20 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 21:19 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 21:04 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1288370{{!}}Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819{{!}}Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] (duration: 07m 38s) * 20:59 ebernhardson@deploy1003: matmarex, ebernhardson, pppery: Continuing with deployment * 20:58 ebernhardson@deploy1003: matmarex, ebernhardson, pppery: Backport for [[gerrit:1288370{{!}}Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819{{!}}Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:56 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1288370{{!}}Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819{{!}}Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] * 20:51 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294373{{!}}identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374{{!}}Revert^2 "cirrus: AB test query suggester variants" (T407432)]] (duration: 07m 30s) * 20:47 ebernhardson@deploy1003: ebernhardson: Continuing with deployment * 20:46 ebernhardson@deploy1003: ebernhardson: Backport for [[gerrit:1294373{{!}}identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374{{!}}Revert^2 "cirrus: AB test query suggester variants" (T407432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:44 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294373{{!}}identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374{{!}}Revert^2 "cirrus: AB test query suggester variants" (T407432)]] * 20:43 swfrench-wmf: reprepro include dh-php_5.5+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 20:39 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts lvs1016.eqiad.wmnet * 20:39 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:39 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brett@cumin2002" * 20:38 swfrench-wmf: reprepro include php-defaults_94+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 20:37 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brett@cumin2002" * 20:31 brett@cumin2002: START - Cookbook sre.dns.netbox * 20:27 swfrench-wmf: reprepro include php8.3_8.3.31-1+wmf12u2 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 20:25 brett@cumin2002: START - Cookbook sre.hosts.decommission for hosts lvs1016.eqiad.wmnet * 20:25 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294342{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344{{!}}frwiki: restrict Article Guidance experiment to junior editors (T426871)]] (duration: 08m 11s) * 20:21 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host lvs1016.eqiad.wmnet with OS bullseye * 20:21 sbisson@deploy1003: sbisson: Continuing with deployment * 20:20 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1020.eqiad.wmnet * 20:19 sbisson@deploy1003: sbisson: Backport for [[gerrit:1294342{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344{{!}}frwiki: restrict Article Guidance experiment to junior editors (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be v * 20:17 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294342{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344{{!}}frwiki: restrict Article Guidance experiment to junior editors (T426871)]] * 20:14 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet * 20:05 cmooney@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 12355 * 20:04 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 12355 * 19:51 brett@cumin2002: START - Cookbook sre.hosts.reimage for host lvs1016.eqiad.wmnet with OS bullseye * 19:48 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 19:45 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 19:45 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 19:32 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp6016.drmrs.wmnet,cp[1112,1114].eqiad.wmnet,cp[5024,5031-5032].eqsin.wmnet<nowiki>}</nowiki> and A:cp * 19:32 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5032.eqsin.wmnet * 19:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 19:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 19:01 joal@deploy1003: Finished deploy [analytics/refinery@96cf761] (thin): Regular analytics weekly train THIN [analytics/refinery@96cf761f] (duration: 02m 08s) * 18:59 joal@deploy1003: Started deploy [analytics/refinery@96cf761] (thin): Regular analytics weekly train THIN [analytics/refinery@96cf761f] * 18:58 joal@deploy1003: Finished deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] (duration: 05m 01s) * 18:53 joal@deploy1003: Started deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] * 18:53 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294376{{!}}Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375{{!}}auth: Mark the hidden token field used for reauth as skippable (T427398)]] (duration: 07m 41s) * 18:49 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5031.eqsin.wmnet * 18:49 catrope@deploy1003: catrope: Continuing with deployment * 18:47 catrope@deploy1003: catrope: Backport for [[gerrit:1294376{{!}}Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375{{!}}auth: Mark the hidden token field used for reauth as skippable (T427398)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:45 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1294376{{!}}Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375{{!}}auth: Mark the hidden token field used for reauth as skippable (T427398)]] * 18:40 joal@deploy1003: Finished deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] (duration: 01m 05s) * 18:39 joal@deploy1003: Started deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] * 18:37 joal@deploy1003: Finished deploy [analytics/refinery@96cf761] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96cf761f] (duration: 02m 04s) * 18:35 joal@deploy1003: Started deploy [analytics/refinery@96cf761] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96cf761f] * 18:29 swfrench@deploy1003: Finished scap sync-world: Helmfile-only deployment to clean up unused mesh listeners (duration: 06m 12s) * 18:25 swfrench@deploy1003: swfrench: Continuing with deployment * 18:24 swfrench@deploy1003: swfrench: Helmfile-only deployment to clean up unused mesh listeners synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:23 swfrench@deploy1003: Started scap sync-world: Helmfile-only deployment to clean up unused mesh listeners * 18:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93296 and previous config saved to /var/cache/conftool/dbconfig/20260527-181923-fceratto.json * 18:13 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 18:12 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 18:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 18:11 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 18:11 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 18:10 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 18:10 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 18:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93295 and previous config saved to /var/cache/conftool/dbconfig/20260527-180915-fceratto.json * 18:09 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 18:09 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293776{{!}}ProductionServices: Revert to discovery shellbox listeners]] (duration: 10m 24s) * 18:08 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1017.eqiad.wmnet * 18:08 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1017.eqiad.wmnet * 18:07 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5024.eqsin.wmnet * 18:03 swfrench@deploy1003: swfrench: Continuing with deployment * 18:02 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 18:02 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 18:02 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 18:01 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:01 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 18:01 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 18:00 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 18:00 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293776{{!}}ProductionServices: Revert to discovery shellbox listeners]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:00 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93294 and previous config saved to /var/cache/conftool/dbconfig/20260527-175908-fceratto.json * 17:58 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293776{{!}}ProductionServices: Revert to discovery shellbox listeners]] * 17:55 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93293 and previous config saved to /var/cache/conftool/dbconfig/20260527-174900-fceratto.json * 17:43 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293774{{!}}ProductionServices: Temporarily use shellbox in codfw]] (duration: 15m 01s) * 17:38 swfrench@deploy1003: swfrench: Continuing with deployment * 17:31 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293774{{!}}ProductionServices: Temporarily use shellbox in codfw]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:28 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293774{{!}}ProductionServices: Temporarily use shellbox in codfw]] * 17:25 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp1114.eqiad.wmnet * 17:18 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:15 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:15 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:14 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:05 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293775{{!}}ProductionServices: Temporarily use shellbox in eqiad]] (duration: 08m 44s) * 17:00 swfrench@deploy1003: swfrench: Continuing with deployment * 16:58 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293775{{!}}ProductionServices: Temporarily use shellbox in eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:56 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293775{{!}}ProductionServices: Temporarily use shellbox in eqiad]] * 16:53 atsuko@dns1004: END - running authdns-update * 16:51 atsuko@dns1004: START - running authdns-update * 16:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93292 and previous config saved to /var/cache/conftool/dbconfig/20260527-164846-fceratto.json * 16:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1264.eqiad.wmnet with reason: Maintenance * 16:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93291 and previous config saved to /var/cache/conftool/dbconfig/20260527-164815-fceratto.json * 16:43 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp1112.eqiad.wmnet * 16:41 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1017.eqiad.wmnet with reason: Setting up * 16:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P93290 and previous config saved to /var/cache/conftool/dbconfig/20260527-163808-fceratto.json * 16:37 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2163: Repooling after testing patch * 16:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P93287 and previous config saved to /var/cache/conftool/dbconfig/20260527-162800-fceratto.json * 16:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93285 and previous config saved to /var/cache/conftool/dbconfig/20260527-161753-fceratto.json * 16:14 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 16:13 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 16:13 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 16:12 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 16:11 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93284 and previous config saved to /var/cache/conftool/dbconfig/20260527-161101-fceratto.json * 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: Maintenance * 16:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93283 and previous config saved to /var/cache/conftool/dbconfig/20260527-161034-fceratto.json * 16:10 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 16:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1178: Recovering from failure in cookbook * 16:10 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 16:05 sukhe@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host durum5003.eqsin.wmnet with OS trixie * 16:03 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp6016.drmrs.wmnet * 16:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220', diff saved to https://phabricator.wikimedia.org/P93280 and previous config saved to /var/cache/conftool/dbconfig/20260527-160027-fceratto.json * 15:59 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1017.eqiad.wmnet * 15:53 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2163.codfw.wmnet * 15:53 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2163.codfw.wmnet * 15:52 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs1017.eqiad.wmnet * 15:52 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Repooling after testing patch * 15:52 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp6016.drmrs.wmnet,cp[1112,1114].eqiad.wmnet,cp[5024,5031-5032].eqsin.wmnet<nowiki>}</nowiki> and A:cp * 15:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2163: Testing cookbook * 15:50 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2163: Testing cookbook * 15:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220', diff saved to https://phabricator.wikimedia.org/P93276 and previous config saved to /var/cache/conftool/dbconfig/20260527-155019-fceratto.json * 15:45 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:45 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93274 and previous config saved to /var/cache/conftool/dbconfig/20260527-154011-fceratto.json * 15:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 15:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2163: Migration of db2163.codfw.wmnet completed * 15:32 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Migration of db2163.codfw.wmnet completed * 15:32 cwilliams@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2163: Migration of db2163.codfw.wmnet completed * 15:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1178: Recovering from failure in cookbook * 15:22 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1178.eqiad.wmnet * 15:22 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1178.eqiad.wmnet * 15:19 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 15:19 cdanis: 💙cdanis@cp4047.ulsfo.wmnet ~ 🕦☕ sudo apt install lua5.4-ciderbloom lua5.4-ciderbloom-dbgsym * 15:13 cdanis: 💙cdanis@cp5026.eqsin.wmnet ~ 🕚☕ sudo apt install lua5.4-ciderbloom lua5.4-ciderbloom-dbgsym * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:11 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:11 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:11 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Icinga wait failed during run * 15:10 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:10 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:10 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:09 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:09 cdanis: 💔cdanis@apt1002.wikimedia.org ~ 🕚☕ sudo -i reprepro --component main --restrict cidergrinder update trixie-wikimedia * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:05 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93268 and previous config saved to /var/cache/conftool/dbconfig/20260527-150508-fceratto.json * 15:05 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1220.eqiad.wmnet with reason: Maintenance * 15:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93267 and previous config saved to /var/cache/conftool/dbconfig/20260527-150438-fceratto.json * 14:59 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Migration of db2163.codfw.wmnet completed * 14:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P93264 and previous config saved to /var/cache/conftool/dbconfig/20260527-145430-fceratto.json * 14:54 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 14:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2163.codfw.wmnet with OS trixie * 14:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply * 14:50 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply * 14:46 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290926{{!}}Re-enable ReadingLists QuickSurvey (T426781)]] (duration: 08m 32s) * 14:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1178.eqiad.wmnet with OS trixie * 14:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P93263 and previous config saved to /var/cache/conftool/dbconfig/20260527-144423-fceratto.json * 14:42 aude@deploy1003: aude: Continuing with deployment * 14:40 aude@deploy1003: aude: Backport for [[gerrit:1290926{{!}}Re-enable ReadingLists QuickSurvey (T426781)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:38 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db2189.codfw.wmnet with reason: crashed [[phab:T427376|T427376]] * 14:38 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1290926{{!}}Re-enable ReadingLists QuickSurvey (T426781)]] * 14:35 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290924{{!}}Make logging of title and page ID optional (T426457)]] (duration: 11m 30s) * 14:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93262 and previous config saved to /var/cache/conftool/dbconfig/20260527-143416-fceratto.json * 14:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2163.codfw.wmnet with reason: host reimage * 14:29 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2163.codfw.wmnet with reason: host reimage * 14:29 aude@deploy1003: aude: Continuing with deployment * 14:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1178.eqiad.wmnet with reason: host reimage * 14:27 aude@deploy1003: aude: Backport for [[gerrit:1290924{{!}}Make logging of title and page ID optional (T426457)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:27 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93260 and previous config saved to /var/cache/conftool/dbconfig/20260527-142659-fceratto.json * 14:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1179.eqiad.wmnet with reason: Maintenance * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:23 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1290924{{!}}Make logging of title and page ID optional (T426457)]] * 14:22 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1178.eqiad.wmnet with reason: host reimage * 14:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1033.eqiad.wmnet with reason: Maintenance * 14:18 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294247{{!}}Update Direct Reporting email (T427358)]] (duration: 33m 01s) * 14:10 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2163.codfw.wmnet with OS trixie * 14:09 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1178.eqiad.wmnet with OS trixie * 14:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2163: Upgrading db2163.codfw.wmnet * 14:08 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2163: Upgrading db2163.codfw.wmnet * 14:08 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1178: Upgrading db1178.eqiad.wmnet * 14:07 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1178: Upgrading db1178.eqiad.wmnet * 14:06 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:06 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:06 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:06 stran@deploy1003: stran: Continuing with deployment * 14:02 stran@deploy1003: stran: Backport for [[gerrit:1294247{{!}}Update Direct Reporting email (T427358)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:56 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 13:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2164: Migration of db2164.codfw.wmnet completed * 13:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1192: Migration of db1192.eqiad.wmnet completed * 13:45 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1294247{{!}}Update Direct Reporting email (T427358)]] * 13:40 phuedx@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294217{{!}}ext.wikimediaEvents: Add hoisting error detection test (T427092)]] (duration: 11m 35s) * 13:36 phuedx@deploy1003: phuedx: Continuing with deployment * 13:30 phuedx@deploy1003: phuedx: Backport for [[gerrit:1294217{{!}}ext.wikimediaEvents: Add hoisting error detection test (T427092)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:28 phuedx@deploy1003: Started scap sync-world: Backport for [[gerrit:1294217{{!}}ext.wikimediaEvents: Add hoisting error detection test (T427092)]] * 13:21 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290781{{!}}mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264{{!}}MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] (duration: 13m 23s) * 13:15 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2189: Test * 13:15 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2189: Test * 13:15 mlitn@deploy1003: krinkle, mlitn: Continuing with deployment * 13:13 mlitn@deploy1003: krinkle, mlitn: Backport for [[gerrit:1290781{{!}}mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264{{!}}MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 13:10 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2164: Migration of db2164.codfw.wmnet completed * 13:08 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1290781{{!}}mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264{{!}}MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] * 13:06 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 13:05 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db2212.codfw.wmnet with reason: failed to reboot [[phab:T427388|T427388]] [[phab:T426633|T426633]] * 13:05 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1192: Migration of db1192.eqiad.wmnet completed * 13:01 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2164.codfw.wmnet with OS trixie * 12:57 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1192.eqiad.wmnet with OS trixie * 12:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2164.codfw.wmnet with reason: host reimage * 12:40 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1192.eqiad.wmnet with reason: host reimage * 12:40 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2164.codfw.wmnet with reason: host reimage * 12:35 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1192.eqiad.wmnet with reason: host reimage * 12:28 Amir1: deleting binlogs older than a year * 12:22 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2164.codfw.wmnet with OS trixie * 12:21 cmooney@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 36692 * 12:21 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1192.eqiad.wmnet with OS trixie * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1077 * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1080 * 12:20 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1077 * 12:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2164: Upgrading db2164.codfw.wmnet * 12:20 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 36692 * 12:20 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1080 * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1078 * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1079 * 12:20 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2164: Upgrading db2164.codfw.wmnet * 12:19 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:19 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1079 * 12:19 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1078 * 12:19 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:19 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1192: Upgrading db1192.eqiad.wmnet * 12:19 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:18 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1192: Upgrading db1192.eqiad.wmnet * 12:18 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:15 jclark@cumin1003: START - Cookbook sre.dns.netbox * 12:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2165: Migration of db2165.codfw.wmnet completed * 12:14 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:14 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:14 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:12 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db2189: Test * 12:11 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2189: Test * 12:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1193: Migration of db1193.eqiad.wmnet completed * 12:09 jclark@cumin1003: START - Cookbook sre.dns.netbox * 12:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93243 and previous config saved to /var/cache/conftool/dbconfig/20260527-120452-fceratto.json * 12:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: Maintenance * 12:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93242 and previous config saved to /var/cache/conftool/dbconfig/20260527-120205-fceratto.json * 12:01 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 11:58 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 11:58 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "is everything alright? /cc effie - ayounsi@cumin1003" * 11:58 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "is everything alright? /cc effie - ayounsi@cumin1003" * 11:56 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 11:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P93239 and previous config saved to /var/cache/conftool/dbconfig/20260527-115157-fceratto.json * 11:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P93237 and previous config saved to /var/cache/conftool/dbconfig/20260527-114149-fceratto.json * 11:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93235 and previous config saved to /var/cache/conftool/dbconfig/20260527-113142-fceratto.json * 11:29 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2165: Migration of db2165.codfw.wmnet completed * 11:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1193: Migration of db1193.eqiad.wmnet completed * 11:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93231 and previous config saved to /var/cache/conftool/dbconfig/20260527-112327-fceratto.json * 11:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2188.codfw.wmnet with reason: Maintenance * 11:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93230 and previous config saved to /var/cache/conftool/dbconfig/20260527-112257-fceratto.json * 11:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2165.codfw.wmnet with OS trixie * 11:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1193.eqiad.wmnet with OS trixie * 11:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P93229 and previous config saved to /var/cache/conftool/dbconfig/20260527-111250-fceratto.json * 11:10 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:10 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:08 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:08 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:02 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P93227 and previous config saved to /var/cache/conftool/dbconfig/20260527-110242-fceratto.json * 11:02 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:02 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 11:01 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 11:01 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2165.codfw.wmnet with reason: host reimage * 11:00 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2189', diff saved to https://phabricator.wikimedia.org/P93226 and previous config saved to /var/cache/conftool/dbconfig/20260527-110016-marostegui.json * 10:58 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1193.eqiad.wmnet with reason: host reimage * 10:57 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2165.codfw.wmnet with reason: host reimage * 10:56 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 10:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93225 and previous config saved to /var/cache/conftool/dbconfig/20260527-105235-fceratto.json * 10:52 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1193.eqiad.wmnet with reason: host reimage * 10:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1050: repool after maintenance * 10:45 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2176 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93223 and previous config saved to /var/cache/conftool/dbconfig/20260527-104518-fceratto.json * 10:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2176.codfw.wmnet with reason: Maintenance * 10:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93222 and previous config saved to /var/cache/conftool/dbconfig/20260527-104449-fceratto.json * 10:39 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2165.codfw.wmnet with OS trixie * 10:38 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1193.eqiad.wmnet with OS trixie * 10:36 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1193: Upgrading db1193.eqiad.wmnet * 10:35 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1193: Upgrading db1193.eqiad.wmnet * 10:35 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2165: Upgrading db2165.codfw.wmnet * 10:35 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2165: Upgrading db2165.codfw.wmnet * 10:34 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P93218 and previous config saved to /var/cache/conftool/dbconfig/20260527-103441-fceratto.json * 10:29 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:29 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P93217 and previous config saved to /var/cache/conftool/dbconfig/20260527-102434-fceratto.json * 10:22 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:21 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93215 and previous config saved to /var/cache/conftool/dbconfig/20260527-101426-fceratto.json * 10:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1203: Migration of db1203.eqiad.wmnet completed * 10:10 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2166: Migration of db2166.codfw.wmnet completed * 10:08 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93212 and previous config saved to /var/cache/conftool/dbconfig/20260527-100701-fceratto.json * 10:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance * 10:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93211 and previous config saved to /var/cache/conftool/dbconfig/20260527-100632-fceratto.json * 10:05 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1050: repool after maintenance * 10:04 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:02 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1050.eqiad.wmnet with OS trixie * 09:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P93208 and previous config saved to /var/cache/conftool/dbconfig/20260527-095624-fceratto.json * 09:47 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 09:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P93206 and previous config saved to /var/cache/conftool/dbconfig/20260527-094616-fceratto.json * 09:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1050.eqiad.wmnet with reason: host reimage * 09:43 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 09:41 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1050.eqiad.wmnet with reason: host reimage * 09:38 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 09:38 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 09:37 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 09:37 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 09:36 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 09:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93203 and previous config saved to /var/cache/conftool/dbconfig/20260527-093609-fceratto.json * 09:34 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 09:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93202 and previous config saved to /var/cache/conftool/dbconfig/20260527-092842-fceratto.json * 09:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance * 09:28 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1203: Migration of db1203.eqiad.wmnet completed * 09:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93200 and previous config saved to /var/cache/conftool/dbconfig/20260527-092814-fceratto.json * 09:27 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1050.eqiad.wmnet with OS trixie * 09:26 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1050: Upgrading es1050.eqiad.wmnet * 09:25 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1050: Upgrading es1050.eqiad.wmnet * 09:25 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1050: repool after maintenance * 09:25 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1050: repool after maintenance * 09:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2166: Migration of db2166.codfw.wmnet completed * 09:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2051: repool after maintenance * 09:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1203.eqiad.wmnet with OS trixie * 09:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P93196 and previous config saved to /var/cache/conftool/dbconfig/20260527-091806-fceratto.json * 09:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2166.codfw.wmnet with OS trixie * 09:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P93194 and previous config saved to /var/cache/conftool/dbconfig/20260527-090759-fceratto.json * 09:03 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp3074.* * 09:03 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp3066.* * 09:03 fabfur: repooling cp3074 and cp3066 ([[phab:T419825|T419825]]) * 09:02 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp6015.drmrs.wmnet * 09:02 slyngshede@cumin1003: START - Cookbook sre.hosts.remove-downtime for cp6015.drmrs.wmnet * 09:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1203.eqiad.wmnet with reason: host reimage * 09:02 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: name=cp6015.* * 08:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2166.codfw.wmnet with reason: host reimage * 08:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93193 and previous config saved to /var/cache/conftool/dbconfig/20260527-085751-fceratto.json * 08:55 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1203.eqiad.wmnet with reason: host reimage * 08:54 Emperor: restart swift on ms-fe2011 [[phab:T360913|T360913]] * 08:54 jayme@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:54 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2166.codfw.wmnet with reason: host reimage * 08:54 jayme@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 08:51 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 08:51 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 08:51 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp3066.* * 08:51 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp3074.* * 08:51 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 08:50 fabfur: depooling and installing haproxy-awslc on cp3074 and cp3066 ([[phab:T419825|T419825]]) * 08:50 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93191 and previous config saved to /var/cache/conftool/dbconfig/20260527-085024-fceratto.json * 08:50 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance * 08:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93190 and previous config saved to /var/cache/conftool/dbconfig/20260527-085005-fceratto.json * 08:41 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1203.eqiad.wmnet with OS trixie * 08:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P93189 and previous config saved to /var/cache/conftool/dbconfig/20260527-083957-fceratto.json * 08:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2051: repool after maintenance * 08:37 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 08:36 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1203: Upgrading db1203.eqiad.wmnet * 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1004.wikimedia.org * 08:36 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1203: Upgrading db1203.eqiad.wmnet * 08:36 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:35 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2166.codfw.wmnet with OS trixie * 08:35 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2051.codfw.wmnet with OS trixie * 08:34 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2166: Upgrading db2166.codfw.wmnet * 08:33 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2166: Upgrading db2166.codfw.wmnet * 08:33 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader1004.wikimedia.org * 08:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org * 08:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P93185 and previous config saved to /var/cache/conftool/dbconfig/20260527-082950-fceratto.json * 08:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org * 08:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93184 and previous config saved to /var/cache/conftool/dbconfig/20260527-081942-fceratto.json * 08:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2051.codfw.wmnet with reason: host reimage * 08:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2051.codfw.wmnet with reason: host reimage * 08:11 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 08:11 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2153 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93183 and previous config saved to /var/cache/conftool/dbconfig/20260527-081112-fceratto.json * 08:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance * 08:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93182 and previous config saved to /var/cache/conftool/dbconfig/20260527-081054-fceratto.json * 08:07 jmm@dns1004: END - running authdns-update * 08:05 jmm@dns1004: START - running authdns-update * 08:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248', diff saved to https://phabricator.wikimedia.org/P93181 and previous config saved to /var/cache/conftool/dbconfig/20260527-080046-fceratto.json * 07:59 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2051.codfw.wmnet with OS trixie * 07:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248', diff saved to https://phabricator.wikimedia.org/P93180 and previous config saved to /var/cache/conftool/dbconfig/20260527-075039-fceratto.json * 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1026.eqiad.wmnet * 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1026.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:43 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1026.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2051: Upgrading es2051.codfw.wmnet * 07:42 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2051: Upgrading es2051.codfw.wmnet * 07:41 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93178 and previous config saved to /var/cache/conftool/dbconfig/20260527-074031-fceratto.json * 07:40 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294125{{!}}Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126{{!}}Add script to demote ineligible members of restricted global groups (T425395)]] (duration: 06m 42s) * 07:36 mszwarc@deploy1003: mszwarc: Continuing with deployment * 07:35 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1294125{{!}}Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126{{!}}Add script to demote ineligible members of restricted global groups (T425395)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93177 and previous config saved to /var/cache/conftool/dbconfig/20260527-073504-fceratto.json * 07:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2248.codfw.wmnet with reason: Maintenance * 07:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93176 and previous config saved to /var/cache/conftool/dbconfig/20260527-073434-fceratto.json * 07:33 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1294125{{!}}Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126{{!}}Add script to demote ineligible members of restricted global groups (T425395)]] * 07:28 jmm@cumin2002: START - Cookbook sre.dns.netbox * 07:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P93175 and previous config saved to /var/cache/conftool/dbconfig/20260527-072426-fceratto.json * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) * 07:23 marostegui@cumin1003: Removing pc1014 from zarcillo [[phab:T427190|T427190]] * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1014.eqiad.wmnet * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 07:23 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 07:18 marostegui@cumin1003: START - Cookbook sre.dns.netbox * 07:15 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1026.eqiad.wmnet * 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1025.eqiad.wmnet * 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P93174 and previous config saved to /var/cache/conftool/dbconfig/20260527-071418-fceratto.json * 07:13 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1014.eqiad.wmnet * 07:13 marostegui@cumin1003: START - Cookbook sre.mysql.decommission * 07:13 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org * 07:07 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2055: repool after maintenance * 07:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org * 07:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1003.wikimedia.org * 07:06 jmm@cumin2002: START - Cookbook sre.dns.netbox * 07:06 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1190.eqiad.wmnet with reason: Maintenance on db1190 * 07:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93172 and previous config saved to /var/cache/conftool/dbconfig/20260527-070410-fceratto.json * 07:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader1003.wikimedia.org * 06:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93171 and previous config saved to /var/cache/conftool/dbconfig/20260527-065545-fceratto.json * 06:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2247.codfw.wmnet with reason: Maintenance * 06:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93170 and previous config saved to /var/cache/conftool/dbconfig/20260527-065526-fceratto.json * 06:54 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1025.eqiad.wmnet * 06:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P93168 and previous config saved to /var/cache/conftool/dbconfig/20260527-064519-fceratto.json * 06:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P93166 and previous config saved to /var/cache/conftool/dbconfig/20260527-063511-fceratto.json * 06:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93165 and previous config saved to /var/cache/conftool/dbconfig/20260527-062503-fceratto.json * 06:22 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2055: repool after maintenance * 06:21 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 06:21 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2055.codfw.wmnet with OS trixie * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2246 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93163 and previous config saved to /var/cache/conftool/dbconfig/20260527-061643-fceratto.json * 06:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2246.codfw.wmnet with reason: Maintenance * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93162 and previous config saved to /var/cache/conftool/dbconfig/20260527-061613-fceratto.json * 06:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P93161 and previous config saved to /var/cache/conftool/dbconfig/20260527-060606-fceratto.json * 06:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2055.codfw.wmnet with reason: host reimage * 05:56 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2055.codfw.wmnet with reason: host reimage * 05:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P93160 and previous config saved to /var/cache/conftool/dbconfig/20260527-055558-fceratto.json * 05:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93159 and previous config saved to /var/cache/conftool/dbconfig/20260527-054550-fceratto.json * 05:41 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2055.codfw.wmnet with OS trixie * 05:40 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2055: Upgrading es2055.codfw.wmnet * 05:40 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2055: Upgrading es2055.codfw.wmnet * 05:40 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:38 moritzm: remove ganeti1026 from eqiad Ganeti cluster [[phab:T424680|T424680]] * 05:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2245 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93157 and previous config saved to /var/cache/conftool/dbconfig/20260527-053727-fceratto.json * 05:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: Maintenance * 05:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93156 and previous config saved to /var/cache/conftool/dbconfig/20260527-053708-fceratto.json * 05:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P93155 and previous config saved to /var/cache/conftool/dbconfig/20260527-052700-fceratto.json * 05:26 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1014 from dbctl [[phab:T427270|T427270]]', diff saved to https://phabricator.wikimedia.org/P93154 and previous config saved to /var/cache/conftool/dbconfig/20260527-052624-marostegui.json * 05:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P93153 and previous config saved to /var/cache/conftool/dbconfig/20260527-051653-fceratto.json * 05:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93152 and previous config saved to /var/cache/conftool/dbconfig/20260527-050645-fceratto.json * 04:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2237 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93151 and previous config saved to /var/cache/conftool/dbconfig/20260527-045827-fceratto.json * 04:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: Maintenance * 04:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93150 and previous config saved to /var/cache/conftool/dbconfig/20260527-045759-fceratto.json * 04:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P93149 and previous config saved to /var/cache/conftool/dbconfig/20260527-044751-fceratto.json * 04:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P93148 and previous config saved to /var/cache/conftool/dbconfig/20260527-043744-fceratto.json * 04:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93147 and previous config saved to /var/cache/conftool/dbconfig/20260527-042737-fceratto.json * 04:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2236 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93146 and previous config saved to /var/cache/conftool/dbconfig/20260527-041921-fceratto.json * 04:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2236.codfw.wmnet with reason: Maintenance * 04:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93145 and previous config saved to /var/cache/conftool/dbconfig/20260527-041852-fceratto.json * 04:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P93144 and previous config saved to /var/cache/conftool/dbconfig/20260527-040844-fceratto.json * 03:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P93143 and previous config saved to /var/cache/conftool/dbconfig/20260527-035836-fceratto.json * 03:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93142 and previous config saved to /var/cache/conftool/dbconfig/20260527-034828-fceratto.json * 03:40 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93141 and previous config saved to /var/cache/conftool/dbconfig/20260527-034008-fceratto.json * 03:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2219.codfw.wmnet with reason: Maintenance * 03:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93140 and previous config saved to /var/cache/conftool/dbconfig/20260527-033938-fceratto.json * 03:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P93139 and previous config saved to /var/cache/conftool/dbconfig/20260527-032931-fceratto.json * 03:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P93138 and previous config saved to /var/cache/conftool/dbconfig/20260527-031923-fceratto.json * 03:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93137 and previous config saved to /var/cache/conftool/dbconfig/20260527-030915-fceratto.json * 03:00 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93136 and previous config saved to /var/cache/conftool/dbconfig/20260527-030045-fceratto.json * 03:00 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2210.codfw.wmnet with reason: Maintenance * 03:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93135 and previous config saved to /var/cache/conftool/dbconfig/20260527-030016-fceratto.json * 02:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P93134 and previous config saved to /var/cache/conftool/dbconfig/20260527-025008-fceratto.json * 02:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P93133 and previous config saved to /var/cache/conftool/dbconfig/20260527-024000-fceratto.json * 02:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93132 and previous config saved to /var/cache/conftool/dbconfig/20260527-022953-fceratto.json * 02:21 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93131 and previous config saved to /var/cache/conftool/dbconfig/20260527-022133-fceratto.json * 02:21 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2206.codfw.wmnet with reason: Maintenance * 02:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93130 and previous config saved to /var/cache/conftool/dbconfig/20260527-022100-fceratto.json * 02:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P93129 and previous config saved to /var/cache/conftool/dbconfig/20260527-021053-fceratto.json * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 29s) * 02:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P93128 and previous config saved to /var/cache/conftool/dbconfig/20260527-020045-fceratto.json * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93127 and previous config saved to /var/cache/conftool/dbconfig/20260527-015037-fceratto.json * 01:42 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93126 and previous config saved to /var/cache/conftool/dbconfig/20260527-014204-fceratto.json * 01:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2179.codfw.wmnet with reason: Maintenance * 01:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93125 and previous config saved to /var/cache/conftool/dbconfig/20260527-014134-fceratto.json * 01:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P93124 and previous config saved to /var/cache/conftool/dbconfig/20260527-013126-fceratto.json * 01:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P93123 and previous config saved to /var/cache/conftool/dbconfig/20260527-012119-fceratto.json * 01:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93122 and previous config saved to /var/cache/conftool/dbconfig/20260527-011111-fceratto.json * 01:02 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2172 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93121 and previous config saved to /var/cache/conftool/dbconfig/20260527-010234-fceratto.json * 01:02 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance * 01:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93120 and previous config saved to /var/cache/conftool/dbconfig/20260527-010205-fceratto.json * 00:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P93119 and previous config saved to /var/cache/conftool/dbconfig/20260527-005157-fceratto.json * 00:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P93118 and previous config saved to /var/cache/conftool/dbconfig/20260527-004149-fceratto.json * 00:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93117 and previous config saved to /var/cache/conftool/dbconfig/20260527-003141-fceratto.json * 00:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2155 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93116 and previous config saved to /var/cache/conftool/dbconfig/20260527-002309-fceratto.json * 00:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance * 00:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93115 and previous config saved to /var/cache/conftool/dbconfig/20260527-002228-fceratto.json * 00:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P93114 and previous config saved to /var/cache/conftool/dbconfig/20260527-001220-fceratto.json * 00:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P93113 and previous config saved to /var/cache/conftool/dbconfig/20260527-000209-fceratto.json == 2026-05-26 == * 23:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93112 and previous config saved to /var/cache/conftool/dbconfig/20260526-235201-fceratto.json * 23:44 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93111 and previous config saved to /var/cache/conftool/dbconfig/20260526-234451-fceratto.json * 23:44 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance * 23:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93110 and previous config saved to /var/cache/conftool/dbconfig/20260526-234421-fceratto.json * 23:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P93109 and previous config saved to /var/cache/conftool/dbconfig/20260526-233414-fceratto.json * 23:27 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5026.* * 23:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P93108 and previous config saved to /var/cache/conftool/dbconfig/20260526-232406-fceratto.json * 23:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93107 and previous config saved to /var/cache/conftool/dbconfig/20260526-231358-fceratto.json * 23:07 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5026.* * 23:06 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93106 and previous config saved to /var/cache/conftool/dbconfig/20260526-230650-fceratto.json * 23:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance * 23:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93105 and previous config saved to /var/cache/conftool/dbconfig/20260526-230620-fceratto.json * 22:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P93104 and previous config saved to /var/cache/conftool/dbconfig/20260526-225612-fceratto.json * 22:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P93103 and previous config saved to /var/cache/conftool/dbconfig/20260526-224604-fceratto.json * 22:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93101 and previous config saved to /var/cache/conftool/dbconfig/20260526-223556-fceratto.json * 22:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93100 and previous config saved to /var/cache/conftool/dbconfig/20260526-222848-fceratto.json * 22:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance * 22:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93099 and previous config saved to /var/cache/conftool/dbconfig/20260526-222828-fceratto.json * 22:23 robh@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts cp6015.drmrs.wmnet * 22:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P93098 and previous config saved to /var/cache/conftool/dbconfig/20260526-221819-fceratto.json * 22:10 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1009.eqiad.wmnet with OS trixie * 22:08 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1008.eqiad.wmnet with OS trixie * 22:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P93097 and previous config saved to /var/cache/conftool/dbconfig/20260526-220811-fceratto.json * 22:04 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293701{{!}}MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] (duration: 09m 30s) * 22:03 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1009.eqiad.wmnet with reason: host reimage * 22:00 egardner@deploy1003: egardner, mfossati: Continuing with deployment * 21:59 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1008.eqiad.wmnet with reason: host reimage * 21:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93096 and previous config saved to /var/cache/conftool/dbconfig/20260526-215803-fceratto.json * 21:57 egardner@deploy1003: egardner, mfossati: Backport for [[gerrit:1293701{{!}}MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:56 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp6015.drmrs.wmnet * 21:56 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1010.eqiad.wmnet with OS trixie * 21:56 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp6015.drmrs.wmnet * 21:55 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1293701{{!}}MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] * 21:54 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1009.eqiad.wmnet with reason: host reimage * 21:51 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1008.eqiad.wmnet with reason: host reimage * 21:50 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93095 and previous config saved to /var/cache/conftool/dbconfig/20260526-215043-fceratto.json * 21:50 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance * 21:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93094 and previous config saved to /var/cache/conftool/dbconfig/20260526-215011-fceratto.json * 21:49 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1010.eqiad.wmnet with reason: host reimage * 21:47 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp6015.drmrs.wmnet * 21:44 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1009 * 21:44 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host relforge1009 * 21:43 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host relforge1009 * 21:43 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) relforge1009.eqiad.wmnet 120.48.64.10.in-addr.arpa 0.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:43 bking@cumin2002: START - Cookbook sre.dns.wipe-cache relforge1009.eqiad.wmnet 120.48.64.10.in-addr.arpa 0.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:43 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:42 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1009 - bking@cumin2002" * 21:42 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1010.eqiad.wmnet with reason: host reimage * 21:42 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1009 - bking@cumin2002" * 21:41 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1008 * 21:40 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host relforge1008 * 21:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P93093 and previous config saved to /var/cache/conftool/dbconfig/20260526-214003-fceratto.json * 21:36 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host relforge1008 * 21:36 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) relforge1008.eqiad.wmnet 100.32.64.10.in-addr.arpa 0.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:36 bking@cumin2002: START - Cookbook sre.dns.wipe-cache relforge1008.eqiad.wmnet 100.32.64.10.in-addr.arpa 0.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:36 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:36 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1008 - bking@cumin2002" * 21:36 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1008 - bking@cumin2002" * 21:35 bking@cumin2002: START - Cookbook sre.dns.netbox * 21:32 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1010 * 21:32 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1010 * 21:31 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1010.eqiad.wmnet with OS trixie * 21:31 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1009 * 21:30 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1009.eqiad.wmnet with OS trixie * 21:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P93092 and previous config saved to /var/cache/conftool/dbconfig/20260526-212955-fceratto.json * 21:29 bking@cumin2002: START - Cookbook sre.dns.netbox * 21:29 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1008 * 21:29 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1008.eqiad.wmnet with OS trixie * 21:27 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist "all.dblist - mediamoderation-continuous-scan.dblist - preinstall.dblist" extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` in tmux session - [[phab:T421688|T421688]] * 21:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93091 and previous config saved to /var/cache/conftool/dbconfig/20260526-211948-fceratto.json * 21:19 jhathaway: dmarc ingress test run mx-in1001 * 21:15 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_codfw and A:cp * 21:15 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2057.codfw.wmnet * 21:14 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_codfw and A:cp * 21:14 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2058.codfw.wmnet * 21:12 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2222 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93090 and previous config saved to /var/cache/conftool/dbconfig/20260526-211238-fceratto.json * 21:12 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance * 21:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93089 and previous config saved to /var/cache/conftool/dbconfig/20260526-211207-fceratto.json * 21:06 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 21:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P93088 and previous config saved to /var/cache/conftool/dbconfig/20260526-210159-fceratto.json * 20:55 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on phab2003.codfw.wmnet with reason: WIP * 20:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P93087 and previous config saved to /var/cache/conftool/dbconfig/20260526-205152-fceratto.json * 20:50 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:50 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 20:50 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 20:45 dzahn@cumin2002: START - Cookbook sre.dns.netbox * 20:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93086 and previous config saved to /var/cache/conftool/dbconfig/20260526-204143-fceratto.json * 20:38 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2055.codfw.wmnet * 20:34 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93085 and previous config saved to /var/cache/conftool/dbconfig/20260526-203430-fceratto.json * 20:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance * 20:34 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2056.codfw.wmnet * 20:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93084 and previous config saved to /var/cache/conftool/dbconfig/20260526-203357-fceratto.json * 20:32 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:32 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:32 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:31 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P93083 and previous config saved to /var/cache/conftool/dbconfig/20260526-202349-fceratto.json * 20:18 alexsanford@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293161{{!}}Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794{{!}}Re-enable ReadingLists survey on beta cluster (T426781)]] (duration: 09m 14s) * 20:14 alexsanford@deploy1003: alexsanford, aude: Continuing with deployment * 20:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P93082 and previous config saved to /var/cache/conftool/dbconfig/20260526-201341-fceratto.json * 20:11 alexsanford@deploy1003: alexsanford, aude: Backport for [[gerrit:1293161{{!}}Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794{{!}}Re-enable ReadingLists survey on beta cluster (T426781)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:09 alexsanford@deploy1003: Started scap sync-world: Backport for [[gerrit:1293161{{!}}Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794{{!}}Re-enable ReadingLists survey on beta cluster (T426781)]] * 20:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93081 and previous config saved to /var/cache/conftool/dbconfig/20260526-200333-fceratto.json * 19:59 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2053.codfw.wmnet * 19:58 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2029.codfw.wmnet with OS trixie * 19:57 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2028.codfw.wmnet with OS trixie * 19:56 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2208 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93080 and previous config saved to /var/cache/conftool/dbconfig/20260526-195632-fceratto.json * 19:56 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance * 19:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93079 and previous config saved to /var/cache/conftool/dbconfig/20260526-195557-fceratto.json * 19:55 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2054.codfw.wmnet * 19:51 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:51 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P93078 and previous config saved to /var/cache/conftool/dbconfig/20260526-194549-fceratto.json * 19:45 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 19:44 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:43 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2029 * 19:43 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 * 19:43 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 * 19:43 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 19:40 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb2014.codfw.wmnet with OS trixie * 19:40 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:40 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb2013.codfw.wmnet with OS trixie * 19:40 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:39 brett@cumin2002: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 19:38 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 19:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P93077 and previous config saved to /var/cache/conftool/dbconfig/20260526-193541-fceratto.json * 19:35 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:35 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 19:30 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 19:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93076 and previous config saved to /var/cache/conftool/dbconfig/20260526-192533-fceratto.json * 19:24 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:21 dzahn@cumin2002: START - Cookbook sre.dns.netbox * 19:20 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2051.codfw.wmnet * 19:19 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:19 brett@cumin2002: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 19:18 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93075 and previous config saved to /var/cache/conftool/dbconfig/20260526-191818-fceratto.json * 19:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance * 19:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93074 and previous config saved to /var/cache/conftool/dbconfig/20260526-191748-fceratto.json * 19:16 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2052.codfw.wmnet * 19:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P93073 and previous config saved to /var/cache/conftool/dbconfig/20260526-190740-fceratto.json * 19:07 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb2014.codfw.wmnet with reason: host reimage * 19:03 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb2013.codfw.wmnet with reason: host reimage * 18:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1026.eqiad.wmnet * 18:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P93072 and previous config saved to /var/cache/conftool/dbconfig/20260526-185732-fceratto.json * 18:56 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2014.codfw.wmnet with reason: host reimage * 18:56 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2013.codfw.wmnet with reason: host reimage * 18:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93071 and previous config saved to /var/cache/conftool/dbconfig/20260526-184724-fceratto.json * 18:44 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb2014.codfw.wmnet with OS trixie * 18:43 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb2013.codfw.wmnet with OS trixie * 18:41 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host rdb2014.codfw.wmnet with OS trixie * 18:41 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2049.codfw.wmnet * 18:40 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93070 and previous config saved to /var/cache/conftool/dbconfig/20260526-184009-fceratto.json * 18:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance * 18:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93069 and previous config saved to /var/cache/conftool/dbconfig/20260526-183939-fceratto.json * 18:37 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2050.codfw.wmnet * 18:30 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 18:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P93068 and previous config saved to /var/cache/conftool/dbconfig/20260526-182931-fceratto.json * 18:29 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:29 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_magru-v4 - dzahn@cumin2002" * 18:29 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_magru-v4 - dzahn@cumin2002" * 18:24 dzahn@cumin2002: START - Cookbook sre.dns.netbox * 18:21 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 18:21 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 18:21 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 18:20 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 18:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P93066 and previous config saved to /var/cache/conftool/dbconfig/20260526-181923-fceratto.json * 18:15 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 18:15 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 18:15 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 18:15 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 18:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93065 and previous config saved to /var/cache/conftool/dbconfig/20260526-180915-fceratto.json * 18:02 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93064 and previous config saved to /var/cache/conftool/dbconfig/20260526-180205-fceratto.json * 18:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2159.codfw.wmnet with reason: Maintenance * 18:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93063 and previous config saved to /var/cache/conftool/dbconfig/20260526-180132-fceratto.json * 18:00 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2047.codfw.wmnet * 17:59 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2048.codfw.wmnet * 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:54 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:54 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:54 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P93062 and previous config saved to /var/cache/conftool/dbconfig/20260526-175124-fceratto.json * 17:42 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293779{{!}}Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] (duration: 07m 25s) * 17:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P93060 and previous config saved to /var/cache/conftool/dbconfig/20260526-174117-fceratto.json * 17:39 mvernon@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ms-be2089.codfw.wmnet * 17:37 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 17:37 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:36 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:36 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:36 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1293779{{!}}Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:36 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:34 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1293779{{!}}Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] * 17:33 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:33 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:33 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:33 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93059 and previous config saved to /var/cache/conftool/dbconfig/20260526-173109-fceratto.json * 17:27 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:26 jclark@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:25 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:25 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:25 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:24 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:24 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1001 to eqiad - jclark@cumin1003" * 17:24 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:24 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1001 to eqiad - jclark@cumin1003" * 17:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93058 and previous config saved to /var/cache/conftool/dbconfig/20260526-172332-fceratto.json * 17:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2227.codfw.wmnet with reason: Maintenance * 17:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93057 and previous config saved to /var/cache/conftool/dbconfig/20260526-172303-fceratto.json * 17:21 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2045.codfw.wmnet * 17:20 jclark@cumin1003: START - Cookbook sre.dns.netbox * 17:20 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2046.codfw.wmnet * 17:18 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:17 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:17 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:17 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:17 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:17 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:17 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:17 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:16 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:15 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 17:14 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:14 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:14 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:14 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:13 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:13 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:13 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:13 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:13 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P93056 and previous config saved to /var/cache/conftool/dbconfig/20260526-171255-fceratto.json * 17:11 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:11 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:11 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:11 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:07 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:05 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:05 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:05 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:05 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P93055 and previous config saved to /var/cache/conftool/dbconfig/20260526-170247-fceratto.json * 17:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:57 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:55 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:52 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93054 and previous config saved to /var/cache/conftool/dbconfig/20260526-165240-fceratto.json * 16:50 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:50 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:50 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:50 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:45 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:45 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:45 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:45 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:45 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:44 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:44 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2209 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93053 and previous config saved to /var/cache/conftool/dbconfig/20260526-164421-fceratto.json * 16:44 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:44 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1002 to eqiad - jclark@cumin1003" * 16:44 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2209.codfw.wmnet with reason: Maintenance * 16:44 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1002 to eqiad - jclark@cumin1003" * 16:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93052 and previous config saved to /var/cache/conftool/dbconfig/20260526-164352-fceratto.json * 16:42 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2043.codfw.wmnet * 16:41 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2044.codfw.wmnet * 16:40 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:40 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:40 jclark@cumin1003: START - Cookbook sre.dns.netbox * 16:40 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:40 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:40 brett: reboot lvs 101[345].eqiad.wmnet * 16:39 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:39 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:39 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:39 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:37 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:37 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 16:37 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:37 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:37 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:37 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 16:37 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 16:36 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 16:36 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:36 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:36 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:36 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:35 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 16:34 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 16:34 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:34 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:34 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:34 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:33 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_codfw and A:cp * 16:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P93051 and previous config saved to /var/cache/conftool/dbconfig/20260526-163344-fceratto.json * 16:33 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_codfw and A:cp * 16:31 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:31 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:30 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:30 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P93050 and previous config saved to /var/cache/conftool/dbconfig/20260526-162336-fceratto.json * 16:13 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2089.codfw.wmnet * 16:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93049 and previous config saved to /var/cache/conftool/dbconfig/20260526-161328-fceratto.json * 16:11 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:11 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:10 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:10 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:07 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search,name=eqiad * 16:06 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:06 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:06 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:06 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93047 and previous config saved to /var/cache/conftool/dbconfig/20260526-160450-fceratto.json * 16:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2194.codfw.wmnet with reason: Maintenance * 16:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93046 and previous config saved to /var/cache/conftool/dbconfig/20260526-160420-fceratto.json * 16:03 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:03 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:03 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] (duration: 00m 28s) * 16:02 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] * 16:00 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:00 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:00 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:00 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:55 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:55 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:55 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:55 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] (duration: 00m 22s) * 15:55 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:55 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] * 15:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P93045 and previous config saved to /var/cache/conftool/dbconfig/20260526-155413-fceratto.json * 15:46 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search,name=eqiad * 15:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P93044 and previous config saved to /var/cache/conftool/dbconfig/20260526-154405-fceratto.json * 15:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93043 and previous config saved to /var/cache/conftool/dbconfig/20260526-153357-fceratto.json * 15:30 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:30 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:30 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:30 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93042 and previous config saved to /var/cache/conftool/dbconfig/20260526-152629-fceratto.json * 15:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2190.codfw.wmnet with reason: Maintenance * 15:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93041 and previous config saved to /var/cache/conftool/dbconfig/20260526-152559-fceratto.json * 15:24 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:24 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:24 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:24 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:23 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:22 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:22 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:22 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P93040 and previous config saved to /var/cache/conftool/dbconfig/20260526-151552-fceratto.json * 15:12 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2196: Rack maintenance completed * 15:10 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2196.codfw.wmnet * 15:10 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2196.codfw.wmnet * 15:07 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw * 15:06 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2222: Rack maintenance completed * 15:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P93037 and previous config saved to /var/cache/conftool/dbconfig/20260526-150546-fceratto.json * 15:04 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2221: Rack maintenance completed * 15:04 brennen@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab1004 for [[phab:T427286|T427286]] (duration: 00m 39s) * 15:03 brennen@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab1004 for [[phab:T427286|T427286]] * 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2002 for [[phab:T427286|T427286]] (duration: 00m 45s) * 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2002 for [[phab:T427286|T427286]] * 15:02 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2002.codfw.wmnet with reason: Phabricator deploy * 15:01 bjensen: uploading prometheus-memcached-exporter_0.16.0-1_amd64 on apt1002 * 15:01 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator deploy * 15:00 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2223: switch maintenance * 14:56 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2196: Rack maintenance completed * 14:55 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2221.codfw.wmnet * 14:55 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2221.codfw.wmnet * 14:55 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2222.codfw.wmnet * 14:55 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2222.codfw.wmnet * 14:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93033 and previous config saved to /var/cache/conftool/dbconfig/20260526-145538-fceratto.json * 14:55 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet * 14:54 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1026.eqiad.wmnet * 14:52 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet * 14:52 moritzm: remove ganeti1025 from eqiad Ganeti cluster [[phab:T424680|T424680]] * 14:51 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2030.codfw.wmnet to cluster codfw and group A * 14:51 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2222: Rack maintenance completed * 14:49 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:49 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2221: Rack maintenance completed * 14:49 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2030.codfw.wmnet to cluster codfw and group A * 14:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2029.codfw.wmnet to cluster codfw and group A * 14:47 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2029.codfw.wmnet to cluster codfw and group A * 14:47 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2177 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93030 and previous config saved to /var/cache/conftool/dbconfig/20260526-144718-fceratto.json * 14:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance * 14:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93029 and previous config saved to /var/cache/conftool/dbconfig/20260526-144651-fceratto.json * 14:45 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=wdqs-scholarly,name=codfw * 14:45 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=wdqs-scholarly,name=codfw * 14:43 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw * 14:40 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:40 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2167: Migration of db2167.codfw.wmnet completed * 14:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P93026 and previous config saved to /var/cache/conftool/dbconfig/20260526-143643-fceratto.json * 14:31 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1054.eqiad.wmnet with OS trixie * 14:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P93023 and previous config saved to /var/cache/conftool/dbconfig/20260526-142636-fceratto.json * 14:26 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:25 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:24 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1014: Rack maintenance completed * 14:24 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.parsercache (exit_code=99) * 14:24 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache * 14:24 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc1014: Rack maintenance completed * 14:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1025.eqiad.wmnet * 14:19 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for backup2015.codfw.wmnet,db2197.codfw.wmnet * 14:19 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for backup2015.codfw.wmnet,db2197.codfw.wmnet * 14:18 jynus: restarting mediabackups@codfw after maintenance on a codfw backup media storage server [[phab:T426199|T426199]] * 14:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93021 and previous config saved to /var/cache/conftool/dbconfig/20260526-141628-fceratto.json * 14:16 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:14 fabfur: repooled cp2043 ([[phab:T426199|T426199]]) * 14:14 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2223: switch maintenance * 14:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1054.eqiad.wmnet with reason: host reimage * 14:14 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2043.* * 14:13 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293710{{!}}Site info should output thumblimits as array (T427066)]] (duration: 06m 40s) * 14:12 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:10 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1054.eqiad.wmnet with reason: host reimage * 14:10 fabfur@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2011.codfw.wmnet * 14:10 fabfur@cumin1003: START - Cookbook sre.hosts.remove-downtime for lvs2011.codfw.wmnet * 14:09 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 14:09 fabfur: restoring lvs2011 as primary ([[phab:T426199|T426199]]) * 14:08 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1293710{{!}}Site info should output thumblimits as array (T427066)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:08 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 14:08 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 14:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93017 and previous config saved to /var/cache/conftool/dbconfig/20260526-140748-fceratto.json * 14:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance * 14:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93016 and previous config saved to /var/cache/conftool/dbconfig/20260526-140718-fceratto.json * 14:07 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1293710{{!}}Site info should output thumblimits as array (T427066)]] * 14:05 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.decommission (exit_code=99) * 14:05 marostegui@cumin1003: Removing pc1013 from zarcillo [[phab:T427190|T427190]] * 14:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1013.eqiad.wmnet * 14:04 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:04 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 14:04 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 14:00 marostegui@cumin1003: START - Cookbook sre.dns.netbox * 13:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P93014 and previous config saved to /var/cache/conftool/dbconfig/20260526-135711-fceratto.json * 13:56 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1054.eqiad.wmnet with OS trixie * 13:55 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2167: Migration of db2167.codfw.wmnet completed * 13:53 Amir1: drop flaggedrevs tables on cawikinews ([[phab:T423577|T423577]]) * 13:49 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1013.eqiad.wmnet * 13:49 marostegui@cumin1003: START - Cookbook sre.mysql.decommission * 13:48 Lucas_WMDE: UTC afternoon backport+config window done * 13:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P93012 and previous config saved to /var/cache/conftool/dbconfig/20260526-134703-fceratto.json * 13:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2167.codfw.wmnet with OS trixie * 13:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93011 and previous config saved to /var/cache/conftool/dbconfig/20260526-133656-fceratto.json * 13:36 XioNoX: reboot lsw1-a2-codfw for software upgrade - [[phab:T426199|T426199]] * 13:36 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2223: switch maintenance * 13:35 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2223: switch maintenance * 13:35 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2222: switch maintenance * 13:35 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2222: switch maintenance * 13:35 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2221: switch maintenance * 13:35 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293662{{!}}Enable IRS Direct Reporting on testwiki (T425025)]] (duration: 09m 28s) * 13:34 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2221: switch maintenance * 13:34 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2196: switch maintenance * 13:34 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2196: switch maintenance * 13:31 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 13:30 stran@deploy1003: stran: Continuing with deployment * 13:29 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 13:29 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93006 and previous config saved to /var/cache/conftool/dbconfig/20260526-132927-fceratto.json * 13:29 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2167.codfw.wmnet with reason: host reimage * 13:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance * 13:29 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 34 hosts with reason: Switch maintenance * 13:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93005 and previous config saved to /var/cache/conftool/dbconfig/20260526-132857-fceratto.json * 13:28 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lsw1-a2-codfw,lsw1-a2-codfw IPv6,lsw1-a2-codfw.mgmt with reason: Switch maintenance * 13:27 stran@deploy1003: stran: Backport for [[gerrit:1293662{{!}}Enable IRS Direct Reporting on testwiki (T425025)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:25 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1293662{{!}}Enable IRS Direct Reporting on testwiki (T425025)]] * 13:25 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2167.codfw.wmnet with reason: host reimage * 13:22 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293706{{!}}Disable the `no` language code for translation (T424613)]] (duration: 08m 30s) * 13:22 ladsgroup@dns1004: END - running authdns-update * 13:20 ladsgroup@dns1004: START - running authdns-update * 13:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P93004 and previous config saved to /var/cache/conftool/dbconfig/20260526-131850-fceratto.json * 13:18 lucaswerkmeister-wmde@deploy1003: jhsoby, lucaswerkmeister-wmde: Continuing with deployment * 13:16 lucaswerkmeister-wmde@deploy1003: jhsoby, lucaswerkmeister-wmde: Backport for [[gerrit:1293706{{!}}Disable the `no` language code for translation (T424613)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:14 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1293706{{!}}Disable the `no` language code for translation (T424613)]] * 13:12 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293177{{!}}Instrumentation: log new articles namespace and source (T422146)]] (duration: 07m 09s) * 13:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P93003 and previous config saved to /var/cache/conftool/dbconfig/20260526-130842-fceratto.json * 13:08 sbisson@deploy1003: sbisson: Continuing with deployment * 13:07 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2167.codfw.wmnet with OS trixie * 13:07 sbisson@deploy1003: sbisson: Backport for [[gerrit:1293177{{!}}Instrumentation: log new articles namespace and source (T422146)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:05 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2167: Upgrading db2167.codfw.wmnet * 13:05 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1293177{{!}}Instrumentation: log new articles namespace and source (T422146)]] * 13:04 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2167: Upgrading db2167.codfw.wmnet * 13:04 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:04 kart_: Update Recommendation API to 2026-05-26-074931-production * 13:03 kartik@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 13:00 topranks: deactivate CR BGP to doh2002 to test backup path via doh2001 * 12:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93000 and previous config saved to /var/cache/conftool/dbconfig/20260526-125834-fceratto.json * 12:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2226 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92999 and previous config saved to /var/cache/conftool/dbconfig/20260526-125135-fceratto.json * 12:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance * 12:51 kartik@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92998 and previous config saved to /var/cache/conftool/dbconfig/20260526-125105-fceratto.json * 12:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P92997 and previous config saved to /var/cache/conftool/dbconfig/20260526-124059-fceratto.json * 12:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2003.wikimedia.org * 12:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1214: Migration of db1214.eqiad.wmnet completed * 12:33 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc2003.wikimedia.org * 12:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P92995 and previous config saved to /var/cache/conftool/dbconfig/20260526-123052-fceratto.json * 12:26 fabfur: depooled cp204 for network activity ([[phab:T426199|T426199]]) * 12:26 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2043.* * 12:24 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-a1-codfw,ssw1-a1-codfw IPv6,ssw1-a1-codfw.mgmt with reason: Switch maintenance * 12:24 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mirror1001.wikimedia.org * 12:23 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:23 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:22 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92993 and previous config saved to /var/cache/conftool/dbconfig/20260526-122044-fceratto.json * 12:20 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:19 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mirror1001.wikimedia.org * 12:13 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2225 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92991 and previous config saved to /var/cache/conftool/dbconfig/20260526-121336-fceratto.json * 12:13 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance * 12:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92990 and previous config saved to /var/cache/conftool/dbconfig/20260526-121306-fceratto.json * 12:09 fabfur@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2011.codfw.wmnet with reason: Planned downtime for rack maintenance * 12:08 fabfur: downtime, disable puppet and stop pybal for rack maintenance ([[phab:T426199|T426199]]) * 12:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2181: Migration of db2181.codfw.wmnet completed * 12:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P92987 and previous config saved to /var/cache/conftool/dbconfig/20260526-120258-fceratto.json * 12:01 XioNoX: start ssw1-a1-codfw network maintenance (no impact expected as the spines are redundant) * 11:59 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293167{{!}}hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055{{!}}hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] (duration: 15m 26s) * 11:56 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on backup2015.codfw.wmnet,db2197.codfw.wmnet with reason: network maintenance * 11:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1005.eqiad.wmnet * 11:55 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment * 11:54 jynus: stopping mediabackups@codfw for maintenance on a codfw backup media storage server [[phab:T426199|T426199]] * 11:54 jmm@dns1004: END - running authdns-update * 11:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P92985 and previous config saved to /var/cache/conftool/dbconfig/20260526-115251-fceratto.json * 11:52 jmm@dns1004: START - running authdns-update * 11:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1005.eqiad.wmnet * 11:49 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1214: Migration of db1214.eqiad.wmnet completed * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1004.eqiad.wmnet * 11:47 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet * 11:46 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1293167{{!}}hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055{{!}}hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:45 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1004.eqiad.wmnet * 11:44 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1293167{{!}}hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055{{!}}hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] * 11:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92983 and previous config saved to /var/cache/conftool/dbconfig/20260526-114243-fceratto.json * 11:42 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet * 11:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1214.eqiad.wmnet with OS trixie * 11:35 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293691{{!}}Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] (duration: 06m 46s) * 11:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92981 and previous config saved to /var/cache/conftool/dbconfig/20260526-113542-fceratto.json * 11:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance * 11:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92980 and previous config saved to /var/cache/conftool/dbconfig/20260526-113521-fceratto.json * 11:31 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 11:31 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1293691{{!}}Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1222: Migration of db1222.eqiad.wmnet completed * 11:29 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1293691{{!}}Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] * 11:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P92978 and previous config saved to /var/cache/conftool/dbconfig/20260526-112513-fceratto.json * 11:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1214.eqiad.wmnet with reason: host reimage * 11:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc4 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92977 and previous config saved to /var/cache/conftool/dbconfig/20260526-112326-marostegui.json * 11:22 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2181: Migration of db2181.codfw.wmnet completed * 11:22 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1024 to dbctl [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92975 and previous config saved to /var/cache/conftool/dbconfig/20260526-112215-marostegui.json * 11:20 fceratto@cumin1003: dbctl commit (dc=all): 'Switchover es2042 es2041 for [[phab:T426199|T426199]]', diff saved to https://phabricator.wikimedia.org/P92974 and previous config saved to /var/cache/conftool/dbconfig/20260526-112028-fceratto.json * 11:17 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1214.eqiad.wmnet with reason: host reimage * 11:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P92972 and previous config saved to /var/cache/conftool/dbconfig/20260526-111506-fceratto.json * 11:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2181.codfw.wmnet with OS trixie * 11:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92971 and previous config saved to /var/cache/conftool/dbconfig/20260526-110458-fceratto.json * 11:02 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1214.eqiad.wmnet with OS trixie * 11:00 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293095{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] (duration: 15m 50s) * 11:00 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1214: Upgrading db1214.eqiad.wmnet * 10:59 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1214: Upgrading db1214.eqiad.wmnet * 10:59 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92968 and previous config saved to /var/cache/conftool/dbconfig/20260526-105755-fceratto.json * 10:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance * 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92967 and previous config saved to /var/cache/conftool/dbconfig/20260526-105726-fceratto.json * 10:56 jiji@deploy1003: jiji: Continuing with deployment * 10:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2181.codfw.wmnet with reason: host reimage * 10:51 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2181.codfw.wmnet with reason: host reimage * 10:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P92966 and previous config saved to /var/cache/conftool/dbconfig/20260526-104718-fceratto.json * 10:46 jiji@deploy1003: jiji: Backport for [[gerrit:1293095{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:44 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1293095{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] * 10:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P92964 and previous config saved to /var/cache/conftool/dbconfig/20260526-103711-fceratto.json * 10:36 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2181.codfw.wmnet with OS trixie * 10:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply * 10:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply * 10:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92963 and previous config saved to /var/cache/conftool/dbconfig/20260526-102703-fceratto.json * 10:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1226: Migration of db1226.eqiad.wmnet completed * 10:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2181: Upgrading db2181.codfw.wmnet * 10:24 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2181: Upgrading db2181.codfw.wmnet * 10:24 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92960 and previous config saved to /var/cache/conftool/dbconfig/20260526-101936-fceratto.json * 10:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance * 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92959 and previous config saved to /var/cache/conftool/dbconfig/20260526-101842-fceratto.json * 10:16 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-codfw@codfw * 10:16 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 10:15 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 10:10 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293668{{!}}hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] (duration: 06m 42s) * 10:09 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-codfw@codfw * 10:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92957 and previous config saved to /var/cache/conftool/dbconfig/20260526-100834-fceratto.json * 10:06 kharlan@deploy1003: kharlan: Continuing with deployment * 10:05 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293668{{!}}hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:03 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293668{{!}}hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] * 10:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2195: Migration of db2195.codfw.wmnet completed * 10:01 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>kubestage200*<nowiki>}</nowiki> and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw) * 10:01 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet * 10:01 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet * 10:00 jmm@cumin2002: END (PASS) - Cookbook sre.netbox.restart-reboot (exit_code=0) rolling reboot on A:netbox * 09:58 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 09:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92955 and previous config saved to /var/cache/conftool/dbconfig/20260526-095827-fceratto.json * 09:58 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 09:58 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 09:57 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 09:56 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-eqiad@eqiad * 09:56 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:55 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 09:55 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:55 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:55 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet * 09:54 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet * 09:54 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet * 09:54 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet * 09:53 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>kubestage100*<nowiki>}</nowiki> and (A:wikikube-staging-master-eqiad or A:wikikube-staging-worker-eqiad) * 09:53 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet * 09:53 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet * 09:52 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-eqiad@eqiad * 09:52 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293665{{!}}hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] (duration: 08m 07s) * 09:51 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2043.* * 09:51 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2044.* * 09:48 fabfur: repooling cp2043 and cp2044 (haproxy-awslc) ([[phab:T419825|T419825]]) * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92953 and previous config saved to /var/cache/conftool/dbconfig/20260526-094819-fceratto.json * 09:47 kharlan@deploy1003: kharlan: Continuing with deployment * 09:46 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet * 09:45 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293665{{!}}hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:44 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs3009.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:44 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293665{{!}}hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] * 09:41 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet * 09:41 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet * 09:41 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet * 09:41 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92951 and previous config saved to /var/cache/conftool/dbconfig/20260526-094115-fceratto.json * 09:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2229.codfw.wmnet with reason: Maintenance * 09:41 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs3009.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92950 and previous config saved to /var/cache/conftool/dbconfig/20260526-094045-fceratto.json * 09:40 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1226: Migration of db1226.eqiad.wmnet completed * 09:39 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-codfw@codfw * 09:39 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:38 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:34 fabfur: depooling cp2044 to install haproxy-awslc ([[phab:T419825|T419825]]) * 09:34 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet * 09:34 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet * 09:34 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2044.* * 09:33 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet * 09:33 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet * 09:33 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet * 09:33 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2043.* * 09:32 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293661{{!}}hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] (duration: 06m 52s) * 09:32 fabfur: depooling cp2043 to install haproxy-awslc ([[phab:T419825|T419825]]) * 09:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1226.eqiad.wmnet with OS trixie * 09:30 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-codfw@codfw * 09:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P92947 and previous config saved to /var/cache/conftool/dbconfig/20260526-093031-fceratto.json * 09:29 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet * 09:29 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet * 09:29 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet * 09:28 kharlan@deploy1003: kharlan: Continuing with deployment * 09:28 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs3008.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:28 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293661{{!}}hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:27 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet * 09:26 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet * 09:26 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet * 09:26 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet * 09:26 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293661{{!}}hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] * 09:25 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs3008.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:25 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs3010.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet * 09:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet * 09:21 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs3010.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:20 fabfur: start rebooting esams liberica instances ([[phab:T426563|T426563]]) * 09:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P92946 and previous config saved to /var/cache/conftool/dbconfig/20260526-092024-fceratto.json * 09:20 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet * 09:16 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2195: Migration of db2195.codfw.wmnet completed * 09:15 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet * 09:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1226.eqiad.wmnet with reason: host reimage * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>kubestage100*<nowiki>}</nowiki> and (A:wikikube-staging-master-eqiad or A:wikikube-staging-worker-eqiad) * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>kubestage200*<nowiki>}</nowiki> and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw) * 09:14 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293658{{!}}Fix TypeError in Mandatory2FAChecker (T427251)]] (duration: 06m 47s) * 09:10 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1226.eqiad.wmnet with reason: host reimage * 09:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92944 and previous config saved to /var/cache/conftool/dbconfig/20260526-091016-fceratto.json * 09:09 mszwarc@deploy1003: mszwarc: Continuing with deployment * 09:09 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1293658{{!}}Fix TypeError in Mandatory2FAChecker (T427251)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2195.codfw.wmnet with OS trixie * 09:07 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293658{{!}}Fix TypeError in Mandatory2FAChecker (T427251)]] * 09:06 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs4009.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 09:03 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92943 and previous config saved to /var/cache/conftool/dbconfig/20260526-090315-fceratto.json * 09:03 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2224.codfw.wmnet with reason: Maintenance * 09:03 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs4009.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92942 and previous config saved to /var/cache/conftool/dbconfig/20260526-090256-fceratto.json * 08:57 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs4008.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 08:56 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox.discovery.wmnet. on all recursors * 08:56 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors * 08:55 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1226.eqiad.wmnet with OS trixie * 08:53 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs4008.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 08:53 fabfur: start rebooting ulsfo liberica instances ([[phab:T426563|T426563]]) * 08:53 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293594{{!}}Allow to remove passkeys when there's only one standard 2FA method (T426872)]] (duration: 07m 23s) * 08:53 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs5005.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:53 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1226: Upgrading db1226.eqiad.wmnet * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P92941 and previous config saved to /var/cache/conftool/dbconfig/20260526-085248-fceratto.json * 08:51 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox.discovery.wmnet. on all recursors * 08:51 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors * 08:51 jmm@cumin2002: START - Cookbook sre.netbox.restart-reboot rolling reboot on A:netbox * 08:50 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1226: Upgrading db1226.eqiad.wmnet * 08:50 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs5005.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:50 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2195.codfw.wmnet with reason: host reimage * 08:49 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1222: Migration of db1222.eqiad.wmnet completed * 08:48 mszwarc@deploy1003: mszwarc: Continuing with deployment * 08:47 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1293594{{!}}Allow to remove passkeys when there's only one standard 2FA method (T426872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:46 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293594{{!}}Allow to remove passkeys when there's only one standard 2FA method (T426872)]] * 08:43 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs5004.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet * 08:43 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2195.codfw.wmnet with reason: host reimage * 08:43 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1292032{{!}}Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964{{!}}hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] (duration: 09m 56s) * 08:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P92939 and previous config saved to /var/cache/conftool/dbconfig/20260526-084240-fceratto.json * 08:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1222.eqiad.wmnet with OS trixie * 08:40 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs5004.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:40 fabfur: start rebooting eqsin liberica instances ([[phab:T426563|T426563]]) * 08:39 kartik@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 08:39 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet * 08:39 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 08:39 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs5006.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:35 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs5006.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1024.eqiad.wmnet * 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1024.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 08:35 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1292032{{!}}Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964{{!}}hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:33 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs6002.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:33 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1292032{{!}}Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964{{!}}hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] * 08:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92938 and previous config saved to /var/cache/conftool/dbconfig/20260526-083233-fceratto.json * 08:30 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs6002.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:25 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2217 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92937 and previous config saved to /var/cache/conftool/dbconfig/20260526-082531-fceratto.json * 08:25 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2217.codfw.wmnet with reason: Maintenance * 08:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92936 and previous config saved to /var/cache/conftool/dbconfig/20260526-082458-fceratto.json * 08:23 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2195.codfw.wmnet with OS trixie * 08:23 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage * 08:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2195: Upgrading db2195.codfw.wmnet * 08:20 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2195: Upgrading db2195.codfw.wmnet * 08:19 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:18 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage * 08:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P92934 and previous config saved to /var/cache/conftool/dbconfig/20260526-081451-fceratto.json * 08:13 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs6001.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 08:10 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs6001.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:09 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1024.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 08:04 jmm@cumin2002: START - Cookbook sre.dns.netbox * 08:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P92932 and previous config saved to /var/cache/conftool/dbconfig/20260526-080443-fceratto.json * 08:01 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1222.eqiad.wmnet with OS trixie * 08:00 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs6003.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet * 07:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet * 07:59 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1024.eqiad.wmnet * 07:59 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1023.eqiad.wmnet * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1023.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:59 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 07:59 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:58 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1023.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:56 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs6003.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 07:56 fabfur: start rebooting drmrs liberica instances ([[phab:T426563|T426563]]) * 07:56 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs7002.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:54 jmm@cumin2002: START - Cookbook sre.dns.netbox * 07:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92931 and previous config saved to /var/cache/conftool/dbconfig/20260526-075435-fceratto.json * 07:52 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs7002.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:51 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1047.eqiad.wmnet * 07:51 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:51 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1047.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:49 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1023.eqiad.wmnet * 07:47 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2193 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92930 and previous config saved to /var/cache/conftool/dbconfig/20260526-074739-fceratto.json * 07:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2193.codfw.wmnet with reason: Maintenance * 07:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92929 and previous config saved to /var/cache/conftool/dbconfig/20260526-074710-fceratto.json * 07:46 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet * 07:45 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet * 07:45 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:45 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs7001.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:44 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1025.eqiad.wmnet * 07:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:43 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:41 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs7001.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:40 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1046.eqiad.wmnet * 07:40 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1046.eqiad.wmnet * 07:38 arthurtaylor@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291951{{!}}Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] (duration: 12m 01s) * 07:38 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1047.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P92928 and previous config saved to /var/cache/conftool/dbconfig/20260526-073702-fceratto.json * 07:37 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet * 07:36 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet * 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance * 07:35 fabfur: start rebooting magru liberica instances ([[phab:T426563|T426563]]) * 07:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92926 and previous config saved to /var/cache/conftool/dbconfig/20260526-073459-fceratto.json * 07:32 arthurtaylor@deploy1003: arthurtaylor: Continuing with deployment * 07:31 arthurtaylor@deploy1003: arthurtaylor: Backport for [[gerrit:1291951{{!}}Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:30 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1046.eqiad.wmnet * 07:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20260526-072643-fceratto.json * 07:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1046.eqiad.wmnet * 07:26 arthurtaylor@deploy1003: Started scap sync-world: Backport for [[gerrit:1291951{{!}}Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] * 07:25 jiji@cumin1003: START - Cookbook sre.dns.netbox * 07:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P92924 and previous config saved to /var/cache/conftool/dbconfig/20260526-072452-fceratto.json * 07:24 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet * 07:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1047.eqiad.wmnet * 07:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1047.eqiad.wmnet * 07:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92923 and previous config saved to /var/cache/conftool/dbconfig/20260526-071635-fceratto.json * 07:15 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet * 07:15 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1026.eqiad.wmnet * 07:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P92922 and previous config saved to /var/cache/conftool/dbconfig/20260526-071444-fceratto.json * 07:13 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet * 07:11 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1025.eqiad.wmnet * 07:10 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1025.eqiad.wmnet * 07:09 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92921 and previous config saved to /var/cache/conftool/dbconfig/20260526-070946-fceratto.json * 07:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2180.codfw.wmnet with reason: Maintenance * 07:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92920 and previous config saved to /var/cache/conftool/dbconfig/20260526-070916-fceratto.json * 07:09 moritzm: failover Ganeti master in eqiad to ganeti1048 * 07:09 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1047.eqiad.wmnet * 07:07 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1046.eqiad.wmnet * 07:07 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:06 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1046.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org * 07:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92919 and previous config saved to /var/cache/conftool/dbconfig/20260526-070436-fceratto.json * 07:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet * 07:04 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1046.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet * 07:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org * 06:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P92918 and previous config saved to /var/cache/conftool/dbconfig/20260526-065909-fceratto.json * 06:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org * 06:58 jiji@cumin1003: START - Cookbook sre.dns.netbox * 06:58 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet * 06:55 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet * 06:53 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1046.eqiad.wmnet * 06:53 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1045.eqiad.wmnet * 06:53 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:53 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1045.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 06:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org * 06:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P92917 and previous config saved to /var/cache/conftool/dbconfig/20260526-064901-fceratto.json * 06:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92916 and previous config saved to /var/cache/conftool/dbconfig/20260526-064833-fceratto.json * 06:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1222.eqiad.wmnet with reason: Maintenance * 06:47 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1222: Switchover * 06:41 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92914 and previous config saved to /var/cache/conftool/dbconfig/20260526-063853-fceratto.json * 06:35 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 06:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92912 and previous config saved to /var/cache/conftool/dbconfig/20260526-063155-fceratto.json * 06:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance * 06:28 fceratto@cumin1003: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance * 06:23 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1222: Switchover * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1222 [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92910 and previous config saved to /var/cache/conftool/dbconfig/20260526-061656-fceratto.json * 06:15 fceratto@dns1005: END - running authdns-update * 06:14 fceratto@dns1005: START - running authdns-update * 06:11 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1162 to s2 primary and set section read-write [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92909 and previous config saved to /var/cache/conftool/dbconfig/20260526-061114-fceratto.json * 06:10 fceratto@cumin1003: dbctl commit (dc=all): 'Set s2 eqiad as read-only for maintenance - [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92908 and previous config saved to /var/cache/conftool/dbconfig/20260526-061021-fceratto.json * 06:10 federico3: Starting s2 eqiad failover from db1222 to db1162 - [[phab:T425622|T425622]] * 06:04 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1162 with weight 0 [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92907 and previous config saved to /var/cache/conftool/dbconfig/20260526-060443-fceratto.json * 06:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s2 [[phab:T425622|T425622]] * 06:02 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:02 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:01 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:00 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 05:15 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1014.eqiad.wmnet: Maintenance on pc4 * 05:15 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 05:15 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 05:15 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1014.eqiad.wmnet: Maintenance on pc4 * 05:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2024.codfw.wmnet,pc[1014,1024].eqiad.wmnet with reason: Maintenance on pc4 * 04:37 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 04:34 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.1 (duration: 02m 32s) * 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] (duration: 36m 24s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 20s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-25 == * 21:00 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1045.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:49 jiji@cumin1003: START - Cookbook sre.dns.netbox * 20:38 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1045.eqiad.wmnet * 20:37 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1044.eqiad.wmnet * 20:37 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:37 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1044.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:25 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1044.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:15 moritzm: truncate krb5kdc.log1 (which made log rotation fail) * 20:06 jiji@cumin1003: START - Cookbook sre.dns.netbox * 19:57 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1044.eqiad.wmnet * 19:25 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1043.eqiad.wmnet * 19:25 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:25 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1043.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 19:22 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1043.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 18:49 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_eqiad * 18:49 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1115.eqiad.wmnet * 18:34 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5023.eqsin.wmnet [reason: manually pooling after reboot as icinga was down] * 18:33 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5030.eqsin.wmnet [reason: manually pooling after reboot as icinga was down] * 18:22 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp5030*<nowiki>}</nowiki> and A:cp * 18:22 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5030.eqsin.wmnet * 18:15 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp5023*<nowiki>}</nowiki> and A:cp * 18:15 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5023.eqsin.wmnet * 18:10 jiji@cumin1003: START - Cookbook sre.dns.netbox * 18:10 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp5030*<nowiki>}</nowiki> and A:cp * 18:09 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp1113*<nowiki>}</nowiki> and A:cp * 18:09 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet * 18:09 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet * 18:03 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp1113*<nowiki>}</nowiki> and A:cp * 18:02 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp5023*<nowiki>}</nowiki> and A:cp * 18:01 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-text_eqiad * 18:01 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-upload_eqsin * 18:01 sukhe: sre.cdn.roll-reboot cookbooks stalled due to icinga reboot * 18:00 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-text_eqsin * 17:35 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1043.eqiad.wmnet * 17:31 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1110.eqiad.wmnet [reason: manually pooling after reboot as icinga was down] * 17:30 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1042.eqiad.wmnet * 17:30 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:30 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1042.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:29 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1111.eqiad.wmnet * 17:28 sukhe: sukhe@alert1002:~$ sudo systemctl restart icinga.service * 17:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92903 and previous config saved to /var/cache/conftool/dbconfig/20260525-171310-fceratto.json * 17:11 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1042.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:06 jiji@cumin1003: START - Cookbook sre.dns.netbox * 17:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P92902 and previous config saved to /var/cache/conftool/dbconfig/20260525-170302-fceratto.json * 16:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P92901 and previous config saved to /var/cache/conftool/dbconfig/20260525-165255-fceratto.json * 16:51 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1042.eqiad.wmnet * 16:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92900 and previous config saved to /var/cache/conftool/dbconfig/20260525-164247-fceratto.json * 16:42 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1041.eqiad.wmnet * 16:42 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:42 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1041.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:41 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1041.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:40 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5021.eqsin.wmnet * 16:39 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5029.eqsin.wmnet * 16:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92899 and previous config saved to /var/cache/conftool/dbconfig/20260525-163559-fceratto.json * 16:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2158.codfw.wmnet with reason: Maintenance * 16:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92898 and previous config saved to /var/cache/conftool/dbconfig/20260525-163512-fceratto.json * 16:34 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1108.eqiad.wmnet * 16:30 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1109.eqiad.wmnet * 16:26 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249', diff saved to https://phabricator.wikimedia.org/P92897 and previous config saved to /var/cache/conftool/dbconfig/20260525-162505-fceratto.json * 16:20 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1041.eqiad.wmnet * 16:20 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1040.eqiad.wmnet * 16:20 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:20 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1040.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:16 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1040.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249', diff saved to https://phabricator.wikimedia.org/P92896 and previous config saved to /var/cache/conftool/dbconfig/20260525-161457-fceratto.json * 16:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92895 and previous config saved to /var/cache/conftool/dbconfig/20260525-160450-fceratto.json * 16:02 jiji@cumin1003: START - Cookbook sre.dns.netbox * 15:59 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2249 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92894 and previous config saved to /var/cache/conftool/dbconfig/20260525-155930-fceratto.json * 15:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2249.codfw.wmnet with reason: Maintenance * 15:57 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5020.eqsin.wmnet * 15:57 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5028.eqsin.wmnet * 15:52 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1106.eqiad.wmnet * 15:51 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1107.eqiad.wmnet * 15:29 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1040.eqiad.wmnet * 15:29 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1039.eqiad.wmnet * 15:29 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:29 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1039.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:27 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1039.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:17 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1013 from dbctl [[phab:T427190|T427190]]', diff saved to https://phabricator.wikimedia.org/P92893 and previous config saved to /var/cache/conftool/dbconfig/20260525-151718-marostegui.json * 15:15 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5019.eqsin.wmnet * 15:15 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5027.eqsin.wmnet * 15:12 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1104.eqiad.wmnet * 15:11 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1105.eqiad.wmnet * 15:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92892 and previous config saved to /var/cache/conftool/dbconfig/20260525-150309-fceratto.json * 14:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P92891 and previous config saved to /var/cache/conftool/dbconfig/20260525-145301-fceratto.json * 14:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P92890 and previous config saved to /var/cache/conftool/dbconfig/20260525-144253-fceratto.json * 14:33 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1102.eqiad.wmnet * 14:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92889 and previous config saved to /var/cache/conftool/dbconfig/20260525-143246-fceratto.json * 14:32 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5026.eqsin.wmnet * 14:32 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5018.eqsin.wmnet * 14:31 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1103.eqiad.wmnet * 14:25 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2228 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92888 and previous config saved to /var/cache/conftool/dbconfig/20260525-142551-fceratto.json * 14:25 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance * 14:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92887 and previous config saved to /var/cache/conftool/dbconfig/20260525-142520-fceratto.json * 14:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P92885 and previous config saved to /var/cache/conftool/dbconfig/20260525-141513-fceratto.json * 14:12 jiji@cumin1003: START - Cookbook sre.dns.netbox * 14:06 sukhe: curl localhost:9090/pools/inference-staging-grpc_30051 shows ml-staging200[1-3].codfw.wmnet as enabled and pooled: [[phab:T424049|T424049]] * 14:05 sukhe: sukhe@lvs2013:~$ sudo systemctl restart pybal.service: [[phab:T424049|T424049]] * 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P92884 and previous config saved to /var/cache/conftool/dbconfig/20260525-140505-fceratto.json * 14:03 sukhe: sudo cumin 'A:lvs and A:lvs-low-traffic-codfw' 'run-puppet-agent --enable "adding new ml-serve (grpc) [[phab:T424049|T424049]]"' * 14:02 sukhe: sukhe@lvs2014:~$ sudo systemctl restart pybal.service": [[phab:T424049|T424049]] * 14:02 sukhe: sukhe@lvs2014:~$ sudo systemctl restart pybal.service * 14:00 sukhe: sudo cumin 'A:lvs and A:lvs-secondary-codfw' 'run-puppet-agent --enable "adding new ml-serve (grpc) [[phab:T424049|T424049]]"' * 13:59 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1039.eqiad.wmnet * 13:58 sukhe: sudo cumin 'A:lvs and A:eqiad' 'run-puppet-agent --enable "adding new ml-serve (grpc) [[phab:T424049|T424049]]": NOOP change, since service is codfw only * 13:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92882 and previous config saved to /var/cache/conftool/dbconfig/20260525-135458-fceratto.json * 13:52 Msz2001: Everything deployed, UTC afternoon config+backport window done * 13:52 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293119{{!}}Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] (duration: 09m 43s) * 13:51 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1101.eqiad.wmnet * 13:51 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1100.eqiad.wmnet * 13:50 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5025.eqsin.wmnet * 13:50 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5017.eqsin.wmnet * 13:49 kart_: Updated Recommendation API to 2026-05-21-044522-production * 13:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2223 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92881 and previous config saved to /var/cache/conftool/dbconfig/20260525-134807-fceratto.json * 13:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2223.codfw.wmnet with reason: Maintenance * 13:47 mszwarc@deploy1003: vadymts1, mszwarc: Continuing with deployment * 13:47 kartik@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 13:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92880 and previous config saved to /var/cache/conftool/dbconfig/20260525-134737-fceratto.json * 13:45 mszwarc@deploy1003: vadymts1, mszwarc: Backport for [[gerrit:1293119{{!}}Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:45 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1162: Reboot * 13:43 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293119{{!}}Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] * 13:40 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqiad * 13:39 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqiad * 13:38 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290813{{!}}Article Guidance: enable experiment on phase 2 wikis (T426871)]] (duration: 08m 14s) * 13:38 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqsin * 13:38 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqsin * 13:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P92878 and previous config saved to /var/cache/conftool/dbconfig/20260525-133729-fceratto.json * 13:34 sbisson@deploy1003: sbisson: Continuing with deployment * 13:33 kartik@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 13:32 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1038.eqiad.wmnet * 13:32 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:32 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1038.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 13:31 sbisson@deploy1003: sbisson: Backport for [[gerrit:1290813{{!}}Article Guidance: enable experiment on phase 2 wikis (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:30 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1290813{{!}}Article Guidance: enable experiment on phase 2 wikis (T426871)]] * 13:27 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293094{{!}}Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953{{!}}Update logo, wordmark and tagline for zghwiki (T426406)]] (duration: 07m 43s) * 13:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P92876 and previous config saved to /var/cache/conftool/dbconfig/20260525-132722-fceratto.json * 13:23 mszwarc@deploy1003: mszwarc, jhsoby: Continuing with deployment * 13:21 mszwarc@deploy1003: mszwarc, jhsoby: Backport for [[gerrit:1293094{{!}}Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953{{!}}Update logo, wordmark and tagline for zghwiki (T426406)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:20 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1038.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 13:20 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293094{{!}}Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953{{!}}Update logo, wordmark and tagline for zghwiki (T426406)]] * 13:19 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291966{{!}}Modify various configurations for English Wikibooks (T426992)]] (duration: 15m 53s) * 13:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92875 and previous config saved to /var/cache/conftool/dbconfig/20260525-131714-fceratto.json * 13:12 mszwarc@deploy1003: vadymts1, mszwarc: Continuing with deployment * 13:12 jiji@cumin1003: START - Cookbook sre.dns.netbox * 13:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92873 and previous config saved to /var/cache/conftool/dbconfig/20260525-131023-fceratto.json * 13:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2211.codfw.wmnet with reason: Maintenance * 13:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92872 and previous config saved to /var/cache/conftool/dbconfig/20260525-130950-fceratto.json * 13:07 mszwarc@deploy1003: vadymts1, mszwarc: Backport for [[gerrit:1291966{{!}}Modify various configurations for English Wikibooks (T426992)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:03 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1291966{{!}}Modify various configurations for English Wikibooks (T426992)]] * 12:59 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1162: Reboot * 12:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P92870 and previous config saved to /var/cache/conftool/dbconfig/20260525-125942-fceratto.json * 12:59 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1162: Reboot * 12:59 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1162: Reboot * 12:58 kart_: Updated cxserver to 2026-05-24-103047-production ([[phab:T426808|T426808]], [[phab:T373418|T373418]]) * 12:56 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply * 12:56 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply * 12:54 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db1162: Reboot * 12:54 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1162: Reboot * 12:54 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply * 12:53 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply * 12:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1162.eqiad.wmnet with reason: Reboot * 12:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P92868 and previous config saved to /var/cache/conftool/dbconfig/20260525-124934-fceratto.json * 12:40 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply * 12:39 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply * 12:39 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1038.eqiad.wmnet * 12:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92867 and previous config saved to /var/cache/conftool/dbconfig/20260525-123927-fceratto.json * 12:32 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2192 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92866 and previous config saved to /var/cache/conftool/dbconfig/20260525-123239-fceratto.json * 12:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2192.codfw.wmnet with reason: Maintenance * 12:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92865 and previous config saved to /var/cache/conftool/dbconfig/20260525-123208-fceratto.json * 12:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P92864 and previous config saved to /var/cache/conftool/dbconfig/20260525-122201-fceratto.json * 12:17 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1037.eqiad.wmnet * 12:17 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:17 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1037.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 12:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P92863 and previous config saved to /var/cache/conftool/dbconfig/20260525-121153-fceratto.json * 12:10 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1037.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 12:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92862 and previous config saved to /var/cache/conftool/dbconfig/20260525-120145-fceratto.json * 11:58 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2178 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92861 and previous config saved to /var/cache/conftool/dbconfig/20260525-115504-fceratto.json * 11:54 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2178.codfw.wmnet with reason: Maintenance * 11:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92860 and previous config saved to /var/cache/conftool/dbconfig/20260525-115434-fceratto.json * 11:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P92859 and previous config saved to /var/cache/conftool/dbconfig/20260525-114426-fceratto.json * 11:43 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1037.eqiad.wmnet * 11:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P92858 and previous config saved to /var/cache/conftool/dbconfig/20260525-113419-fceratto.json * 11:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2160.codfw.wmnet with OS trixie * 11:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92857 and previous config saved to /var/cache/conftool/dbconfig/20260525-112411-fceratto.json * 11:17 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2171 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92856 and previous config saved to /var/cache/conftool/dbconfig/20260525-111717-fceratto.json * 11:17 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: Maintenance * 11:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92855 and previous config saved to /var/cache/conftool/dbconfig/20260525-111648-fceratto.json * 11:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P92854 and previous config saved to /var/cache/conftool/dbconfig/20260525-110640-fceratto.json * 11:05 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2160.codfw.wmnet with reason: host reimage * 11:00 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2160.codfw.wmnet with reason: host reimage * 10:58 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:57 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:56 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P92853 and previous config saved to /var/cache/conftool/dbconfig/20260525-105633-fceratto.json * 10:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92852 and previous config saved to /var/cache/conftool/dbconfig/20260525-104625-fceratto.json * 10:43 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2160.codfw.wmnet with OS trixie * 10:41 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc3 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92851 and previous config saved to /var/cache/conftool/dbconfig/20260525-104141-marostegui.json * 10:40 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1023 to pc3 as master [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92850 and previous config saved to /var/cache/conftool/dbconfig/20260525-104055-marostegui.json * 10:40 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1023 to dbctl', diff saved to https://phabricator.wikimedia.org/P92849 and previous config saved to /var/cache/conftool/dbconfig/20260525-104027-marostegui.json * 10:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92848 and previous config saved to /var/cache/conftool/dbconfig/20260525-103944-fceratto.json * 10:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance * 10:31 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply * 10:30 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply * 10:27 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 10:18 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 10:16 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1011.eqiad.wmnet * 10:08 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1011.eqiad.wmnet * 10:08 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1007.eqiad.wmnet * 09:59 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1007.eqiad.wmnet * 09:59 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1006.eqiad.wmnet * 09:57 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:49 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1006.eqiad.wmnet * 09:48 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:46 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:45 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:40 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:40 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:28 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:17 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:13 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92847 and previous config saved to /var/cache/conftool/dbconfig/20260525-091302-fceratto.json * 09:12 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231', diff saved to https://phabricator.wikimedia.org/P92846 and previous config saved to /var/cache/conftool/dbconfig/20260525-090255-fceratto.json * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231', diff saved to https://phabricator.wikimedia.org/P92845 and previous config saved to /var/cache/conftool/dbconfig/20260525-085247-fceratto.json * 08:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92844 and previous config saved to /var/cache/conftool/dbconfig/20260525-084239-fceratto.json * 08:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92843 and previous config saved to /var/cache/conftool/dbconfig/20260525-083540-fceratto.json * 08:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2231.codfw.wmnet with reason: Maintenance * 08:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92842 and previous config saved to /var/cache/conftool/dbconfig/20260525-083511-fceratto.json * 08:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215', diff saved to https://phabricator.wikimedia.org/P92841 and previous config saved to /var/cache/conftool/dbconfig/20260525-082504-fceratto.json * 08:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215', diff saved to https://phabricator.wikimedia.org/P92840 and previous config saved to /var/cache/conftool/dbconfig/20260525-081456-fceratto.json * 08:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92839 and previous config saved to /var/cache/conftool/dbconfig/20260525-080448-fceratto.json * 07:57 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2215 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92838 and previous config saved to /var/cache/conftool/dbconfig/20260525-075739-fceratto.json * 07:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2215.codfw.wmnet with reason: Maintenance * 07:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92837 and previous config saved to /var/cache/conftool/dbconfig/20260525-075708-fceratto.json * 07:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196', diff saved to https://phabricator.wikimedia.org/P92836 and previous config saved to /var/cache/conftool/dbconfig/20260525-074700-fceratto.json * 07:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196', diff saved to https://phabricator.wikimedia.org/P92835 and previous config saved to /var/cache/conftool/dbconfig/20260525-073653-fceratto.json * 07:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92834 and previous config saved to /var/cache/conftool/dbconfig/20260525-072645-fceratto.json * 07:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92833 and previous config saved to /var/cache/conftool/dbconfig/20260525-071953-fceratto.json * 07:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2196.codfw.wmnet with reason: Maintenance * 07:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92832 and previous config saved to /var/cache/conftool/dbconfig/20260525-071924-fceratto.json * 07:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186', diff saved to https://phabricator.wikimedia.org/P92831 and previous config saved to /var/cache/conftool/dbconfig/20260525-070917-fceratto.json * 07:03 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2233.codfw.wmnet with OS trixie * 06:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186', diff saved to https://phabricator.wikimedia.org/P92830 and previous config saved to /var/cache/conftool/dbconfig/20260525-065909-fceratto.json * 06:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92829 and previous config saved to /var/cache/conftool/dbconfig/20260525-064902-fceratto.json * 06:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92828 and previous config saved to /var/cache/conftool/dbconfig/20260525-064305-fceratto.json * 06:42 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance * 06:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2233.codfw.wmnet with reason: host reimage * 06:35 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2233.codfw.wmnet with reason: host reimage * 06:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2233.codfw.wmnet with OS trixie * 06:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2233.codfw.wmnet with reason: Reimage to Trixie * 06:17 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 06:17 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:15 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2160.codfw.wmnet with reason: Reboot upgrade m2 * 06:15 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2233.codfw.wmnet with reason: Reboot upgrade m2 * 06:08 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy1027.eqiad.wmnet with reason: Reboot * 05:18 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2023.codfw.wmnet,pc[1013,1023].eqiad.wmnet with reason: Maintenance on pc3 * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1013.eqiad.wmnet: Maintenance on pc3 * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 05:17 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 05:17 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1013.eqiad.wmnet: Maintenance on pc3 * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 43s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-24 == * 19:08 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down * 02:06 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 23s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-23 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 35s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-22 == * 23:39 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 23:39 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 23:39 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 23:39 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 23:38 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 23:37 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 23:37 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 23:37 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 22:20 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 22:12 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 22:11 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 20:29 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 20:28 inflatador: bking@deploy1003 set eqiad prod cirrus `node_concurrent_recoveries` up to 7 from 4 [[phab:T426585|T426585]] * 20:27 inflatador: bking@deploy1003 set codfw prod cirrus `node_concurrent_recoveries` back down to 4 from 7 [[phab:T426585|T426585]] * 18:39 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 17:34 topranks: enable ttl protection on esams CRs IBGP session * 17:28 topranks: enable ttl protection on ulsfo CRs IBGP session * 16:50 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 16:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 16:16 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 16:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 16:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 15:58 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 15:14 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 15:02 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 15:02 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 14:34 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudnet2008-dev.codfw.wmnet * 14:34 andrew@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:34 andrew@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2008-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 14:33 andrew@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2008-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 14:33 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb[1020,1022-1025].eqiad.wmnet * 14:29 andrew@cumin2002: START - Cookbook sre.dns.netbox * 14:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 14:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 14:23 andrew@cumin2002: START - Cookbook sre.hosts.decommission for hosts cloudnet2008-dev.codfw.wmnet * 14:23 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudnet2007-dev.codfw.wmnet * 14:23 andrew@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:23 andrew@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2007-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 14:03 andrew@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2007-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 13:59 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb[1020,1022-1025].eqiad.wmnet * 13:58 andrew@cumin2002: START - Cookbook sre.dns.netbox * 13:53 andrew@cumin2002: START - Cookbook sre.hosts.decommission for hosts cloudnet2007-dev.codfw.wmnet * 13:52 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1018.eqiad.wmnet * 13:50 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 13:50 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 13:46 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1018.eqiad.wmnet * 13:25 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for clouddb1018.eqiad.wmnet * 13:25 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1018.eqiad.wmnet * 13:25 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for 6 hosts * 13:16 inflatador: bking@deploy1002 set search_codfw cluster recovery settings from 4 to 7 [[phab:T426560|T426560]] * 13:15 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for 6 hosts * 13:15 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 13:11 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp5017.eqsin.wmnet<nowiki>}</nowiki> and A:cp * 13:11 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5017.eqsin.wmnet * 13:10 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1017.eqiad.wmnet * 13:09 elukey: uploaded spicerack_12.6.0 to apt.wikimedia.org bookworm-wikimedia * 13:08 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for clouddb1017.eqiad.wmnet * 12:59 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp5017.eqsin.wmnet<nowiki>}</nowiki> and A:cp * 12:57 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp308[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 12:57 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3081.esams.wmnet * 12:54 isaranto@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 12:41 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 12:15 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3080.esams.wmnet * 12:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 12:03 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp308[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[2-3].esams.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3073.esams.wmnet * 11:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2154: Migration of db2154.codfw.wmnet completed * 11:19 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3072.esams.wmnet * 11:15 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 11:11 fnegri@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb1017.eqiad.wmnet with reason: Rebooting clouddb1017 * 11:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1172: Migration of db1172.eqiad.wmnet completed * 11:07 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[2-3].esams.wmnet<nowiki>}</nowiki> and A:cp * 11:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1058.eqiad.wmnet * 11:01 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 11:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3079.esams.wmnet * 10:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1058.eqiad.wmnet * 10:55 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 10:55 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 10:48 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 10:47 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 10:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1024.eqiad.wmnet * 10:43 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:43 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:43 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:42 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:42 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:42 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2154: Migration of db2154.codfw.wmnet completed * 10:42 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:41 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1024.eqiad.wmnet * 10:37 moritzm: remove ganeti1024 foom eqiad Ganeti cluster [[phab:T424680|T424680]] * 10:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2154.codfw.wmnet with OS trixie * 10:31 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2010.codfw.wmnet with OS trixie * 10:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1024.eqiad.wmnet * 10:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1172: Migration of db1172.eqiad.wmnet completed * 10:19 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3078.esams.wmnet * 10:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2154.codfw.wmnet with reason: host reimage * 10:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1172.eqiad.wmnet with OS trixie * 10:15 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1017.eqiad.wmnet * 10:13 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2154.codfw.wmnet with reason: host reimage * 10:07 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 10:06 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 10:06 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3071.esams.wmnet * 09:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1172.eqiad.wmnet with reason: host reimage * 09:56 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2154.codfw.wmnet with OS trixie * 09:55 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage * 09:53 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1172.eqiad.wmnet with reason: host reimage * 09:51 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage * 09:39 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2154: Upgrading db2154.codfw.wmnet * 09:39 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2154: Upgrading db2154.codfw.wmnet * 09:38 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:38 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1172.eqiad.wmnet with OS trixie * 09:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1172: Upgrading db1172.eqiad.wmnet * 09:34 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1172: Upgrading db1172.eqiad.wmnet * 09:34 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:34 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2009.codfw.wmnet with OS trixie * 09:33 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS trixie * 09:26 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 09:26 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 09:26 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3070.esams.wmnet * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 09:16 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie * 09:14 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 09:11 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[6-7].esams.wmnet<nowiki>}</nowiki> and A:cp * 09:11 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3077.esams.wmnet * 09:04 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 09:03 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie * 08:47 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 08:46 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie * 08:40 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 08:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 08:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 08:30 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3076.esams.wmnet * 08:18 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[6-7].esams.wmnet<nowiki>}</nowiki> and A:cp * 08:15 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti1058.eqiad.wmnet on all recursors * 08:15 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:15 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change records for ganeti1058 - cmooney@cumin1003" * 08:15 cmooney@cumin1003: START - Cookbook sre.dns.wipe-cache ganeti1058.eqiad.wmnet on all recursors * 08:15 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change records for ganeti1058 - cmooney@cumin1003" * 08:09 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 08:07 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp306[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 08:07 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3069.esams.wmnet * 08:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 08:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 07:31 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1024.eqiad.wmnet * 07:26 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3068.esams.wmnet * 07:14 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp306[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 07:11 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1057.eqiad.wmnet to cluster eqiad and group A * 07:10 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3075.esams.wmnet<nowiki>}</nowiki> and A:cp * 07:10 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3075.esams.wmnet * 07:06 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1057.eqiad.wmnet to cluster eqiad and group A * 07:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1057.eqiad.wmnet * 07:02 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1057 * 07:01 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1057 * 06:58 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3075.esams.wmnet<nowiki>}</nowiki> and A:cp * 06:58 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3067.esams.wmnet<nowiki>}</nowiki> and A:cp * 06:58 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3067.esams.wmnet * 06:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1057.eqiad.wmnet * 06:46 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3067.esams.wmnet<nowiki>}</nowiki> and A:cp * 06:13 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1024.eqiad.wmnet * 06:08 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1024.eqiad.wmnet * 06:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org * 06:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org * 05:25 marostegui@dns1004: END - running authdns-update * 05:24 marostegui@dns1004: START - running authdns-update * 05:23 marostegui: Failover m5-master [[phab:T426633|T426633]] * 05:19 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy1028.eqiad.wmnet with reason: Reboot * 05:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy2005.codfw.wmnet with reason: Reboot * 05:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1012.eqiad.wmnet * 05:11 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 05:11 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1012.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 05:06 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1012.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 05:03 marostegui@cumin1003: START - Cookbook sre.dns.netbox * 04:56 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1012.eqiad.wmnet == 2026-05-21 == * 23:43 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290954{{!}}Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957{{!}}Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958{{!}}Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960{{!}}Drop unused wpReportIncidentLocalLinks]] (duration: 06m 42s) * 23:38 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 23:38 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1290954{{!}}Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957{{!}}Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958{{!}}Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960{{!}}Drop unused wpReportIncidentLocalLinks]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified * 23:36 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1290954{{!}}Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957{{!}}Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958{{!}}Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960{{!}}Drop unused wpReportIncidentLocalLinks]] * 22:26 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul2002.codfw.wmnet with OS trixie * 22:08 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul2002.codfw.wmnet with reason: host reimage * 22:03 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul2002.codfw.wmnet with reason: host reimage * 22:02 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 21:49 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 21:49 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 21:44 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host zuul2002.codfw.wmnet with OS trixie * 21:25 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:25 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 20:26 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 20:16 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 19:22 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-reboot (exit_code=0) rolling reboot on A:restbase * 19:10 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:59 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:53 papaul: rebooting msw1-codfw * 18:50 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:39 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:52 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:52 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:50 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:49 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:49 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:48 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:46 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 17:46 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 17:43 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:43 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:43 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:42 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:42 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:41 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:41 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:41 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:41 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:41 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:41 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:41 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:40 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:40 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:40 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:39 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 * 17:39 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:38 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down * 17:37 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 17:36 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:36 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:30 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:25 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:25 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:24 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:23 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:22 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1016.eqiad.wmnet * 17:22 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:14 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2031.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:14 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2030.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:13 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1016.eqiad.wmnet * 17:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:08 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repool pc2 ([[phab:T421705|T421705]])', diff saved to https://phabricator.wikimedia.org/P92810 and previous config saved to /var/cache/conftool/dbconfig/20260521-170823-ladsgroup.json * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:07 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2031.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:07 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2030.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:06 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:03 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:03 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:00 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2029 * 16:58 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2031 * 16:58 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:58 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 * 16:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 * 16:55 papaul: rebooting msw-d3-codfw * 16:55 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 16:52 papaul: rebooting msw-c7-codfw * 16:51 papaul: rebooting msw-c6-codfw * 16:48 papaul: rebooting msw-b7-codfw * 16:48 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1014.eqiad.wmnet * 16:45 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1014.eqiad.wmnet * 16:43 papaul: rebooting msw-b6-codfw * 16:40 papaul: rebooting msw-a1-codfw * 16:37 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 16:37 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1014.eqiad.wmnet * 16:37 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:36 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:35 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 16:35 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2030 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2030 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 * 16:34 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 16:34 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:33 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2028 to codfw - jhancock@cumin2002" * 16:33 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2028 to codfw - jhancock@cumin2002" * 16:26 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 16:24 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on pc1022.eqiad.wmnet with reason: Move to nftables * 16:24 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on pc2022.codfw.wmnet with reason: Move to nftables * 16:18 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2048: Repooling * 16:18 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depool pc2 ([[phab:T421705|T421705]])', diff saved to https://phabricator.wikimedia.org/P92807 and previous config saved to /var/cache/conftool/dbconfig/20260521-161808-ladsgroup.json * 16:15 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:15 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:15 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:15 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:52 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 15:42 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es2048: Repooling * 15:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92804 and previous config saved to /var/cache/conftool/dbconfig/20260521-154108-fceratto.json * 15:39 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:38 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:34 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:34 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:34 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:34 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:34 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92803 and previous config saved to /var/cache/conftool/dbconfig/20260521-153400-fceratto.json * 15:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2048.codfw.wmnet with reason: Maintenance * 15:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92802 and previous config saved to /var/cache/conftool/dbconfig/20260521-153331-fceratto.json * 15:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:25 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:24 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:24 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:24 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:24 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040', diff saved to https://phabricator.wikimedia.org/P92801 and previous config saved to /var/cache/conftool/dbconfig/20260521-152323-fceratto.json * 15:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1045.eqiad.wmnet * 15:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1045.eqiad.wmnet * 15:19 claime: Enabling puppet on A:cp-text - [[phab:T426323|T426323]] * 15:15 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1045.eqiad.wmnet * 15:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040', diff saved to https://phabricator.wikimedia.org/P92800 and previous config saved to /var/cache/conftool/dbconfig/20260521-151316-fceratto.json * 15:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1014.eqiad.wmnet * 15:11 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1045.eqiad.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2034.codfw.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2034.codfw.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1037.eqiad.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1037.eqiad.wmnet * 15:07 elukey@cumin1003: END (PASS) - Cookbook sre.misc-clusters.restart-reboot-config-master (exit_code=0) rolling reboot on A:config-master * 15:06 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1014.eqiad.wmnet * 15:05 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors * 15:05 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors * 15:04 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290805{{!}}hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] (duration: 10m 11s) * 15:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92799 and previous config saved to /var/cache/conftool/dbconfig/20260521-150308-fceratto.json * 15:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1037.eqiad.wmnet * 15:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2034.codfw.wmnet * 15:00 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors * 15:00 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors * 15:00 elukey@cumin1003: START - Cookbook sre.misc-clusters.restart-reboot-config-master rolling reboot on A:config-master * 15:00 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 15:00 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-lab1002.eqiad.wmnet * 14:59 elukey@cumin1003: END (PASS) - Cookbook sre.pki.restart-reboot (exit_code=0) rolling reboot on A:pki * 14:57 claime: Disabling puppet on A:cp-text - [[phab:T426323|T426323]] * 14:56 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1290805{{!}}hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:55 klausman@cumin1003: START - Cookbook sre.hosts.reboot-single for host ml-lab1002.eqiad.wmnet * 14:54 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-build1001.eqiad.wmnet * 14:54 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1290805{{!}}hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] * 14:54 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet * 14:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1013.eqiad.wmnet * 14:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1037.eqiad.wmnet * 14:53 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1028.eqiad.wmnet * 14:53 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>ml-serve1001.eqiad.wmnet<nowiki>}</nowiki> and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) * 14:53 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1001.eqiad.wmnet * 14:53 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1001.eqiad.wmnet * 14:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1028.eqiad.wmnet * 14:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92798 and previous config saved to /var/cache/conftool/dbconfig/20260521-145132-fceratto.json * 14:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2040.codfw.wmnet with reason: Maintenance * 14:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92797 and previous config saved to /var/cache/conftool/dbconfig/20260521-145103-fceratto.json * 14:50 klausman@cumin1003: START - Cookbook sre.hosts.reboot-single for host ml-build1001.eqiad.wmnet * 14:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: Migration of db2241.codfw.wmnet completed * 14:48 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1001.eqiad.wmnet * 14:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1013.eqiad.wmnet * 14:46 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1028.eqiad.wmnet * 14:45 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:44 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:42 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1001.eqiad.wmnet * 14:42 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>ml-serve1001.eqiad.wmnet<nowiki>}</nowiki> and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) * 14:42 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1028.eqiad.wmnet * 14:42 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-worker-eqiad * 14:42 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1011.eqiad.wmnet * 14:42 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1011.eqiad.wmnet * 14:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039', diff saved to https://phabricator.wikimedia.org/P92795 and previous config saved to /var/cache/conftool/dbconfig/20260521-144055-fceratto.json * 14:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1012.eqiad.wmnet * 14:38 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet. on all recursors * 14:37 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet. on all recursors * 14:37 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1011.eqiad.wmnet * 14:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1027.eqiad.wmnet * 14:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1027.eqiad.wmnet * 14:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1011.eqiad.wmnet * 14:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1012.eqiad.wmnet * 14:32 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1010.eqiad.wmnet * 14:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1010.eqiad.wmnet * 14:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039', diff saved to https://phabricator.wikimedia.org/P92793 and previous config saved to /var/cache/conftool/dbconfig/20260521-143045-fceratto.json * 14:30 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet. on all recursors * 14:30 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet. on all recursors * 14:29 elukey@cumin1003: START - Cookbook sre.pki.restart-reboot rolling reboot on A:pki * 14:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1027.eqiad.wmnet * 14:27 slyngshede@cumin1003: END (FAIL) - Cookbook sre.cdn.roll-reboot (exit_code=1) rolling reboot on P<nowiki>{</nowiki>cp601[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 14:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1027.eqiad.wmnet * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1054.eqiad.wmnet * 14:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1054.eqiad.wmnet * 14:24 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1010.eqiad.wmnet * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1011.eqiad.wmnet * 14:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92792 and previous config saved to /var/cache/conftool/dbconfig/20260521-142037-fceratto.json * 14:19 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1054.eqiad.wmnet * 14:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1054.eqiad.wmnet * 14:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1053.eqiad.wmnet * 14:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1053.eqiad.wmnet * 14:14 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1010.eqiad.wmnet * 14:14 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1009.eqiad.wmnet * 14:14 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1009.eqiad.wmnet * 14:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:13 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1011.eqiad.wmnet * 14:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:12 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2218: repool after maintenance * 14:11 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1053.eqiad.wmnet * 14:09 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92789 and previous config saved to /var/cache/conftool/dbconfig/20260521-140906-fceratto.json * 14:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2039.codfw.wmnet with reason: Maintenance * 14:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92788 and previous config saved to /var/cache/conftool/dbconfig/20260521-140837-fceratto.json * 14:08 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1009.eqiad.wmnet * 14:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:07 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1053.eqiad.wmnet * 14:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1035.eqiad.wmnet * 14:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1035.eqiad.wmnet * 14:04 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2241: Migration of db2241.codfw.wmnet completed * 14:03 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1009.eqiad.wmnet * 14:03 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1008.eqiad.wmnet * 14:03 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1008.eqiad.wmnet * 14:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2241.codfw.wmnet with OS trixie * 13:59 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 13:59 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1035.eqiad.wmnet * 13:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92786 and previous config saved to /var/cache/conftool/dbconfig/20260521-135830-fceratto.json * 13:58 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1008.eqiad.wmnet * 13:53 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1008.eqiad.wmnet * 13:53 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1007.eqiad.wmnet * 13:53 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1007.eqiad.wmnet * 13:51 Lucas_WMDE: UTC afternoon backport+config window done * 13:51 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290743{{!}}composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347{{!}}Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342{{!}}fix: simplify to show only one icon type for password reveal (T419413)]] (duration: 07m 20s) * 13:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92784 and previous config saved to /var/cache/conftool/dbconfig/20260521-134822-fceratto.json * 13:48 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1007.eqiad.wmnet * 13:47 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 13:46 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Continuing with deployment * 13:45 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 13:45 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Backport for [[gerrit:1290743{{!}}composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347{{!}}Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342{{!}}fix: simplify to show only one icon type for password reveal (T419413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes * 13:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2241.codfw.wmnet with reason: host reimage * 13:44 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 13:43 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1290743{{!}}composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347{{!}}Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342{{!}}fix: simplify to show only one icon type for password reveal (T419413)]] * 13:43 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 13:43 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1007.eqiad.wmnet * 13:42 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1006.eqiad.wmnet * 13:42 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1006.eqiad.wmnet * 13:41 dbrant@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290035{{!}}docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] (duration: 06m 52s) * 13:41 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 13:40 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2241.codfw.wmnet with reason: host reimage * 13:39 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1035.eqiad.wmnet * 13:38 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in codfw/ml-serve-codfw: maintenance * 13:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92782 and previous config saved to /var/cache/conftool/dbconfig/20260521-133815-fceratto.json * 13:37 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1006.eqiad.wmnet * 13:37 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in codfw/ml-serve-codfw: maintenance * 13:37 dbrant@deploy1003: dbrant: Continuing with deployment * 13:36 dbrant@deploy1003: dbrant: Backport for [[gerrit:1290035{{!}}docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1032.eqiad.wmnet * 13:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1032.eqiad.wmnet * 13:35 dbrant@deploy1003: Started scap sync-world: Backport for [[gerrit:1290035{{!}}docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] * 13:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1006.eqiad.wmnet * 13:32 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1005.eqiad.wmnet * 13:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1005.eqiad.wmnet * 13:31 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290014{{!}}Enable AG on phase 2 wikis (T426871)]] (duration: 09m 11s) * 13:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92781 and previous config saved to /var/cache/conftool/dbconfig/20260521-133116-fceratto.json * 13:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1048.eqiad.wmnet with reason: Maintenance * 13:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92780 and previous config saved to /var/cache/conftool/dbconfig/20260521-133048-fceratto.json * 13:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1032.eqiad.wmnet * 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1032.eqiad.wmnet * 13:27 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1005.eqiad.wmnet * 13:27 sbisson@deploy1003: sbisson: Continuing with deployment * 13:27 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2218: repool after maintenance * 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1031.eqiad.wmnet * 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1031.eqiad.wmnet * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:25 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2241.codfw.wmnet with OS trixie * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:24 sbisson@deploy1003: sbisson: Backport for [[gerrit:1290014{{!}}Enable AG on phase 2 wikis (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:23 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: Upgrading db2241.codfw.wmnet * 13:23 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2241: Upgrading db2241.codfw.wmnet * 13:23 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:22 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1290014{{!}}Enable AG on phase 2 wikis (T426871)]] * 13:22 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1005.eqiad.wmnet * 13:22 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1004.eqiad.wmnet * 13:22 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1004.eqiad.wmnet * 13:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040', diff saved to https://phabricator.wikimedia.org/P92778 and previous config saved to /var/cache/conftool/dbconfig/20260521-132041-fceratto.json * 13:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1031.eqiad.wmnet * 13:20 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290088{{!}}Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032{{!}}Enable 'flood' user group at en.wikiversity (T426882)]] (duration: 11m 55s) * 13:18 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet * 13:17 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1018.eqiad.wmnet with OS trixie * 13:16 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1031.eqiad.wmnet * 13:16 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1039: Repooling * 13:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1030.eqiad.wmnet * 13:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1030.eqiad.wmnet * 13:15 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Continuing with deployment * 13:15 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1004.eqiad.wmnet * 13:14 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet * 13:11 eevans@cumin1003: START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:restbase * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . * 13:10 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1004.eqiad.wmnet * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . * 13:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040', diff saved to https://phabricator.wikimedia.org/P92776 and previous config saved to /var/cache/conftool/dbconfig/20260521-131033-fceratto.json * 13:10 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1003.eqiad.wmnet * 13:10 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1003.eqiad.wmnet * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 13:10 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2241 [[phab:T426936|T426936]]', diff saved to https://phabricator.wikimedia.org/P92775 and previous config saved to /var/cache/conftool/dbconfig/20260521-131025-cwilliams.json * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'readability' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'logo-detection' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . * 13:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1030.eqiad.wmnet * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . * 13:10 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Backport for [[gerrit:1290088{{!}}Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032{{!}}Enable 'flood' user group at en.wikiversity (T426882)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'llm' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . * 13:08 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1290088{{!}}Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032{{!}}Enable 'flood' user group at en.wikiversity (T426882)]] * 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet * 13:06 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp601[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 13:06 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3074.esams.wmnet<nowiki>}</nowiki> and A:cp * 13:06 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3074.esams.wmnet * 13:06 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2162 to x3 primary [[phab:T426936|T426936]]', diff saved to https://phabricator.wikimedia.org/P92774 and previous config saved to /var/cache/conftool/dbconfig/20260521-130609-cwilliams.json * 13:04 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 13:04 cezmunsta: Starting x3 codfw failover from db2241 to db2162 - [[phab:T426936|T426936]] * 13:04 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1003.eqiad.wmnet * 13:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1030.eqiad.wmnet * 13:03 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet * 13:00 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 13:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92772 and previous config saved to /var/cache/conftool/dbconfig/20260521-130018-fceratto.json * 12:59 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1003.eqiad.wmnet * 12:59 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1018.eqiad.wmnet with reason: host reimage * 12:59 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1002.eqiad.wmnet * 12:59 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1002.eqiad.wmnet * 12:58 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:57 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:56 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2162 with weight 0 [[phab:T426936|T426936]]', diff saved to https://phabricator.wikimedia.org/P92771 and previous config saved to /var/cache/conftool/dbconfig/20260521-125645-cwilliams.json * 12:56 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Primary switchover x3 [[phab:T426936|T426936]] * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:55 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1029.eqiad.wmnet * 12:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1029.eqiad.wmnet * 12:54 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3074.esams.wmnet<nowiki>}</nowiki> and A:cp * 12:54 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1002.eqiad.wmnet * 12:54 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp600[7-8].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 12:54 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6008.drmrs.wmnet * 12:53 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:52 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1018.eqiad.wmnet with reason: host reimage * 12:51 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:49 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1002.eqiad.wmnet * 12:49 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-worker-eqiad * 12:48 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1029.eqiad.wmnet * 12:48 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3066.esams.wmnet<nowiki>}</nowiki> and A:cp * 12:48 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3066.esams.wmnet * 12:47 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:47 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92770 and previous config saved to /var/cache/conftool/dbconfig/20260521-124707-fceratto.json * 12:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1040.eqiad.wmnet with reason: Maintenance * 12:46 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es1039: Repooling * 12:46 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:45 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1029.eqiad.wmnet * 12:45 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:44 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:43 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:43 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290727{{!}}hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] (duration: 07m 54s) * 12:42 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92768 and previous config saved to /var/cache/conftool/dbconfig/20260521-124014-fceratto.json * 12:39 kharlan@deploy1003: kharlan: Continuing with deployment * 12:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1052.eqiad.wmnet * 12:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1052.eqiad.wmnet * 12:37 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1018.eqiad.wmnet with OS trixie * 12:37 kharlan@deploy1003: kharlan: Backport for [[gerrit:1290727{{!}}hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:36 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:36 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3066.esams.wmnet<nowiki>}</nowiki> and A:cp * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:35 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1290727{{!}}hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:34 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1017.eqiad.wmnet with OS trixie * 12:34 kart_: Updated cxserver to 2026-05-20-034002-production ([[phab:T388690|T388690]], [[phab:T404295|T404295]], [[phab:T391703|T391703]], [[phab:T426605|T426605]]) * 12:34 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet * 12:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1052.eqiad.wmnet * 12:30 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply * 12:30 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply * 12:30 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet * 12:29 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:29 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92767 and previous config saved to /var/cache/conftool/dbconfig/20260521-122905-fceratto.json * 12:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1039.eqiad.wmnet with reason: Maintenance * 12:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92766 and previous config saved to /var/cache/conftool/dbconfig/20260521-122839-fceratto.json * 12:27 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply * 12:27 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply * 12:26 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:23 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-staging-worker * 12:23 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2003.codfw.wmnet * 12:23 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2003.codfw.wmnet * 12:22 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1052.eqiad.wmnet * 12:21 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply * 12:21 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply * 12:21 moritzm: installing nginx security updates * 12:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1051.eqiad.wmnet * 12:20 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in codfw/ml-serve-codfw: maintenance * 12:19 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1017.eqiad.wmnet with reason: host reimage * 12:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1051.eqiad.wmnet * 12:19 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in codfw/ml-serve-codfw: maintenance * 12:19 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in codfw/ml-staging-codfw: maintenance * 12:19 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in codfw/ml-staging-codfw: maintenance * 12:19 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in codfw/ml-staging-codfw: maintenance * 12:18 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in codfw/ml-staging-codfw: maintenance * 12:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047', diff saved to https://phabricator.wikimedia.org/P92765 and previous config saved to /var/cache/conftool/dbconfig/20260521-121832-fceratto.json * 12:17 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2003.codfw.wmnet * 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet * 12:15 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1017.eqiad.wmnet with reason: host reimage * 12:14 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1051.eqiad.wmnet * 12:13 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6007.drmrs.wmnet * 12:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet * 12:10 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1051.eqiad.wmnet * 12:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047', diff saved to https://phabricator.wikimedia.org/P92764 and previous config saved to /var/cache/conftool/dbconfig/20260521-120824-fceratto.json * 12:07 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2003.codfw.wmnet * 12:07 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2002.codfw.wmnet * 12:07 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2002.codfw.wmnet * 12:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1050.eqiad.wmnet * 12:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1050.eqiad.wmnet * 12:02 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp600[7-8].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp601[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6014.drmrs.wmnet * 12:00 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1017.eqiad.wmnet with OS trixie * 12:00 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2002.codfw.wmnet * 11:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1002.wikimedia.org * 11:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92763 and previous config saved to /var/cache/conftool/dbconfig/20260521-115817-fceratto.json * 11:57 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1050.eqiad.wmnet * 11:53 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt1002.wikimedia.org * 11:51 taavi: disabling puppet on C:bird to roll out {{Gerrit|1289919}} * 11:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92762 and previous config saved to /var/cache/conftool/dbconfig/20260521-115112-fceratto.json * 11:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2047.codfw.wmnet with reason: Maintenance * 11:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1050.eqiad.wmnet * 11:50 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2002.codfw.wmnet * 11:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92761 and previous config saved to /var/cache/conftool/dbconfig/20260521-115043-fceratto.json * 11:50 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2001.codfw.wmnet * 11:50 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2001.codfw.wmnet * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1049.eqiad.wmnet * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt2002.wikimedia.org * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1049.eqiad.wmnet * 11:45 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2001.codfw.wmnet * 11:45 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker-exp1001.eqiad.wmnet * 11:44 kartik@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 11:44 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1049.eqiad.wmnet * 11:43 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt2002.wikimedia.org * 11:42 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet * 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet * 11:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037', diff saved to https://phabricator.wikimedia.org/P92760 and previous config saved to /var/cache/conftool/dbconfig/20260521-114036-fceratto.json * 11:39 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker-exp1001.eqiad.wmnet * 11:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker-exp2001.codfw.wmnet * 11:38 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host testreduce1002.eqiad.wmnet * 11:37 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1049.eqiad.wmnet * 11:36 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet * 11:36 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet * 11:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1038.eqiad.wmnet * 11:35 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2001.codfw.wmnet * 11:35 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-staging-worker * 11:35 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet * 11:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1038.eqiad.wmnet * 11:34 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host testreduce1002.eqiad.wmnet * 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker-exp2001.codfw.wmnet * 11:32 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet * 11:31 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet * 11:30 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt-staging2001.codfw.wmnet * 11:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037', diff saved to https://phabricator.wikimedia.org/P92759 and previous config saved to /var/cache/conftool/dbconfig/20260521-113028-fceratto.json * 11:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2014.codfw.wmnet * 11:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1038.eqiad.wmnet * 11:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt-staging2001.codfw.wmnet * 11:26 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet * 11:24 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1038.eqiad.wmnet * 11:24 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1034.eqiad.wmnet * 11:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1034.eqiad.wmnet * 11:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2014.codfw.wmnet * 11:20 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6013.drmrs.wmnet * 11:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92758 and previous config saved to /var/cache/conftool/dbconfig/20260521-112021-fceratto.json * 11:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1034.eqiad.wmnet * 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling reboot on A:ldap-replicas-eqiad * 11:13 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2013.codfw.wmnet * 11:11 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1034.eqiad.wmnet * 11:09 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp601[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92757 and previous config saved to /var/cache/conftool/dbconfig/20260521-110851-fceratto.json * 11:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2037.codfw.wmnet with reason: Maintenance * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92756 and previous config saved to /var/cache/conftool/dbconfig/20260521-110822-fceratto.json * 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1033.eqiad.wmnet * 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1033.eqiad.wmnet * 11:05 jmm@cumin2002: START - Cookbook sre.ldap.roll-restart-reboot-replica rolling reboot on A:ldap-replicas-eqiad * 11:05 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2013.codfw.wmnet * 11:04 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp600[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 11:04 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6006.drmrs.wmnet * 11:02 jmm@cumin2002: END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling reboot on A:ldap-replicas-codfw * 11:00 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1033.eqiad.wmnet * 10:59 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1016.eqiad.wmnet with reason: host reimage * 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036', diff saved to https://phabricator.wikimedia.org/P92753 and previous config saved to /var/cache/conftool/dbconfig/20260521-105815-fceratto.json * 10:57 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1033.eqiad.wmnet * 10:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1044.eqiad.wmnet * 10:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1044.eqiad.wmnet * 10:55 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1016.eqiad.wmnet with reason: host reimage * 10:54 jmm@cumin2002: START - Cookbook sre.ldap.roll-restart-reboot-replica rolling reboot on A:ldap-replicas-codfw * 10:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2012.codfw.wmnet * 10:51 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:51 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:51 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1044.eqiad.wmnet * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036', diff saved to https://phabricator.wikimedia.org/P92752 and previous config saved to /var/cache/conftool/dbconfig/20260521-104807-fceratto.json * 10:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2012.codfw.wmnet * 10:46 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1044.eqiad.wmnet * 10:44 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290709{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] (duration: 08m 02s) * 10:43 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:41 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:40 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet * 10:40 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:39 jiji@deploy1003: jiji: Continuing with deployment * 10:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92751 and previous config saved to /var/cache/conftool/dbconfig/20260521-103759-fceratto.json * 10:37 jiji@deploy1003: jiji: Backport for [[gerrit:1290709{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:36 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1290709{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] * 10:35 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet * 10:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1043.eqiad.wmnet * 10:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1043.eqiad.wmnet * 10:34 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:29 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 10:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1043.eqiad.wmnet * 10:27 dcausse: [[phab:T423993|T423993]]: reindexing all archive indices * 10:27 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . * 10:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92749 and previous config saved to /var/cache/conftool/dbconfig/20260521-102630-fceratto.json * 10:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance * 10:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1043.eqiad.wmnet * 10:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92748 and previous config saved to /var/cache/conftool/dbconfig/20260521-102601-fceratto.json * 10:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2011.codfw.wmnet * 10:24 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6005.drmrs.wmnet * 10:22 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1042.eqiad.wmnet * 10:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1042.eqiad.wmnet * 10:17 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2011.codfw.wmnet * 10:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1042.eqiad.wmnet * 10:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047', diff saved to https://phabricator.wikimedia.org/P92747 and previous config saved to /var/cache/conftool/dbconfig/20260521-101552-fceratto.json * 10:15 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:14 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . * 10:13 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1042.eqiad.wmnet * 10:13 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1041.eqiad.wmnet * 10:12 moritzm: installing postgresql security updates * 10:12 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp600[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 10:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1041.eqiad.wmnet * 10:10 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet * 10:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org * 10:09 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 10:08 fnegri@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb1013.eqiad.wmnet * 10:08 fnegri@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb1013.eqiad.wmnet * 10:07 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1013.eqiad.wmnet * 10:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1041.eqiad.wmnet * 10:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047', diff saved to https://phabricator.wikimedia.org/P92746 and previous config saved to /var/cache/conftool/dbconfig/20260521-100545-fceratto.json * 10:05 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet * 10:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1041.eqiad.wmnet * 10:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1040.eqiad.wmnet * 10:04 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet * 10:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1040.eqiad.wmnet * 10:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org * 10:01 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1013.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 10:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1040.eqiad.wmnet * 10:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org * 09:59 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet * 09:58 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-codfw * 09:58 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2005.codfw.wmnet * 09:58 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2005.codfw.wmnet * 09:56 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1040.eqiad.wmnet * 09:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1039.eqiad.wmnet * 09:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1039.eqiad.wmnet * 09:56 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 09:56 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:55 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ml-serve1013.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 09:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92745 and previous config saved to /var/cache/conftool/dbconfig/20260521-095536-fceratto.json * 09:54 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1384.eqiad.wmnet * 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org * 09:54 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:54 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:52 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2005.codfw.wmnet * 09:52 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2005.codfw.wmnet * 09:52 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply * 09:52 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2004.codfw.wmnet * 09:52 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2004.codfw.wmnet * 09:51 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply * 09:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1039.eqiad.wmnet * 09:49 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1384.eqiad.wmnet * 09:49 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 09:49 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1383.eqiad.wmnet * 09:48 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1039.eqiad.wmnet * 09:48 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1036.eqiad.wmnet * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92744 and previous config saved to /var/cache/conftool/dbconfig/20260521-094829-fceratto.json * 09:48 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1036.eqiad.wmnet * 09:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1047.eqiad.wmnet with reason: Maintenance * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92743 and previous config saved to /var/cache/conftool/dbconfig/20260521-094801-fceratto.json * 09:47 fnegri@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1013.eqiad.wmnet * 09:47 fnegri@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1013.eqiad.wmnet with reason: Rebooting clouddb1013 [[phab:T426563|T426563]] * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2004.codfw.wmnet * 09:45 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2004.codfw.wmnet * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet * 09:45 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet * 09:45 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet * 09:44 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1383.eqiad.wmnet * 09:44 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 09:44 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1382.eqiad.wmnet * 09:42 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2002.codfw.wmnet * 09:40 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1036.eqiad.wmnet * 09:39 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet * 09:38 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1382.eqiad.wmnet * 09:38 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1381.eqiad.wmnet * 09:38 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1036.eqiad.wmnet * 09:38 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2003.codfw.wmnet * 09:38 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2003.codfw.wmnet * 09:38 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2002.codfw.wmnet * 09:38 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2002.codfw.wmnet * 09:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037', diff saved to https://phabricator.wikimedia.org/P92742 and previous config saved to /var/cache/conftool/dbconfig/20260521-093754-fceratto.json * 09:37 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 09:37 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet * 09:37 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet * 09:37 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet * 09:37 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet * 09:36 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host build2002.codfw.wmnet * 09:36 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 09:35 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp601[1-2].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 09:35 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6012.drmrs.wmnet * 09:34 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet * 09:33 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet * 09:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1381.eqiad.wmnet * 09:33 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1380.eqiad.wmnet * 09:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1023.eqiad.wmnet * 09:31 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet * 09:31 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2002.codfw.wmnet * 09:31 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2002.codfw.wmnet * 09:31 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2001.codfw.wmnet * 09:31 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2001.codfw.wmnet * 09:30 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet * 09:30 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet * 09:30 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet * 09:30 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet * 09:29 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet * 09:29 jayme@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad * 09:29 jayme@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw * 09:29 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet * 09:28 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet * 09:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037', diff saved to https://phabricator.wikimedia.org/P92741 and previous config saved to /var/cache/conftool/dbconfig/20260521-092746-fceratto.json * 09:27 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1380.eqiad.wmnet * 09:27 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1379.eqiad.wmnet * 09:27 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet * 09:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1023.eqiad.wmnet * 09:25 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet * 09:24 jayme@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw * 09:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1056.eqiad.wmnet to cluster eqiad and group A * 09:23 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet * 09:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad * 09:22 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1379.eqiad.wmnet * 09:22 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1378.eqiad.wmnet * 09:21 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2001.codfw.wmnet * 09:21 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2001.codfw.wmnet * 09:21 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-codfw * 09:21 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1056.eqiad.wmnet to cluster eqiad and group A * 09:20 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 09:18 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-jumbo1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:18 moritzm: remove ganeti1023 foom eqiad Ganeti cluster [[phab:T424680|T424680]] * 09:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92740 and previous config saved to /var/cache/conftool/dbconfig/20260521-091738-fceratto.json * 09:16 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1378.eqiad.wmnet * 09:16 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1377.eqiad.wmnet * 09:12 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1377.eqiad.wmnet * 09:12 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1376.eqiad.wmnet * 09:07 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1036: Repooling * 09:07 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1376.eqiad.wmnet * 09:07 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1375.eqiad.wmnet * 09:06 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92738 and previous config saved to /var/cache/conftool/dbconfig/20260521-090609-fceratto.json * 09:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1037.eqiad.wmnet with reason: Maintenance * 09:02 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1375.eqiad.wmnet * 09:01 btullis@cumin1003: START - Cookbook sre.hosts.provision for host kafka-jumbo1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:55 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6011.drmrs.wmnet * 08:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1023.eqiad.wmnet * 08:47 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 08:47 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1256: Migration of db1256.eqiad.wmnet completed * 08:44 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp601[1-2].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 08:42 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp600[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 08:42 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6004.drmrs.wmnet * 08:37 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es1036: Repooling * 08:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92733 and previous config saved to /var/cache/conftool/dbconfig/20260521-082951-fceratto.json * 08:29 hashar@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.3 refs [[phab:T423912|T423912]] * 08:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92731 and previous config saved to /var/cache/conftool/dbconfig/20260521-081642-fceratto.json * 08:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1036.eqiad.wmnet with reason: Maintenance * 08:02 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1256: Migration of db1256.eqiad.wmnet completed * 08:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6003.drmrs.wmnet * 08:00 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1256.eqiad.wmnet with OS trixie * 07:52 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp600[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:51 marostegui@dns1004: END - running authdns-update * 07:50 marostegui@dns1004: START - running authdns-update * 07:48 marostegui: Failover m3-master [[phab:T426633|T426633]] * 07:47 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1023.eqiad.wmnet * 07:46 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp6010.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:46 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6010.drmrs.wmnet * 07:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster1005.eqiad.wmnet to plain * 07:44 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster1005.eqiad.wmnet to plain * 07:43 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1256.eqiad.wmnet with reason: host reimage * 07:42 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster1005.eqiad.wmnet to drbd * 07:38 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1256.eqiad.wmnet with reason: host reimage * 07:35 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp6010.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:35 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp6002.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:35 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6002.drmrs.wmnet * 07:27 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster1005.eqiad.wmnet to drbd * 07:24 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp6002.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:24 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1256.eqiad.wmnet with OS trixie * 07:22 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1256: Upgrading db1256.eqiad.wmnet * 07:21 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1256: Upgrading db1256.eqiad.wmnet * 07:21 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to plain * 07:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to plain * 07:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbproxy1025.eqiad.wmnet with reason: Rebooting * 07:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to drbd * 06:54 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to drbd * 06:53 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to plain * 06:52 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to plain * 06:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to drbd * 06:42 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists1004.wikimedia.org * 06:40 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org * 06:39 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1003.eqiad.wmnet * 06:34 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org * 06:34 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host lists1004.wikimedia.org * 06:33 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host vrts1003.eqiad.wmnet * 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to drbd * 06:23 arnaudb@cumin1003: END (FAIL) - Cookbook sre.gerrit.reboot-gerrit (exit_code=99) Rebooting Gerrit on gerrit2003 * 06:22 arnaudb@cumin1003: START - Cookbook sre.gerrit.reboot-gerrit Rebooting Gerrit on gerrit2003 * 06:15 marostegui@dns1004: END - running authdns-update * 06:14 marostegui: Failover m2-master [[phab:T426633|T426633]] * 06:13 marostegui@dns1004: START - running authdns-update * 05:39 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1012 from dbctl [[phab:T426930|T426930]]', diff saved to https://phabricator.wikimedia.org/P92728 and previous config saved to /var/cache/conftool/dbconfig/20260521-053858-marostegui.json * 05:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc2 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92727 and previous config saved to /var/cache/conftool/dbconfig/20260521-053000-marostegui.json * 05:29 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1022 to pc2 master [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92726 and previous config saved to /var/cache/conftool/dbconfig/20260521-052905-marostegui.json * 05:21 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc1012.eqiad.wmnet with reason: Cloning * 02:41 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on planet1003.eqiad.wmnet with reason: debug wip * 02:11 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 29s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:29 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1027.eqiad.wmnet * 01:22 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1027.eqiad.wmnet * 00:55 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 4r172gf9ll5eypufx60sw74utnhwulj 2426650 2426645 2026-06-14T11:02:29Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2426650 wikitext text/x-wiki == 2026-06-14 == * 11:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:06 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-06-13 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 35s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-06-12 == * 19:54 dwisehaupt@dns1004: END - running authdns-update * 19:52 dwisehaupt@dns1004: START - running authdns-update * 18:33 dwisehaupt@dns1006: END - running authdns-update * 18:32 dwisehaupt@dns1006: START - running authdns-update * 16:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:10 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:10 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:59 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 15:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:43 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1301371{{!}}Hotfix for T428620 (T428620)]] (duration: 11m 17s) * 14:36 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 14:35 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1301371{{!}}Hotfix for T428620 (T428620)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:31 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1301371{{!}}Hotfix for T428620 (T428620)]] * 14:29 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:28 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 13:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 12:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 12:22 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:22 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:22 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:22 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 12:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 12:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 12:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 12:04 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:04 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:04 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 12:02 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of prometheus5003.eqsin.wmnet to drbd * 12:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus5003.eqsin.wmnet to drbd * 11:40 moritzm: installing Linux 5.10.257 on Bullseye hosts * 11:36 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 11:35 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:24 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 11:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:56 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 10:56 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 10:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:49 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 10:49 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 10:40 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply * 10:37 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply * 10:36 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply * 10:35 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply * 10:35 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply * 10:35 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply * 10:12 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 10:12 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 10:08 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 09:59 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 09:58 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 09:57 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 06:13 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) * 06:11 jmm@cumin2002: START - Cookbook sre.puppet.disable-merges * 03:07 ryankemper: [[phab:T427951|T427951]] sorry, `[eqiad,codfw].mediawiki.page_html_content_change.rc0` (accidentally a word) * 03:06 ryankemper: [[phab:T427951|T427951]] Deleted all 20 unused dev/test topics on kafka-jumbo (verified empty first); 2 (`[eqiad,codfw]page_html_content_change.rc0`) were immediately auto-recreated empty by a still-running `dse-k8s` enrichment consumer; awaiting owner confirmation before final re-delete * 02:01 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 01m 13s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:00 bblack@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () == 2026-06-11 == * 22:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 22:26 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 22:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 22:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 22:05 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300906{{!}}Restore MediaViewer toggle in Special:Preferences (T428742)]] (duration: 30m 51s) * 21:58 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host releases2003.codfw.wmnet with OS trixie * 21:52 egardner@deploy1003: egardner: Continuing with deployment * 21:51 egardner@deploy1003: egardner: Backport for [[gerrit:1300906{{!}}Restore MediaViewer toggle in Special:Preferences (T428742)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:34 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1300906{{!}}Restore MediaViewer toggle in Special:Preferences (T428742)]] * 21:34 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet with reason: host reimage * 21:29 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300913{{!}}Avoid the escaping from nowiki processing (T398967)]] (duration: 09m 09s) * 21:28 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on releases2003.codfw.wmnet with reason: host reimage * 21:25 arlolra@deploy1003: arlolra: Continuing with deployment * 21:22 arlolra@deploy1003: arlolra: Backport for [[gerrit:1300913{{!}}Avoid the escaping from nowiki processing (T398967)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:20 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1300913{{!}}Avoid the escaping from nowiki processing (T398967)]] * 21:07 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300911{{!}}hCaptcha: Enable for badlogin for all small wikis (T426875)]], [[gerrit:1300905{{!}}RadioRangeBallot: Fix strict mode issue (T428947)]] (duration: 10m 43s) * 21:06 bblack@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text and not P<nowiki>{</nowiki>cp7008*<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 21:01 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 21:00 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300911{{!}}hCaptcha: Enable for badlogin for all small wikis (T426875)]], [[gerrit:1300905{{!}}RadioRangeBallot: Fix strict mode issue (T428947)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:56 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300911{{!}}hCaptcha: Enable for badlogin for all small wikis (T426875)]], [[gerrit:1300905{{!}}RadioRangeBallot: Fix strict mode issue (T428947)]] * 20:51 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300842{{!}}Donor Delight Badge: Unify on "Remove badge" language across treatments (T427313)]], [[gerrit:1300843{{!}}[A11y] Donor Badge: Remove Badge button disappears too quickly (T428646)]], [[gerrit:1300896{{!}}Donor Delight Badge, styles: Amending to final design review feedback (T427313)]] (duration: 34m 10s) * 20:39 jdrewniak@deploy1003: annet, jdrewniak: Continuing with deployment * 20:35 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host releases2003.codfw.wmnet with OS trixie * 20:34 jdrewniak@deploy1003: annet, jdrewniak: Backport for [[gerrit:1300842{{!}}Donor Delight Badge: Unify on "Remove badge" language across treatments (T427313)]], [[gerrit:1300843{{!}}[A11y] Donor Badge: Remove Badge button disappears too quickly (T428646)]], [[gerrit:1300896{{!}}Donor Delight Badge, styles: Amending to final design review feedback (T427313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug * 20:17 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1300842{{!}}Donor Delight Badge: Unify on "Remove badge" language across treatments (T427313)]], [[gerrit:1300843{{!}}[A11y] Donor Badge: Remove Badge button disappears too quickly (T428646)]], [[gerrit:1300896{{!}}Donor Delight Badge, styles: Amending to final design review feedback (T427313)]] * 19:12 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 18:12 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 18:12 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 17:52 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300865{{!}}UploadWizard.config.php: Fix cc-by-4.0-heirs msg issue (T428935 T405146)]] (duration: 08m 15s) * 17:48 reedy@deploy1003: reedy: Continuing with deployment * 17:46 reedy@deploy1003: reedy: Backport for [[gerrit:1300865{{!}}UploadWizard.config.php: Fix cc-by-4.0-heirs msg issue (T428935 T405146)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:44 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1300865{{!}}UploadWizard.config.php: Fix cc-by-4.0-heirs msg issue (T428935 T405146)]] * 17:26 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 17:25 blake@deploy1003: Scap cancelled without rolling back. * 17:25 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 17:24 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 17:24 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 17:24 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 17:24 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 17:23 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 17:23 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 17:23 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 17:23 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 17:23 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 17:23 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 17:20 blake@deploy1003: blake: apache config update ([[phab:T428772|T428772]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:20 blake@deploy1003: Started scap sync-world: apache config update ([[phab:T428772|T428772]]) * 17:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 17:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2212: Migration of db2212.codfw.wmnet completed * 17:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 17:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1235: Migration of db1235.eqiad.wmnet completed * 17:08 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 16:45 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:43 dzahn@dns1005: END - running authdns-update * 16:42 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:41 dzahn@dns1005: START - running authdns-update * 16:41 mutante: releases.wikimedia.org - switching backend from codfw to eqiad - releases1003 is now the source of rsync for uploaded releases files (use releases.discovery.wmnet to not have to think about it) - [[phab:T418299|T418299]] * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts rdb2007.codfw.wmnet * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts rdb1011.eqiad.wmnet * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) * 16:34 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb2009.codfw.wmnet * 16:34 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:34 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2009.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:33 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Migration of db2212.codfw.wmnet completed * 16:27 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2009.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:27 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1235: Migration of db1235.eqiad.wmnet completed * 16:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2212.codfw.wmnet with OS trixie * 16:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1235.eqiad.wmnet with OS trixie * 16:13 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:07 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:06 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 16:05 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 16:05 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 16:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 16:04 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2212.codfw.wmnet with reason: host reimage * 16:01 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 16:01 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:01 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 16:01 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 16:00 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 16:00 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 16:00 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 16:00 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2212.codfw.wmnet with reason: host reimage * 15:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1235.eqiad.wmnet with reason: host reimage * 15:58 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 15:58 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 15:57 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 15:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 15:57 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 15:57 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 15:56 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2009.codfw.wmnet * 15:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:55 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb1011.eqiad.wmnet * 15:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 15:55 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2007.codfw.wmnet * 15:54 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 15:54 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1235.eqiad.wmnet with reason: host reimage * 15:54 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 15:53 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 15:53 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 15:40 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 15:40 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2212.codfw.wmnet with OS trixie * 15:39 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 15:39 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1235.eqiad.wmnet with OS trixie * 15:36 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 15:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1235: Upgrading db1235.eqiad.wmnet * 15:35 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 15:35 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1235: Upgrading db1235.eqiad.wmnet * 15:35 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:32 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 15:32 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:31 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 15:30 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300822{{!}}T428849: temporarily disable noisy warnings in HandleParsoidSectionLinks (T428849 T417530)]] (duration: 11m 29s) * 15:27 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2212: Upgrading db2212.codfw.wmnet * 15:26 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2212: Upgrading db2212.codfw.wmnet * 15:26 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:26 cscott@deploy1003: cscott: Continuing with deployment * 15:26 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1235: Upgrading db1235.eqiad.wmnet * 15:25 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1235: Upgrading db1235.eqiad.wmnet * 15:25 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:21 cscott@deploy1003: cscott: Backport for [[gerrit:1300822{{!}}T428849: temporarily disable noisy warnings in HandleParsoidSectionLinks (T428849 T417530)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:19 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1300822{{!}}T428849: temporarily disable noisy warnings in HandleParsoidSectionLinks (T428849 T417530)]] * 15:18 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 15:17 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 15:13 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 15:13 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 15:13 moritzm: installing libdbi-perl security updates * 14:53 moritzm: installing Bind security updates (just client-side tools/libraries) * 14:51 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry * 14:48 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry * 14:43 moritzm: installing Poppler security updates * 14:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:33 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 14:32 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 14:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1234: Migration of db1234.eqiad.wmnet completed * 14:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin02 and group 01 * 14:24 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin02 and group 01 * 14:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:18 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet * 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet * 14:00 Lucas_WMDE: UTC afternoon backport+config window done * 13:58 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300733{{!}}stream: webrequest.page_view_stats.dev0 (T428725)]] (duration: 08m 12s) * 13:57 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp5024.* * 13:55 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: name=cp5024.* * 13:55 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp5020.* * 13:54 javiermonton@deploy1003: javiermonton: Continuing with deployment * 13:52 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1300733{{!}}stream: webrequest.page_view_stats.dev0 (T428725)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:51 slyngshede@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P<nowiki>{</nowiki>lvs5004*<nowiki>}</nowiki> and A:liberica * 13:50 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1300733{{!}}stream: webrequest.page_view_stats.dev0 (T428725)]] * 13:50 slyngshede@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P<nowiki>{</nowiki>lvs5004*<nowiki>}</nowiki> and A:liberica * 13:50 slyngs: reloading liberica config on lvs5004 * 13:50 moritzm: installing openssl security updates * 13:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:46 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 13:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm * 13:46 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1234: Migration of db1234.eqiad.wmnet completed * 13:46 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 13:45 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 13:45 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 13:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2202.codfw.wmnet with OS trixie * 13:43 alexsanford@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298890{{!}}Add 2FA enforcement demotion config for phase 3 groups (T423120)]] (duration: 07m 19s) * 13:39 alexsanford@deploy1003: alexsanford: Continuing with deployment * 13:38 alexsanford@deploy1003: alexsanford: Backport for [[gerrit:1298890{{!}}Add 2FA enforcement demotion config for phase 3 groups (T423120)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:36 alexsanford@deploy1003: Started scap sync-world: Backport for [[gerrit:1298890{{!}}Add 2FA enforcement demotion config for phase 3 groups (T423120)]] * 13:36 slyngshede@dns1004: END - running authdns-update * 13:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1234.eqiad.wmnet with OS trixie * 13:34 moritzm: installing dovecot security updates * 13:34 slyngshede@dns1004: START - running authdns-update * 13:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300787{{!}}hCaptcha: Enable for MobileFrontend on all group1 wikis (T425940)]] (duration: 06m 59s) * 13:29 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 13:29 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 13:29 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 13:29 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 13:28 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 13:28 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 13:28 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 13:27 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300787{{!}}hCaptcha: Enable for MobileFrontend on all group1 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:26 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2202.codfw.wmnet with reason: host reimage * 13:25 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300787{{!}}hCaptcha: Enable for MobileFrontend on all group1 wikis (T425940)]] * 13:25 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/Android_FAQ 'Wikimedia Apps/FAQ/Android' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:24 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/Android_FAQ 'Wikimedia Apps/FAQ/Android' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:22 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300736{{!}}fix: correct intake-url and payload type for NCS experiment events (T422295)]] (duration: 06m 51s) * 13:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage * 13:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1234.eqiad.wmnet with reason: host reimage * 13:18 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Continuing with deployment * 13:18 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2202.codfw.wmnet with reason: host reimage * 13:18 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Backport for [[gerrit:1300736{{!}}fix: correct intake-url and payload type for NCS experiment events (T422295)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:18 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 13:17 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 13:16 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1300736{{!}}fix: correct intake-url and payload type for NCS experiment events (T422295)]] * 13:15 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage * 13:14 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/Android_FAQ 'Wikimedia Apps/FAQ/Android' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:13 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:13 gkyziridis@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300731{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] (duration: 08m 47s) * 13:13 andrewbogott: sudo -i reprepro --noskipold --component thirdparty/openstack-trixie-flamingo-backports update trixie-wikimedia * 13:12 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1234.eqiad.wmnet with reason: host reimage * 13:12 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 13:12 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/iOS_FAQ 'Wikimedia Apps/FAQ/iOS' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:12 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 13:12 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 13:11 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 13:11 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 13:11 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 13:11 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 13:11 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 13:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 13:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 13:09 gkyziridis@deploy1003: gkyziridis: Continuing with deployment * 13:06 gkyziridis@deploy1003: gkyziridis: Backport for [[gerrit:1300731{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:06 claime: echo 'https://api.wikimedia.org/service/lw/specs/openapi.yaml' {{!}} mwscript-k8s --attach -- purgeList.php * 13:04 gkyziridis@deploy1003: Started scap sync-world: Backport for [[gerrit:1300731{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] * 13:02 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2202.codfw.wmnet with OS trixie * 13:00 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:57 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1234.eqiad.wmnet with OS trixie * 12:55 moritzm: installing Exim security updates on Bullseye * 12:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ganeti5006 * 12:47 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti5006 * 12:46 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti5006 * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti5006.eqsin.wmnet 9.0.132.10.in-addr.arpa 9.0.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 12:46 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache ganeti5006.eqsin.wmnet 9.0.132.10.in-addr.arpa 9.0.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5006 - jmm@cumin2002" * 12:46 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5006 - jmm@cumin2002" * 12:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1234: Upgrading db1234.eqiad.wmnet * 12:44 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1234: Upgrading db1234.eqiad.wmnet * 12:44 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2188: Migration of db2188.codfw.wmnet completed * 12:29 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "UX improvements - oblivian@cumin1003" * 12:29 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: UX improvements - oblivian@cumin1003 * 12:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1232: Migration of db1232.eqiad.wmnet completed * 12:28 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: UX improvements - oblivian@cumin1003 * 12:28 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "UX improvements - oblivian@cumin1003" * 12:27 jmm@cumin2002: START - Cookbook sre.dns.netbox * 12:26 jmm@cumin2002: START - Cookbook sre.hosts.move-vlan for host ganeti5006 * 12:26 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm * 12:21 moritzm: remove ganeti5006 from eqsin cluster for reimage [[phab:T428229|T428229]] * 12:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet * 12:10 moritzm: installing openjdk-21 security updates on Bookworm * 12:03 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300764{{!}}Remove GrowthExperiments extension from closed wikis (T428884)]] (duration: 06m 53s) * 11:59 urbanecm@deploy1003: urbanecm: Continuing with deployment * 11:58 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1300764{{!}}Remove GrowthExperiments extension from closed wikis (T428884)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:56 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1300764{{!}}Remove GrowthExperiments extension from closed wikis (T428884)]] * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb1012.eqiad.wmnet * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb2010.codfw.wmnet * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:48 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2010.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 11:46 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:46 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb2008.codfw.wmnet * 11:46 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:46 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2188: Migration of db2188.codfw.wmnet completed * 11:44 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 11:43 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:43 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2010.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 11:43 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1232: Migration of db1232.eqiad.wmnet completed * 11:38 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:37 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 11:37 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 11:36 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 11:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2188.codfw.wmnet with OS trixie * 11:35 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb1012.eqiad.wmnet * 11:34 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2008.codfw.wmnet * 11:34 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2010.codfw.wmnet * 11:33 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 11:32 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 11:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1232.eqiad.wmnet with OS trixie * 11:27 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2002.codfw.wmnet * 11:25 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300749{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300751{{!}}hCaptcha: Enable for DiscussionTools on all wikis (T426039)]] (duration: 08m 38s) * 11:21 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 11:19 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300749{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300751{{!}}hCaptcha: Enable for DiscussionTools on all wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2188.codfw.wmnet with reason: host reimage * 11:17 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300749{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300751{{!}}hCaptcha: Enable for DiscussionTools on all wikis (T426039)]] * 11:15 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2188.codfw.wmnet with reason: host reimage * 11:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1232.eqiad.wmnet with reason: host reimage * 11:13 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc2002.codfw.wmnet * 11:13 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet * 11:11 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 11:09 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2001.codfw.wmnet * 11:09 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1232.eqiad.wmnet with reason: host reimage * 11:08 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet * 11:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:04 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc2001.codfw.wmnet * 11:04 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host testreduce1002.eqiad.wmnet * 11:04 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:02 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1262.eqiad.wmnet with reason: crash * 11:00 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 11:00 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host testreduce1002.eqiad.wmnet * 10:59 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 10:59 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 10:58 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 10:55 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2188.codfw.wmnet with OS trixie * 10:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2188: Upgrading db2188.codfw.wmnet * 10:52 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2188: Upgrading db2188.codfw.wmnet * 10:52 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:52 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1232.eqiad.wmnet with OS trixie * 10:48 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1232: Upgrading db1232.eqiad.wmnet * 10:48 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1232: Upgrading db1232.eqiad.wmnet * 10:48 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:33 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:32 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:31 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300734{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300727{{!}}hCaptcha: Enable for DiscussionTools on group 1 wikis (T426039)]] (duration: 11m 01s) * 10:26 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:23 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:23 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:22 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300734{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300727{{!}}hCaptcha: Enable for DiscussionTools on group 1 wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:20 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300734{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300727{{!}}hCaptcha: Enable for DiscussionTools on group 1 wikis (T426039)]] * 10:18 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:18 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:10 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 10:10 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 10:09 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2045.codfw.wmnet with OS trixie * 10:09 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repool es2046', diff saved to https://phabricator.wikimedia.org/P94069 and previous config saved to /var/cache/conftool/dbconfig/20260611-100221-marostegui.json * 10:01 marostegui@cumin1003: dbctl commit (dc=all): 'Depool es2046', diff saved to https://phabricator.wikimedia.org/P94068 and previous config saved to /var/cache/conftool/dbconfig/20260611-100145-marostegui.json * 10:01 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:59 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300580{{!}}ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] (duration: 15m 41s) * 09:54 jiji@deploy1003: jiji: Continuing with deployment * 09:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2045.codfw.wmnet with reason: host reimage * 09:45 jiji@deploy1003: jiji: Backport for [[gerrit:1300580{{!}}ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:43 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1300580{{!}}ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] * 09:42 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2045.codfw.wmnet with reason: host reimage * 09:37 elukey: uploaded spicerack_12.8.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia * 09:26 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS trixie * 09:26 marostegui@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host es2045.codfw.wmnet with OS bookworm * 09:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2176: Migration of db2176.codfw.wmnet completed * 09:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1219: Migration of db1219.eqiad.wmnet completed * 09:11 claime: cumin -x 'A:swift-fe' "disable-puppet 'Disabling puppet for ratelimit deploy - cgoubert'" * 08:57 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS bookworm * 08:39 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2176: Migration of db2176.codfw.wmnet completed * 08:34 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94055) * 08:34 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1219: Migration of db1219.eqiad.wmnet completed * 08:33 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94053) * 08:30 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T428823|T428823]] (duration: 01m 18s) * 08:29 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T428823|T428823]] * 08:27 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2176.codfw.wmnet with OS trixie * 08:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Migration to 10.11.17 * 08:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 08:25 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 08:25 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool pc1021: Migration to 10.11.17 * 08:25 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94052) * 08:24 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): Testing upgrade for [[phab:T428823|T428823]] (duration: 01m 17s) * 08:23 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): Testing upgrade for [[phab:T428823|T428823]] * 08:22 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94051) * 08:22 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1219.eqiad.wmnet with OS trixie * 08:17 moritzm: installing PHP 8.2 security updates * 08:15 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 08:14 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 08:11 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 08:11 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 08:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2176.codfw.wmnet with reason: host reimage * 08:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1013.eqiad.wmnet with OS trixie * 08:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5004.eqsin.wmnet to cluster eqsin02 and group 01 * 08:06 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 08:06 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 08:05 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2021.codfw.wmnet,pc1021.eqiad.wmnet with reason: upgrade * 08:05 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1219.eqiad.wmnet with reason: host reimage * 08:05 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5004.eqsin.wmnet to cluster eqsin02 and group 01 * 08:05 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 08:05 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 08:04 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2176.codfw.wmnet with reason: host reimage * 08:04 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 08:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 08:03 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 08:03 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5004.eqsin.wmnet * 07:58 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1219.eqiad.wmnet with reason: host reimage * 07:56 marostegui: install mariadb 10.11.17 on pc1 [[phab:T427345|T427345]] * 07:54 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1013.eqiad.wmnet with reason: host reimage * 07:50 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1013.eqiad.wmnet with reason: host reimage * 07:49 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 07:49 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 07:49 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5004.eqsin.wmnet * 07:47 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 07:47 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 07:46 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2176.codfw.wmnet with OS trixie * 07:43 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1219.eqiad.wmnet with OS trixie * 07:43 moritzm: imported Jenkins 2.541.3 for thirdparty/ci (Bullseye) and thirdparty/jenkins (Bookworm, Trixie) * 07:42 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 07:35 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1013.eqiad.wmnet with OS trixie * 07:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2176: Upgrading db2176.codfw.wmnet * 07:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1219: Upgrading db1219.eqiad.wmnet * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2176: Upgrading db2176.codfw.wmnet * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1219: Upgrading db1219.eqiad.wmnet * 07:31 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:30 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 07:29 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1163: Repooling * 07:19 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 06:51 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS trixie * 06:50 marostegui@cumin1003: dbctl commit (dc=all): 'Repool es2042', diff saved to https://phabricator.wikimedia.org/P94044 and previous config saved to /var/cache/conftool/dbconfig/20260611-065049-marostegui.json * 06:50 marostegui@cumin1003: dbctl commit (dc=all): 'Depool es2042', diff saved to https://phabricator.wikimedia.org/P94043 and previous config saved to /var/cache/conftool/dbconfig/20260611-065027-marostegui.json * 06:44 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1163: Repooling * 06:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1163 [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94041 and previous config saved to /var/cache/conftool/dbconfig/20260611-064319-fceratto.json * 06:42 fceratto@dns1005: END - running authdns-update * 06:40 fceratto@dns1005: START - running authdns-update * 06:33 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:33 fceratto@cumin1003: MariaDB change: Setting sections s1 as read-write for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:33 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:33 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1184 to s1 primary and set section read-write [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94040 and previous config saved to /var/cache/conftool/dbconfig/20260611-063323-fceratto.json * 06:32 fceratto@cumin1003: dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94039 and previous config saved to /var/cache/conftool/dbconfig/20260611-063251-fceratto.json * 06:32 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:32 fceratto@cumin1003: Dbctl change: Setting sections s1 as read-write for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:32 fceratto@cumin1003: MariaDB change: Setting sections s1 as read-write for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:31 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:31 fceratto@cumin1003: dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94037 and previous config saved to /var/cache/conftool/dbconfig/20260611-063100-fceratto.json * 06:30 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:30 fceratto@cumin1003: MariaDB change: Setting sections s1 as read-only for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:30 fceratto@cumin1003: Dbctl change: Setting sections s1 as read-only for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:29 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:29 federico3: Starting s1 eqiad failover from db1163 to db1184 - [[phab:T426083|T426083]] * 06:22 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1184 with weight 0 [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94035 and previous config saved to /var/cache/conftool/dbconfig/20260611-062224-fceratto.json * 06:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 30 hosts with reason: Primary switchover s1 [[phab:T426083|T426083]] * 05:37 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 05:28 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 05:27 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 05:18 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 05:17 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS trixie * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2045: Upgrading es2045.codfw.wmnet * 05:16 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2045: Upgrading es2045.codfw.wmnet * 05:16 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 44s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:23 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2046.* * 01:19 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync * 01:18 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync * 01:18 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1009.eqiad.wmnet with OS trixie * 01:12 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:12 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 01:12 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:12 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:11 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:11 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 01:11 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:10 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:10 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 01:09 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 01:09 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 01:08 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 01:08 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 01:08 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 01:07 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 01:07 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 01:06 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 01:06 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 01:06 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 01:05 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 01:05 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 01:05 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 01:02 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1009.eqiad.wmnet with reason: host reimage * 00:58 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1009.eqiad.wmnet with reason: host reimage * 00:54 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main1009 * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main1009 * 00:41 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main1009 * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main1009.eqiad.wmnet 37.48.64.10.in-addr.arpa 7.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:41 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main1009.eqiad.wmnet 37.48.64.10.in-addr.arpa 7.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1009 - jasmine@cumin2002" * 00:40 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1009 - jasmine@cumin2002" * 00:39 cdanis@cumin1003: dbctl commit (dc=all): 'depool db1262', diff saved to https://phabricator.wikimedia.org/P94032 and previous config saved to /var/cache/conftool/dbconfig/20260611-003950-cdanis.json * 00:36 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 00:34 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5020.* * 00:30 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main1009 * 00:30 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1009.eqiad.wmnet with OS trixie * 00:03 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5024.* == 2026-06-10 == * 23:53 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5024.* * 23:15 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300154{{!}}Disable ShortUrl on bdwikimedia, bhwiki, bnwiki, bnwikisource, eswikibooks, gomwiki (T107188)]] (duration: 11m 37s) * 23:11 krinkle@deploy1003: krinkle: Continuing with deployment * 23:06 krinkle@deploy1003: krinkle: Backport for [[gerrit:1300154{{!}}Disable ShortUrl on bdwikimedia, bhwiki, bnwiki, bnwikisource, eswikibooks, gomwiki (T107188)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:04 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1300154{{!}}Disable ShortUrl on bdwikimedia, bhwiki, bnwiki, bnwikisource, eswikibooks, gomwiki (T107188)]] * 22:57 ladsgroup@dns1004: END - running authdns-update * 22:55 ladsgroup@dns1004: START - running authdns-update * 22:13 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5024.eqsin.wmnet with OS trixie * 22:13 mutante: gerrit - restarting service for logging change * 22:11 dzahn@cumin2002: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 0:10:00 on gerrit.wikimedia.org with reason: service restart * 22:09 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on gerrit2003.wikimedia.org with reason: service restart * 22:06 mutante: gerrit-spare: restarting gerrit * 22:06 mutante: gerrit-replica: restarting gerrit * 21:44 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage * 21:37 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage * 21:22 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300250{{!}}ExecuteTestAndCacheJob: Fix stdClasses serialised wrongly by JobQueue (T428801)]], [[gerrit:1300248{{!}}tests: Fix StandaloneHooksTest ordering, now broken by DB upgrade]] (duration: 08m 23s) * 21:17 jforrester@deploy1003: jforrester: Continuing with deployment * 21:15 jforrester@deploy1003: jforrester: Backport for [[gerrit:1300250{{!}}ExecuteTestAndCacheJob: Fix stdClasses serialised wrongly by JobQueue (T428801)]], [[gerrit:1300248{{!}}tests: Fix StandaloneHooksTest ordering, now broken by DB upgrade]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:13 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1300250{{!}}ExecuteTestAndCacheJob: Fix stdClasses serialised wrongly by JobQueue (T428801)]], [[gerrit:1300248{{!}}tests: Fix StandaloneHooksTest ordering, now broken by DB upgrade]] * 21:03 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5024 * 21:02 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5024 * 21:02 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300247{{!}}Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] (duration: 06m 51s) * 21:00 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp5024 * 21:00 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5024.eqsin.wmnet 35.0.132.10.in-addr.arpa 5.3.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 21:00 brett@cumin2002: START - Cookbook sre.dns.wipe-cache cp5024.eqsin.wmnet 35.0.132.10.in-addr.arpa 5.3.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 21:00 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:00 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5024 - brett@cumin2002" * 20:59 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5024 - brett@cumin2002" * 20:57 catrope@deploy1003: catrope: Continuing with deployment * 20:57 catrope@deploy1003: catrope: Backport for [[gerrit:1300247{{!}}Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:55 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1300247{{!}}Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] * 20:54 brett@cumin2002: START - Cookbook sre.dns.netbox * 20:50 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5024 * 20:49 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5024.eqsin.wmnet with OS trixie * 20:48 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5020.* * 20:44 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300073{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] (duration: 11m 55s) * 20:40 catrope@deploy1003: catrope, gkyziridis: Continuing with deployment * 20:34 catrope@deploy1003: catrope, gkyziridis: Backport for [[gerrit:1300073{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:32 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1300073{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] * 20:30 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5020.eqsin.wmnet with OS trixie * 20:30 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300226{{!}}[arzwiki] Change the wordmark (T427720)]] (duration: 09m 49s) * 20:25 catrope@deploy1003: gergesshamon, catrope: Continuing with deployment * 20:22 catrope@deploy1003: gergesshamon, catrope: Backport for [[gerrit:1300226{{!}}[arzwiki] Change the wordmark (T427720)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1300226{{!}}[arzwiki] Change the wordmark (T427720)]] * 19:59 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5020.eqsin.wmnet with reason: host reimage * 19:53 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5020.eqsin.wmnet with reason: host reimage * 19:30 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 19:27 bblack@cumin1003: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 19:23 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2046.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:19 brett@cumin2002: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2046.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5020 * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5020 * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2044.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:18 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp5020 * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5020.eqsin.wmnet 24.0.132.10.in-addr.arpa 4.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:18 brett@cumin2002: START - Cookbook sre.dns.wipe-cache cp5020.eqsin.wmnet 24.0.132.10.in-addr.arpa 4.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:17 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5020 - brett@cumin2002" * 19:17 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5020 - brett@cumin2002" * 19:14 brett@cumin2002: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2044.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:11 brett@cumin2002: START - Cookbook sre.dns.netbox * 19:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 19:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2174: Migration of db2174.codfw.wmnet completed * 19:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 19:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1218: Migration of db1218.eqiad.wmnet completed * 18:24 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5020 * 18:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5020.eqsin.wmnet with OS trixie * 18:22 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2174: Migration of db2174.codfw.wmnet completed * 18:20 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 18:17 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1218: Migration of db1218.eqiad.wmnet completed * 18:16 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5018.* * 18:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2174.codfw.wmnet with OS trixie * 18:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1218.eqiad.wmnet with OS trixie * 17:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2174.codfw.wmnet with reason: host reimage * 17:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1218.eqiad.wmnet with reason: host reimage * 17:46 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2010.codfw.wmnet with OS trixie * 17:45 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync * 17:44 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync * 17:44 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2174.codfw.wmnet with reason: host reimage * 17:42 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1218.eqiad.wmnet with reason: host reimage * 17:33 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94021) * 17:29 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2010.codfw.wmnet with reason: host reimage * 17:26 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1218.eqiad.wmnet with OS trixie * 17:26 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2174.codfw.wmnet with OS trixie * 17:25 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:24 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 17:24 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1218: Upgrading db1218.eqiad.wmnet * 17:24 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:24 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:24 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1218: Upgrading db1218.eqiad.wmnet * 17:23 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 17:23 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2174: Upgrading db2174.codfw.wmnet * 17:23 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 17:23 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2010.codfw.wmnet with reason: host reimage * 17:23 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:22 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2174: Upgrading db2174.codfw.wmnet * 17:22 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:22 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 17:22 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 17:22 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 17:22 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text and not P<nowiki>{</nowiki>cp7008*<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 17:21 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 17:21 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 17:19 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 17:19 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 17:18 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 17:18 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 17:17 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 17:17 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 17:17 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 17:13 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 17:12 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox) * 17:03 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 17:03 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1206: Migration of db1206.eqiad.wmnet completed * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2010 * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2010 * 17:02 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2010 * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2010.codfw.wmnet 35.48.192.10.in-addr.arpa 5.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:02 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2010.codfw.wmnet 35.48.192.10.in-addr.arpa 5.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2010 - jasmine@cumin2002" * 17:01 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2010 - jasmine@cumin2002" * 16:57 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 16:50 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2010 * 16:50 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2010.codfw.wmnet with OS trixie * 16:41 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 16:39 bblack@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 16:39 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 16:34 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 16:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5018.eqsin.wmnet with OS trixie * 16:22 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 16:20 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 16:17 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1206: Migration of db1206.eqiad.wmnet completed * 16:15 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 16:15 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 16:14 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 16:12 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 16:12 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 16:11 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 16:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1206.eqiad.wmnet with OS trixie * 16:01 blblack: apt: uploaded libvmod-wmfuniq 0.3.0 for trixie * 15:59 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5018.eqsin.wmnet with reason: host reimage * 15:53 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:52 vriley@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:51 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5018.eqsin.wmnet with reason: host reimage * 15:50 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1206.eqiad.wmnet with reason: host reimage * 15:45 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1206.eqiad.wmnet with reason: host reimage * 15:43 sukhe@cumin1003: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool drmrs [reason: no reason specified, no task ID specified] * 15:42 sukhe@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: no reason specified, no task ID specified] * 15:38 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 15:38 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2173: Migration of db2173.codfw.wmnet completed * 15:34 topranks: drain traffic through cr2-drmrs to reset pic0 * 15:33 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94013) * 15:30 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1206.eqiad.wmnet with OS trixie * 15:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1206: Upgrading db1206.eqiad.wmnet * 15:28 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1206: Upgrading db1206.eqiad.wmnet * 15:27 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:25 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:24 vriley@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:24 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1009 * 15:24 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Harroyo-wmf out of all services on: 2436 hosts * 15:23 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1009 * 15:21 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:20 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist translate extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release * 15:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5018 * 15:19 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5018 * 15:18 vriley@cumin1003: START - Cookbook sre.dns.netbox * 15:18 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp5018 * 15:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5018.eqsin.wmnet 18.0.132.10.in-addr.arpa 8.1.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 15:18 brett@cumin2002: START - Cookbook sre.dns.wipe-cache cp5018.eqsin.wmnet 18.0.132.10.in-addr.arpa 8.1.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 15:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:15 brett@cumin2002: START - Cookbook sre.dns.netbox * 15:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 15:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1195: Migration of db1195.eqiad.wmnet completed * 15:12 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:11 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:11 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin1003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:11 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin1003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:08 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300169{{!}}Fix snak value display for rtl languages (T360854)]], [[gerrit:1300168{{!}}Fix snak value display for rtl languages (T360854)]] (duration: 08m 39s) * 15:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 15:01 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1300169{{!}}Fix snak value display for rtl languages (T360854)]], [[gerrit:1300168{{!}}Fix snak value display for rtl languages (T360854)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:59 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:59 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1300169{{!}}Fix snak value display for rtl languages (T360854)]], [[gerrit:1300168{{!}}Fix snak value display for rtl languages (T360854)]] * 14:58 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:55 Lucas_WMDE: lucaswerkmeister-wmde@deploy1003 $ printf 'https://www.mediawiki.org/keys/%s\n' '' 'keys.txt' 'keys.html' {{!}} mwscript-k8s --attach --comment=[[phab:T423267|T423267]] purgeList mediawikiwiki * 14:54 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist translate extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release, now with correct schema * 14:53 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2173: Migration of db2173.codfw.wmnet completed * 14:50 ayounsi@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin2003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:50 ayounsi@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:49 ayounsi@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:48 ayounsi@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:47 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299614{{!}}Add my public key to mediawiki.org/keys (T423267)]] (duration: 08m 33s) * 14:46 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:42 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, matmarex: Continuing with deployment * 14:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2173.codfw.wmnet with OS trixie * 14:40 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, matmarex: Backport for [[gerrit:1299614{{!}}Add my public key to mediawiki.org/keys (T423267)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:40 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:40 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:38 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1299614{{!}}Add my public key to mediawiki.org/keys (T423267)]] * 14:38 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox and (A:dnsbox) * 14:34 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:34 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:33 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:29 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1195: Migration of db1195.eqiad.wmnet completed * 14:28 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:27 vriley@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:26 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 14:26 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 14:24 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist translate extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release, now with dblist translate * 14:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2173.codfw.wmnet with reason: host reimage * 14:23 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 14:22 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 14:22 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply * 14:21 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply * 14:20 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox) * 14:20 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2173.codfw.wmnet with reason: host reimage * 14:20 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:19 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:19 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:18 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:18 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:18 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:18 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1195.eqiad.wmnet with OS trixie * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 14:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 14:15 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 14:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 14:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 14:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 14:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:13 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:13 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 14:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 14:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 14:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 14:09 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 14:08 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:08 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 14:07 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 14:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 14:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 14:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 14:02 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2173.codfw.wmnet with OS trixie * 14:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 14:00 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1195.eqiad.wmnet with reason: host reimage * 14:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 13:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2173: Upgrading db2173.codfw.wmnet * 13:59 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2173: Upgrading db2173.codfw.wmnet * 13:58 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:58 atsuko@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/ttmserver-export.php --wiki=default --ttmserver eqiad-test # [[phab:T425377|T425377]] populating production index on test cluster to estimate time required for the release * 13:56 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1195.eqiad.wmnet with reason: host reimage * 13:54 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Atieno out of all services on: 2436 hosts * 13:42 Lucas_WMDE: UTC afternoon backport+config window done * 13:42 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1195.eqiad.wmnet with OS trixie * 13:36 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297237{{!}}wmf-config: Update private subnets to include additions (T427393)]] (duration: 07m 20s) * 13:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1195: Upgrading db1195.eqiad.wmnet * 13:33 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling restart_daemons on A:hcaptcha-proxy and A:hcaptcha-proxy * 13:33 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum and A:durum * 13:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2170: Migration of db2170.codfw.wmnet completed * 13:33 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1195: Upgrading db1195.eqiad.wmnet * 13:32 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:32 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, brett: Continuing with deployment * 13:32 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough * 13:31 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply * 13:31 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, brett: Backport for [[gerrit:1297237{{!}}wmf-config: Update private subnets to include additions (T427393)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:31 eevans@deploy1003: helmfile [staging] START helmfile.d/services/data-gateway: apply * 13:29 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1297237{{!}}wmf-config: Update private subnets to include additions (T427393)]] * 13:28 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp5018.eqsin.wmnet with reason: host down * 13:28 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy (exit_code=0) rolling restart_daemons on A:tcpproxy and A:tcpproxy * 13:25 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5018.eqsin.wmnet,service=(cdn{{!}}ats-be) * 13:22 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox) * 13:20 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum and A:durum * 13:20 sukhe@cumin1003: START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling restart_daemons on A:hcaptcha-proxy and A:hcaptcha-proxy * 13:19 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299676{{!}}Enable ULS v2 on group0 wikis]] (duration: 17m 00s) * 13:19 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough * 13:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1186: Migration of db1186.eqiad.wmnet completed * 13:18 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-test: apply * 13:18 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-test: apply * 13:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply * 13:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply * 13:15 sbisson@deploy1003: sbisson, abi: Continuing with deployment * 13:10 sukhe@cumin1003: START - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy rolling restart_daemons on A:tcpproxy and A:tcpproxy * 13:05 sbisson@deploy1003: sbisson, abi: Backport for [[gerrit:1299676{{!}}Enable ULS v2 on group0 wikis]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:03 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1014.eqiad.wmnet with OS trixie * 13:02 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299676{{!}}Enable ULS v2 on group0 wikis]] * 12:47 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2170: Migration of db2170.codfw.wmnet completed * 12:46 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5004.eqsin.wmnet with OS bookworm * 12:46 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:46 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:46 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:45 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1014.eqiad.wmnet with reason: host reimage * 12:42 topranks: re-map DSCP AF41 from 'low' to 'normal' priority qos class on network [[phab:T424640|T424640]] * 12:41 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1014.eqiad.wmnet with reason: host reimage * 12:36 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2170.codfw.wmnet with OS trixie * 12:33 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1186: Migration of db1186.eqiad.wmnet completed * 12:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5004.eqsin.wmnet with reason: host reimage * 12:24 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1014 * 12:24 jiji@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host rdb1014 * 12:23 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1186.eqiad.wmnet with OS trixie * 12:21 jiji@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host rdb1014 * 12:21 jiji@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb1014.eqiad.wmnet 42.48.64.10.in-addr.arpa 2.4.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 12:21 jiji@cumin1003: START - Cookbook sre.dns.wipe-cache rdb1014.eqiad.wmnet 42.48.64.10.in-addr.arpa 2.4.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 12:21 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:21 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host rdb1014 - jiji@cumin1003" * 12:21 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host rdb1014 - jiji@cumin1003" * 12:20 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5004.eqsin.wmnet with reason: host reimage * 12:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2170.codfw.wmnet with reason: host reimage * 12:16 jiji@cumin1003: START - Cookbook sre.dns.netbox * 12:13 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1014 * 12:12 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1014.eqiad.wmnet with OS trixie * 12:12 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2170.codfw.wmnet with reason: host reimage * 12:08 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300104{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643{{!}}wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] (duration: 11m 06s) * 12:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1186.eqiad.wmnet with reason: host reimage * 12:03 reedy@deploy1003: reedy: Continuing with deployment * 12:02 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1186.eqiad.wmnet with reason: host reimage * 11:59 reedy@deploy1003: reedy: Backport for [[gerrit:1300104{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643{{!}}wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes c * 11:57 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1300104{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643{{!}}wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] * 11:53 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2170.codfw.wmnet with OS trixie * 11:51 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ganeti5004 * 11:51 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti5004 * 11:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2170: Upgrading db2170.codfw.wmnet * 11:49 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2170: Upgrading db2170.codfw.wmnet * 11:49 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti5004 * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti5004.eqsin.wmnet 40.0.132.10.in-addr.arpa 0.4.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 11:49 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache ganeti5004.eqsin.wmnet 40.0.132.10.in-addr.arpa 0.4.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5004 - jmm@cumin2002" * 11:49 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5004 - jmm@cumin2002" * 11:49 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:48 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1186.eqiad.wmnet with OS trixie * 11:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1186: Upgrading db1186.eqiad.wmnet * 11:45 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1186: Upgrading db1186.eqiad.wmnet * 11:45 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:38 jmm@cumin2002: START - Cookbook sre.dns.netbox * 11:35 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 11:34 jmm@cumin2002: START - Cookbook sre.hosts.move-vlan for host ganeti5004 * 11:34 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 11:34 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5004.eqsin.wmnet with OS bookworm * 11:34 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 11:33 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1151: Security updates * 11:33 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 11:33 root@cumin1003: START - Cookbook sre.mysql.parsercache * 11:33 root@cumin1003: START - Cookbook sre.mysql.pool pool db1151: Security updates * 11:31 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply * 11:30 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply * 11:30 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply * 11:30 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:23 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 11:23 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:23 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 11:23 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 11:16 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 11:15 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 11:15 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 11:15 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:09 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1151: Security updates * 11:09 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 11:09 root@cumin1003: START - Cookbook sre.mysql.parsercache * 11:09 root@cumin1003: START - Cookbook sre.mysql.depool depool db1151: Security updates * 11:08 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300092{{!}}ProductionServices: re-add poolcounter2006 (T426736)]] (duration: 06m 55s) * 11:04 blake@deploy1003: blake: Continuing with deployment * 11:04 blake@deploy1003: blake: Backport for [[gerrit:1300092{{!}}ProductionServices: re-add poolcounter2006 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:03 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:01 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300092{{!}}ProductionServices: re-add poolcounter2006 (T426736)]] * 10:59 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet * 10:57 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 10:57 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 10:57 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 10:56 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 10:56 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply * 10:56 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply * 10:56 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet * 10:56 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300087{{!}}ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] (duration: 06m 42s) * 10:51 blake@deploy1003: blake: Continuing with deployment * 10:51 moritzm: remove ganeti5004 from eqsin cluster for reimage [[phab:T428229|T428229]] * 10:51 blake@deploy1003: blake: Backport for [[gerrit:1300087{{!}}ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:49 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300087{{!}}ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] * 10:47 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet * 10:47 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 10:46 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 10:46 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 10:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 10:43 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet * 10:43 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300082{{!}}ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] (duration: 07m 38s) * 10:41 moritzm: installing nginx security updates * 10:38 blake@deploy1003: blake: Continuing with deployment * 10:38 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1152: Security updates * 10:38 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:38 root@cumin1003: START - Cookbook sre.mysql.parsercache * 10:38 root@cumin1003: START - Cookbook sre.mysql.pool pool db1152: Security updates * 10:38 blake@deploy1003: blake: Backport for [[gerrit:1300082{{!}}ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:37 moritzm: failover Ganeti master in eqsin to ganeti5007 [[phab:T428229|T428229]] * 10:35 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300082{{!}}ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] * 10:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 10:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 10:33 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet * 10:29 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet * 10:29 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300072{{!}}ProductionServices: reboot poolcounter1007 (T426736)]] (duration: 07m 45s) * 10:27 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 10:27 jmm@cumin2002: DONE (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for sretest2009.codfw.wmnet: Renew puppet certificate - jmm@cumin2002 * 10:24 blake@deploy1003: blake: Continuing with deployment * 10:23 blake@deploy1003: blake: Backport for [[gerrit:1300072{{!}}ProductionServices: reboot poolcounter1007 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:22 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 10:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 10:21 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:21 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300072{{!}}ProductionServices: reboot poolcounter1007 (T426736)]] * 10:21 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:21 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:21 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:21 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:20 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet * 10:14 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1152: Security updates * 10:14 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:14 root@cumin1003: START - Cookbook sre.mysql.parsercache * 10:14 root@cumin1003: START - Cookbook sre.mysql.depool depool db1152: Security updates * 10:13 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet * 10:12 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300064{{!}}ProductionServices: reboot poolcounter1006.eqiad (T426736)]] (duration: 07m 46s) * 10:07 blake@deploy1003: blake: Continuing with deployment * 10:06 blake@deploy1003: blake: Backport for [[gerrit:1300064{{!}}ProductionServices: reboot poolcounter1006.eqiad (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:04 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300064{{!}}ProductionServices: reboot poolcounter1006.eqiad (T426736)]] * 09:57 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300058{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] (duration: 09m 32s) * 09:52 kharlan@deploy1003: kharlan: Continuing with deployment * 09:49 kharlan@deploy1003: kharlan: Backport for [[gerrit:1300058{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:47 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1300058{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] * 09:35 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox * 09:34 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 09:32 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 09:32 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 09:26 moritzm: upgrade routinator in eqiad to 0.15.2 [[phab:T428456|T428456]] * 09:23 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 09:23 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 09:22 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 09:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus5003.eqsin.wmnet to plain * 09:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus5003.eqsin.wmnet to plain * 09:15 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:04 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:54 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:20 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:09 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:09 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:07 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 08:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:04 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:01 fceratto@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host db1215.eqiad.wmnet with OS trixie * 07:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:53 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:48 javiermonton@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply * 07:48 javiermonton@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply * 07:44 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1215.eqiad.wmnet with reason: host reimage * 07:41 javiermonton@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply * 07:40 javiermonton@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply * 07:40 moritzm: installing openssl security updates * 07:39 fceratto@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1215.eqiad.wmnet with reason: host reimage * 07:38 javiermonton@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply * 07:37 javiermonton@deploy1003: helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply * 07:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:29 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299556{{!}}ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561{{!}}ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529{{!}}translate: adding separate read/write endpoints (T425377)]] (duration: 14m 03s) * 07:25 atsuko@deploy1003: atsuko: Continuing with deployment * 07:23 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db1215.eqiad.wmnet with OS trixie * 07:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1215.eqiad.wmnet with reason: Reimage * 07:21 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:20 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:20 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:17 atsuko@deploy1003: atsuko: Backport for [[gerrit:1299556{{!}}ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561{{!}}ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529{{!}}translate: adding separate read/write endpoints (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be veri * 07:16 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:15 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1299556{{!}}ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561{{!}}ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529{{!}}translate: adding separate read/write endpoints (T425377)]] * 07:14 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:12 atsukoito: backporting extensions/Translate to wmf/1.47.0-wmf.5 and applying the config * 07:12 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 06:45 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 05:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 05:43 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 05:42 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 05:41 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 47s) * 02:07 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1008.eqiad.wmnet with OS trixie * 02:03 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync * 02:02 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:52 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:51 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 01:51 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:50 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:50 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:49 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1008.eqiad.wmnet with reason: host reimage * 01:49 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 01:49 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:49 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:49 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 01:48 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 01:48 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 01:47 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 01:47 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 01:46 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 01:46 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 01:44 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 01:44 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 01:43 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 01:43 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1008.eqiad.wmnet with reason: host reimage * 01:25 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main1008 * 01:24 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main1008 * 01:24 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main1008 * 01:24 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main1008.eqiad.wmnet 45.32.64.10.in-addr.arpa 5.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 01:23 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main1008.eqiad.wmnet 45.32.64.10.in-addr.arpa 5.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 01:23 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 01:23 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1008 - jasmine@cumin2002" * 01:23 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1008 - jasmine@cumin2002" * 01:19 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 01:12 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main1008 * 01:11 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1008.eqiad.wmnet with OS trixie * 01:00 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2009.codfw.wmnet with OS trixie * 00:54 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync * 00:53 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync * 00:43 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2009.codfw.wmnet with reason: host reimage * 00:40 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:38 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2009.codfw.wmnet with reason: host reimage * 00:38 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 00:38 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:37 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:37 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 00:36 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 00:36 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 00:34 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 00:34 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 00:33 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 00:33 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 00:32 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 00:32 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 00:32 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2009 * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2009 * 00:15 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2009 * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2009.codfw.wmnet 33.48.192.10.in-addr.arpa 3.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:15 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2009.codfw.wmnet 33.48.192.10.in-addr.arpa 3.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2009 - jasmine@cumin2002" * 00:15 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2009 - jasmine@cumin2002" * 00:10 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 00:03 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2009 * 00:03 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2009.codfw.wmnet with OS trixie == 2026-06-09 == * 22:50 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299640{{!}}HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] (duration: 08m 59s) * 22:45 cscott@deploy1003: cscott: Continuing with deployment * 22:43 cscott@deploy1003: cscott: Backport for [[gerrit:1299640{{!}}HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:41 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299640{{!}}HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] * 22:15 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299639{{!}}[Bug] Donor Badge: Remove client prefs for control group (T428501)]] (duration: 20m 57s) * 22:11 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 22:07 mutante: gerrit - apache httpd log file location moved to /srv/gerrit/site_path/review_site/logs/ [[phab:T425667|T425667]] * 22:06 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on gerrit2003.wikimedia.org with reason: debug * 21:56 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1299639{{!}}[Bug] Donor Badge: Remove client prefs for control group (T428501)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:54 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299639{{!}}[Bug] Donor Badge: Remove client prefs for control group (T428501)]] * 21:52 ryankemper: [[phab:T428241|T428241]] removed retired wdqs2009 full-graph journal dump (446G x2, ~892G) from clouddumps100[1-2]:/srv/dumps/xmldatadumps/public/other/wdqs * 21:49 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299602{{!}}Revert "Create VectorComponentPageToolbar component" (T428649)]] (duration: 08m 16s) * 21:48 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 21:45 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 21:43 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1299602{{!}}Revert "Create VectorComponentPageToolbar component" (T428649)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:41 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299602{{!}}Revert "Create VectorComponentPageToolbar component" (T428649)]] * 21:34 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gerrit1003.wikimedia.org with reason: debug * 21:27 maryum: Deployed security fix for [[phab:T428324|T428324]] * 21:24 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 21:15 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart * 21:06 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart * 20:50 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs2002.codfw.wmnet with OS trixie * 20:50 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299588{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] (duration: 11m 13s) * 20:46 cscott@deploy1003: cscott: Continuing with deployment * 20:43 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs2002.codfw.wmnet with OS trixie * 20:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:42 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:41 cscott@deploy1003: cscott: Backport for [[gerrit:1299588{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:39 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299588{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] * 20:38 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:38 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:33 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:33 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:32 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299454{{!}}wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] (duration: 22m 08s) * 20:28 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:28 cscott@deploy1003: cscott, gkyziridis: Continuing with deployment * 20:24 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2004 * 20:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2004 * 20:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2003 * 20:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2003 * 20:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2002 * 20:13 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2002 * 20:13 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2001 * 20:13 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2001 * 20:12 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:12 cscott@deploy1003: cscott, gkyziridis: Backport for [[gerrit:1299454{{!}}wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:10 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299454{{!}}wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] * 20:09 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:04 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:59 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:54 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:53 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:48 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:47 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:47 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:46 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:46 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:45 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:45 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:28 ryankemper@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts wdqs1015.eqiad.wmnet * 19:28 ryankemper@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:28 ryankemper@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002" * 19:27 ryankemper@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002" * 19:20 ryankemper@cumin2002: START - Cookbook sre.dns.netbox * 19:15 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2008.codfw.wmnet with OS trixie * 19:15 ryankemper@cumin2002: START - Cookbook sre.hosts.decommission for hosts wdqs1015.eqiad.wmnet * 19:12 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync * 19:12 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync * 19:00 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:58 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 18:58 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2008.codfw.wmnet with reason: host reimage * 18:58 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 18:58 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 18:57 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 18:57 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 18:56 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 18:56 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 18:55 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 18:55 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 18:55 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 18:54 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 18:54 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:54 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 18:53 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 18:53 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 18:53 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 18:52 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:52 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2003 to codfw - jhancock@cumin2002" * 18:52 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2003 to codfw - jhancock@cumin2002" * 18:52 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 18:52 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 18:51 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2008.codfw.wmnet with reason: host reimage * 18:51 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 18:51 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 18:51 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 18:50 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 18:50 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 18:47 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:47 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:47 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:46 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:46 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:42 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:42 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:31 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 18:29 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2008.codfw.wmnet with OS trixie * 18:26 jasmine@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2008.codfw.wmnet with OS trixie * 17:48 mutante: https://releases.wikimedia.org {{!}} https://releases-jenkins.wikimedia.org - down for maintenance [[phab:T418299|T418299]] * 17:48 cmooney@dns2005: END - running authdns-update * 17:47 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet with reason: reimage * 17:47 cmooney@dns2005: START - running authdns-update * 17:46 sukhe: sudo cumin 'A:hcaptcha-proxy' 'run-puppet-agent': rolling out CR {{Gerrit|1299427}} [[phab:T428539|T428539]] * 17:43 jayme: kafka-main2008 is down due to hardware failure [[phab:T428654|T428654]] * 17:32 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf1002.eqiad.wmnet with OS trixie * 17:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf1002.eqiad.wmnet with reason: host reimage * 17:06 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf1002.eqiad.wmnet with reason: host reimage * 17:05 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2008 * 17:05 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2008 * 17:04 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 17:04 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2008 * 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2008.codfw.wmnet 4.32.192.10.in-addr.arpa 4.0.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:04 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 17:04 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2008.codfw.wmnet 4.32.192.10.in-addr.arpa 4.0.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2008 - jasmine@cumin2002" * 17:04 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5018 * 17:04 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2008 - jasmine@cumin2002" * 17:03 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5018.eqsin.wmnet with OS trixie * 16:58 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 16:58 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 16:57 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 16:57 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 16:57 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 16:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 16:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 16:50 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf1002.eqiad.wmnet with OS trixie * 16:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 16:47 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf1001.eqiad.wmnet with OS trixie * 16:47 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply * 16:47 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply * 16:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 16:41 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 16:41 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 16:35 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2008 * 16:34 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2008.codfw.wmnet with OS trixie * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:31 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply * 16:30 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply * 16:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf1001.eqiad.wmnet with reason: host reimage * 16:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:26 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf1001.eqiad.wmnet with reason: host reimage * 16:23 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply * 16:22 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: apply * 16:20 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:16 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:15 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:13 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:13 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:12 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf1001.eqiad.wmnet with OS trixie * 16:10 jiji@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. * 16:09 jiji@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. * 16:07 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf2002.codfw.wmnet with OS trixie * 16:02 jiji@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 16:02 jiji@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 16:00 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. * 15:59 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 15:59 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. * 15:59 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 15:59 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. * 15:59 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 15:58 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] DONE helmfile.d/services/termbox: apply * 15:58 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] START helmfile.d/services/termbox: apply * 15:57 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 15:57 jiji@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 15:57 lucaswerkmeister-wmde@deploy1003: helmfile [staging] DONE helmfile.d/services/termbox: apply * 15:56 lucaswerkmeister-wmde@deploy1003: helmfile [staging] START helmfile.d/services/termbox: apply * 15:54 jiji@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 15:53 jiji@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 15:51 jiji@deploy1003: Finished scap sync-world: redeploy {{Gerrit|1299468}} (duration: 07m 23s) * 15:49 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf2002.codfw.wmnet with reason: host reimage * 15:47 jiji@deploy1003: jiji: Continuing with deployment * 15:46 jiji@deploy1003: jiji: redeploy {{Gerrit|1299468}} synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:46 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf2002.codfw.wmnet with reason: host reimage * 15:45 jiji@deploy1003: Started scap sync-world: redeploy {{Gerrit|1299468}} * 15:43 brouberol@cumin1003: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-eqiad * 15:34 brennen@deploy1003: Finished deploy [phabricator/deployment@73e57ce]: deploy phab1004 for [[phab:T410849|T410849]] (followup for robots.txt) (duration: 00m 40s) * 15:33 brennen@deploy1003: Started deploy [phabricator/deployment@73e57ce]: deploy phab1004 for [[phab:T410849|T410849]] (followup for robots.txt) * 15:33 brennen@deploy1003: Finished deploy [phabricator/deployment@73e57ce]: deploy phab2002 for [[phab:T410849|T410849]] (followup for robots.txt) (duration: 00m 45s) * 15:32 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299468{{!}}ProductionServices.php: switch filebackend.php to rdb2015:6381 #2 (T418918 T291916)]] (duration: 07m 21s) * 15:32 brennen@deploy1003: Started deploy [phabricator/deployment@73e57ce]: deploy phab2002 for [[phab:T410849|T410849]] (followup for robots.txt) * 15:28 jiji@deploy1003: Rolling back deployment * 15:27 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf2002.codfw.wmnet with OS trixie * 15:27 jiji@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 15:26 jiji@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 15:25 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1299468{{!}}ProductionServices.php: switch filebackend.php to rdb2015:6381 #2 (T418918 T291916)]] * 15:22 urbanecm: Remove `migrateMentorStatusAwayToCommunityConfiguration` from updatelog on all wikis ([[phab:T409170|T409170]]; the script was only ever run as a dry-run) * 15:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 15:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 15:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf2001.codfw.wmnet with OS trixie * 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@d244a3e]: deploy phab1004 for [[phab:T410849|T410849]] (duration: 00m 42s) * 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@d244a3e]: deploy phab1004 for [[phab:T410849|T410849]] * 15:02 brennen@deploy1003: Finished deploy [phabricator/deployment@d244a3e]: deploy phab2002 for [[phab:T410849|T410849]] (duration: 00m 45s) * 15:01 brennen@deploy1003: Started deploy [phabricator/deployment@d244a3e]: deploy phab2002 for [[phab:T410849|T410849]] * 14:58 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf2001.codfw.wmnet with reason: host reimage * 14:52 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf2001.codfw.wmnet with reason: host reimage * 14:52 arnaudb@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phab[2002-2003].codfw.wmnet,phab[1004-1006].eqiad.wmnet with reason: [[phab:T410849|T410849]] * 14:47 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 14:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 14:40 moritzm: upgrade routinator in codfw to 0.15.2 [[phab:T428456|T428456]] * 14:35 brouberol@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad * 14:33 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf2001.codfw.wmnet with OS trixie * 14:26 brouberol@cumin1003: END (ERROR) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=97) rolling reboot on A:cephosd-eqiad * 14:26 brouberol@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad * 14:20 btullis@cumin1003: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-codfw * 14:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host parsoidtest1001.eqiad.wmnet * 14:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2153: Migration of db2153.codfw.wmnet completed * 14:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of rpki2003.codfw.wmnet to drbd * 14:14 moritzm: imported routinator 0.15.2-1bookworm to thirdparty/routinator for bookworm-wikimedia [[phab:T428456|T428456]] * 14:12 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:12 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1184: Migration of db1184.eqiad.wmnet completed * 14:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host parsoidtest1001.eqiad.wmnet * 14:07 Dreamy_Jazz: Afternoon UTC backport window done * 14:07 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 14:06 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299495{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502{{!}}SecurePollLogPager: Cast user IDs to ints before use (T428599)]] (duration: 06m 53s) * 14:06 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 14:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: rack depool * 14:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of rpki2003.codfw.wmnet to drbd * 14:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow2004.codfw.wmnet to drbd * 14:02 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:02 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1299495{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502{{!}}SecurePollLogPager: Cast user IDs to ints before use (T428599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:59 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1299495{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502{{!}}SecurePollLogPager: Cast user IDs to ints before use (T428599)]] * 13:58 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:58 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:56 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 13:56 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 13:55 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:55 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * {{safesubst:SAL entry|1=13:55 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298929{{!}}Simplify fragment processing (T423700)]], [[gerrit:1298926{{!}}Move ::getFragmentsToTransform() to Content<nowiki>{</nowiki>Text,DOM<nowiki>}</nowiki>TransformStage]], [[gerrit:1298927{{!}}OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925{{!}}Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497}} * 13:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:51 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow2004.codfw.wmnet to drbd * 13:50 cscott@deploy1003: cscott: Continuing with deployment * 13:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2045.codfw.wmnet to cluster codfw and group A * 13:48 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2045.codfw.wmnet to cluster codfw and group A * 13:48 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2027.codfw.wmnet to cluster codfw and group A * 13:47 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2027.codfw.wmnet to cluster codfw and group A * 13:46 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 13:45 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 13:44 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * {{safesubst:SAL entry|1=13:42 cscott@deploy1003: cscott: Backport for [[gerrit:1298929{{!}}Simplify fragment processing (T423700)]], [[gerrit:1298926{{!}}Move ::getFragmentsToTransform() to Content<nowiki>{</nowiki>Text,DOM<nowiki>}</nowiki>TransformStage]], [[gerrit:1298927{{!}}OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925{{!}}Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497{{!}}Store indicators}} * 13:41 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * {{safesubst:SAL entry|1=13:40 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1298929{{!}}Simplify fragment processing (T423700)]], [[gerrit:1298926{{!}}Move ::getFragmentsToTransform() to Content<nowiki>{</nowiki>Text,DOM<nowiki>}</nowiki>TransformStage]], [[gerrit:1298927{{!}}OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925{{!}}Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497{{!}}}} * 13:40 btullis@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-codfw * 13:39 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 13:37 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 13:35 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 13:33 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 13:32 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 13:32 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298834{{!}}config: Disable EmailConfirmationBanner on all wikis (T428291)]] (duration: 07m 01s) * 13:30 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2153: Migration of db2153.codfw.wmnet completed * 13:28 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 lucaswerkmeister-wmde@deploy1003: mmartorana, lucaswerkmeister-wmde: Continuing with deployment * 13:27 lucaswerkmeister-wmde@deploy1003: mmartorana, lucaswerkmeister-wmde: Backport for [[gerrit:1298834{{!}}config: Disable EmailConfirmationBanner on all wikis (T428291)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:26 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1184: Migration of db1184.eqiad.wmnet completed * 13:25 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298834{{!}}config: Disable EmailConfirmationBanner on all wikis (T428291)]] * 13:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 13:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 13:23 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:21 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 13:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2153.codfw.wmnet with OS trixie * 13:20 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2241: rack depool * 13:20 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1237: repool after maintenance db1237 * 13:19 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298654{{!}}Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] (duration: 09m 40s) * 13:17 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2006.codfw.wmnet * 13:17 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2006.codfw.wmnet * 13:16 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2251-2253].codfw.wmnet * 13:16 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2251-2253].codfw.wmnet * 13:16 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2005.codfw.wmnet * 13:16 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2005.codfw.wmnet * 13:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1184.eqiad.wmnet with OS trixie * 13:14 lucaswerkmeister-wmde@deploy1003: neriah, lucaswerkmeister-wmde: Continuing with deployment * 13:11 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 * 13:11 lucaswerkmeister-wmde@deploy1003: neriah, lucaswerkmeister-wmde: Backport for [[gerrit:1298654{{!}}Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:09 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298654{{!}}Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] * 13:04 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:04 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:04 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2153.codfw.wmnet with reason: host reimage * 13:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:03 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1015.eqiad.wmnet with OS trixie * 12:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage * 12:58 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2153.codfw.wmnet with reason: host reimage * 12:57 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1016.eqiad.wmnet with OS trixie * 12:57 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 * 12:56 XioNoX: lsw1-a4-codfw> request system reboot - [[phab:T427357|T427357]] * 12:55 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:53 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage * 12:50 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299477{{!}}hCaptcha: Roll out to all wikis for api account creation. (T426050)]] (duration: 07m 21s) * 12:46 kharlan@deploy1003: kharlan, dbrant: Continuing with deployment * 12:46 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 * 12:45 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1015.eqiad.wmnet with reason: host reimage * 12:45 kharlan@deploy1003: kharlan, dbrant: Backport for [[gerrit:1299477{{!}}hCaptcha: Roll out to all wikis for api account creation. (T426050)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:45 topranks: shut sub-interfaces for row A/B legacy vlans on cr1-codfw [[phab:T427357|T427357]] * 12:45 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 * 12:43 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1299477{{!}}hCaptcha: Roll out to all wikis for api account creation. (T426050)]] * 12:42 topranks: increase OSPF cost on ssw1-a1-codfw link to lsw1-a4-codfw to force traffic via alternate spine [[phab:T427357|T427357]] * 12:41 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299478{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]] (duration: 07m 02s) * 12:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1016.eqiad.wmnet with reason: host reimage * 12:40 moritzm: installing wireshark security updates * 12:40 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2153.codfw.wmnet with OS trixie * 12:38 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1184.eqiad.wmnet with OS trixie * 12:37 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 12:36 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1299478{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:34 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2153: Upgrading db2153.codfw.wmnet * 12:34 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1237: repool after maintenance db1237 * 12:34 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1299478{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]] * 12:34 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2153: Upgrading db2153.codfw.wmnet * 12:34 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1184: Upgrading db1184.eqiad.wmnet * 12:33 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1184: Upgrading db1184.eqiad.wmnet * 12:33 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:32 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1237.eqiad.wmnet with OS trixie * 12:32 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1015.eqiad.wmnet with reason: host reimage * 12:32 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1016.eqiad.wmnet with reason: host reimage * 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 12:27 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2005.codfw.wmnet * 12:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2046: repool after maintenance * 12:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2006.codfw.wmnet * 12:23 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298829{{!}}wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] (duration: 16m 04s) * 12:23 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2006.codfw.wmnet * 12:22 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2251-2253].codfw.wmnet * 12:22 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2005.codfw.wmnet * 12:20 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2251-2253].codfw.wmnet * 12:20 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 12:20 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: rack depool * 12:20 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 12:20 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2241: rack depool * 12:19 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1016 * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1016 * 12:19 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1015 * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1015 * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1016.eqiad.wmnet with OS trixie * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1015.eqiad.wmnet with OS trixie * 12:17 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 * 12:17 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 24 hosts with reason: Rack A4 depool * 12:16 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Continuing with deployment * 12:15 topranks: drain traffic on ssw1-a1-codfw - add gshut community in evpn underlay - [[phab:T427357|T427357]] * 12:14 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 * 12:13 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Backport for [[gerrit:1298829{{!}}wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:10 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1237.eqiad.wmnet with reason: host reimage * 12:07 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1298829{{!}}wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] * 12:05 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1237.eqiad.wmnet with reason: host reimage * 12:00 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Dmaza out of all services on: 2435 hosts * 11:51 atsuko@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 11:51 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie * 11:49 atsuko@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 11:48 atsuko@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 11:47 atsuko@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 11:45 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 11:44 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 11:43 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:43 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2046: repool after maintenance * 11:38 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 11:36 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:36 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2046.codfw.wmnet with OS trixie * 11:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2185.codfw.wmnet with reason: Reimage * 11:31 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging HMonroy out of all services on: 2435 hosts * 11:28 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging KSiebert out of all services on: 2435 hosts * 11:26 slyngs: CAS-SSO upgrade to version 7.3.7.2 * 11:26 slyngshede@dns1004: END - running authdns-update * 11:24 slyngshede@dns1004: START - running authdns-update * 11:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2046.codfw.wmnet with reason: host reimage * 11:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1043: repool after upgrade * 11:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2046.codfw.wmnet with reason: host reimage * 10:55 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2046.codfw.wmnet with OS trixie * 10:53 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2046: Upgrading es2046.codfw.wmnet * 10:53 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2046: Upgrading es2046.codfw.wmnet * 10:52 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply * 10:52 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:52 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply * 10:52 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply * 10:52 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:52 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply * 10:51 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:32 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1043: repool after upgrade * 10:31 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:28 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1160: Repooling * 10:26 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1043.eqiad.wmnet with OS trixie * 10:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:17 elukey: complete rollout of apache2 upgrades * 10:16 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:15 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:13 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:12 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:12 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:08 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1043.eqiad.wmnet with reason: host reimage * 10:04 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:04 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1043.eqiad.wmnet with reason: host reimage * 10:04 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:04 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:04 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:04 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:04 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:57 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1160: Repooling * 09:51 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 09:51 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 09:50 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 09:50 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 09:49 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1043.eqiad.wmnet with OS trixie * 09:48 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool es1043: Upgrading es1043.eqiad.wmnet * 09:48 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 09:47 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:45 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 09:41 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 09:36 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=5 --verbose --last-checked="20260603"` (after stopping previous scan run) * 09:34 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=5 --verbose` (after stopping previous scan run) * 09:27 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 09:26 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 09:17 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 09:17 fceratto@cumin1003: MariaDB change: Setting sections s5 as read-write * 09:17 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 09:14 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1043: Upgrading es1043.eqiad.wmnet * 09:14 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:12 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1042 to es4 eqiad primary [[phab:T428386|T428386]]', diff saved to https://phabricator.wikimedia.org/P93943 and previous config saved to /var/cache/conftool/dbconfig/20260609-091215-marostegui.json * 09:11 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1043 to es4 eqiad primary [[phab:T428386|T428386]]', diff saved to https://phabricator.wikimedia.org/P93942 and previous config saved to /var/cache/conftool/dbconfig/20260609-091147-marostegui.json * 09:03 jiji@cumin1003: conftool action : set/pooled=yes; selector: service=docker-registry,name=registry2005.codfw.wmnet * 08:59 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:59 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:57 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1237.eqiad.wmnet with OS trixie * 08:55 jiji@cumin1003: conftool action : set/pooled=no; selector: service=docker-registry,name=registry2005.codfw.wmnet * 08:55 jiji@cumin1003: conftool action : set/pooled=yes; selector: service=docker-registry,name=registry2004.codfw.wmnet * 08:50 jiji@cumin1003: conftool action : set/pooled=no; selector: service=docker-registry,name=registry2004.codfw.wmnet * 08:22 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=docker-registry,name=codfw * 08:22 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=docker-registry,name=eqiad * 08:08 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=docker-registry,name=eqiad * 08:08 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=docker-registry,name=codfw * 07:59 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:59 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix typoes - ayounsi@cumin1003" * 07:59 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix typoes - ayounsi@cumin1003" * 07:52 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 07:47 brouberol@dns1004: END - running authdns-update * 07:46 brouberol@dns1004: START - running authdns-update * 07:44 brouberol@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/kafka-ui: apply * 07:43 brouberol@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/kafka-ui: apply * 07:43 brouberol@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/kafka-ui: apply * 07:42 brouberol@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/kafka-ui: apply * 07:41 brouberol@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/kafka-ui: apply * 07:39 brouberol@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/kafka-ui: apply * 07:38 brouberol@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:37 brouberol@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 07:37 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie * 07:36 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.major-upgrade (exit_code=97) * 07:36 brouberol@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:36 brouberol@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:26 fceratto@dns1004: END - running authdns-update * 07:24 fceratto@dns1004: START - running authdns-update * 07:22 marostegui@dns1004: END - running authdns-update * 07:21 marostegui@dns1004: START - running authdns-update * 07:19 elukey@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:19 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Fix dse-k8s-wdqs2002 duplicate ipv6 address - elukey@cumin1003" * 07:19 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Fix dse-k8s-wdqs2002 duplicate ipv6 address - elukey@cumin1003" * 07:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1160.eqiad.wmnet with reason: Maintenance * 07:12 elukey@cumin1003: START - Cookbook sre.dns.netbox * 07:11 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1160: Repooling * 07:11 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1160: Repooling * 07:11 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1160: Repooling * 07:11 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1160: Repooling * 07:00 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:00 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1237.eqiad.wmnet with OS trixie * 06:24 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1160 [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93940 and previous config saved to /var/cache/conftool/dbconfig/20260609-062412-fceratto.json * 06:17 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 06:16 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 06:16 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 06:16 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 06:15 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 06:15 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 06:15 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 06:14 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 06:12 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1244 to s4 primary and set section read-write [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93939 and previous config saved to /var/cache/conftool/dbconfig/20260609-061222-fceratto.json * 06:11 fceratto@cumin1003: dbctl commit (dc=all): 'Set s4 eqiad as read-only for maintenance - [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93938 and previous config saved to /var/cache/conftool/dbconfig/20260609-061131-fceratto.json * 06:10 federico3: Starting s4 eqiad failover from db1160 to db1244 - [[phab:T426086|T426086]] * 06:01 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1244 with weight 0 [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93937 and previous config saved to /var/cache/conftool/dbconfig/20260609-060121-fceratto.json * 06:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 40 hosts with reason: Primary switchover s4 [[phab:T426086|T426086]] * 05:40 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie * 05:37 marostegui@dns1004: START - running authdns-update * 05:27 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1237: Upgrading db1237.eqiad.wmnet * 05:27 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1237: Upgrading db1237.eqiad.wmnet * 05:27 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:24 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1237 [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93935 and previous config saved to /var/cache/conftool/dbconfig/20260609-052420-marostegui.json * 05:23 marostegui@dns1004: START - running authdns-update * 05:23 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93934 and previous config saved to /var/cache/conftool/dbconfig/20260609-052311-marostegui.json * 05:22 marostegui@cumin1003: dbctl commit (dc=all): 'Set x1 eqiad as read-only for maintenance - [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93933 and previous config saved to /var/cache/conftool/dbconfig/20260609-052253-marostegui.json * 05:22 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T428158|T428158]] * 05:19 marostegui@cumin1003: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93932 and previous config saved to /var/cache/conftool/dbconfig/20260609-051859-marostegui.json * 05:18 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: Primary switchover x1 [[phab:T428158|T428158]] * 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.3 (duration: 02m 43s) * 03:40 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] (duration: 37m 16s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 38s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-06-08 == * 22:00 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298915{{!}}CommonSettings: Set $wgScoreSafeMode = false (T428484)]] (duration: 07m 42s) * 21:56 reedy@deploy1003: reedy: Continuing with deployment * 21:54 reedy@deploy1003: reedy: Backport for [[gerrit:1298915{{!}}CommonSettings: Set $wgScoreSafeMode = false (T428484)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:53 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1298915{{!}}CommonSettings: Set $wgScoreSafeMode = false (T428484)]] * 21:12 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298891{{!}}OOUIHTMLForm: Avoid treating form header as a clickable label (T428359)]] (duration: 08m 10s) * 21:07 mlitn@deploy1003: mlitn, neriah: Continuing with deployment * 21:05 mlitn@deploy1003: mlitn, neriah: Backport for [[gerrit:1298891{{!}}OOUIHTMLForm: Avoid treating form header as a clickable label (T428359)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:03 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1298891{{!}}OOUIHTMLForm: Avoid treating form header as a clickable label (T428359)]] * 20:43 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297162{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias]], [[gerrit:1298841{{!}}Squashed diff to master]] (duration: 07m 05s) * 20:39 mlitn@deploy1003: mlitn: Continuing with deployment * 20:38 mlitn@deploy1003: mlitn: Backport for [[gerrit:1297162{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias]], [[gerrit:1298841{{!}}Squashed diff to master]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:36 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1297162{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias]], [[gerrit:1298841{{!}}Squashed diff to master]] * 20:29 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298390{{!}}English Wikibooks: update FlaggedRevs configuration (T428329)]], [[gerrit:1298328{{!}}English Wikiversity: Add new user group "autopatrolled" (T428269)]] (duration: 08m 58s) * 20:25 mlitn@deploy1003: mlitn, vadymts1: Continuing with deployment * 20:22 mlitn@deploy1003: mlitn, vadymts1: Backport for [[gerrit:1298390{{!}}English Wikibooks: update FlaggedRevs configuration (T428329)]], [[gerrit:1298328{{!}}English Wikiversity: Add new user group "autopatrolled" (T428269)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1298390{{!}}English Wikibooks: update FlaggedRevs configuration (T428329)]], [[gerrit:1298328{{!}}English Wikiversity: Add new user group "autopatrolled" (T428269)]] * 20:03 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298879{{!}}SimpleCaptcha: Re-render captcha when edit form is redisplayed (T428437)]] (duration: 37m 43s) * 19:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:31 kharlan@deploy1003: kharlan: Continuing with deployment * 19:30 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:30 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:29 kharlan@deploy1003: kharlan: Backport for [[gerrit:1298879{{!}}SimpleCaptcha: Re-render captcha when edit form is redisplayed (T428437)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 19:28 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:27 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:25 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1298879{{!}}SimpleCaptcha: Re-render captcha when edit form is redisplayed (T428437)]] * 19:24 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab (duration: 01m 32s) * 19:23 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:22 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab * 19:20 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab (duration: 01m 40s) * 19:19 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab * 19:16 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:14 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:06 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:59 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2004 * 18:52 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2004 * 18:52 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2003 * 18:52 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2003 * 18:51 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:51 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2004 to codfw - jhancock@cumin2002" * 18:51 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2004 to codfw - jhancock@cumin2002" * 18:44 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:42 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:42 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2030 to codfw - jhancock@cumin2002" * 18:42 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2030 to codfw - jhancock@cumin2002" * 18:37 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:33 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2002 * 18:32 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2002 * 18:31 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:31 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2002 to codfw - jhancock@cumin2002" * 18:31 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2002 to codfw - jhancock@cumin2002" * 18:25 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:22 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2001 * 18:22 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2001 * 18:21 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:21 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: updating dse-k8s-wdqs2001 to codfw - jhancock@cumin2002" * 18:21 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: updating dse-k8s-wdqs2001 to codfw - jhancock@cumin2002" * 18:17 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:02 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T427286|T427286]] (duration: 00m 12s) * 18:02 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T427286|T427286]] * 17:37 jnuche@deploy1003: Installation of scap version "4.268.0" completed for 2 hosts * 17:35 jnuche@deploy1003: Installing scap version "4.268.0" for 2 host(s) * 17:21 claime: restarting varnish-frontend service on cp6012 * 17:21 claime: restarting varnish-frontend service on cp6011 * 17:21 claime: restarted varnish-frontend service on cp6009 * 17:13 taavi: bounce sirenbot to get it to re-join a channel * 17:05 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 17:05 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:58 urbanecm@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 16:57 urbanecm@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 16:55 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 16:53 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 16:53 urbanecm@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 16:52 urbanecm@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 16:30 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 16:29 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 16:29 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 16:28 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 16:28 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 16:28 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 16:28 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 16:27 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 16:27 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 16:26 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 16:26 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 16:25 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 16:18 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 16:17 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 16:17 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 16:16 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 16:16 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 16:16 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 16:16 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 16:15 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 16:14 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 16:14 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 16:14 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 16:14 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 16:13 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 16:13 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 16:13 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 16:12 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 16:12 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 16:10 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 16:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 16:10 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 16:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 16:10 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 16:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 16:09 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 16:08 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 16:08 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 16:07 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 16:06 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 15:57 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2042: repool after upgrade * 15:45 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db[2183-2184].codfw.wmnet * 15:45 jynus@cumin2002: START - Cookbook sre.hosts.remove-downtime for db[2183-2184].codfw.wmnet * 15:18 jynus: dbmaint on backup1-codfw@codfw ([[phab:T428467|T428467]]) * 15:12 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2042: repool after upgrade * 15:12 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 15:09 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 15:09 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 15:09 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 15:07 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2042.codfw.wmnet with OS trixie * 15:04 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 15:04 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 15:03 jynus@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db[2183-2184].codfw.wmnet with reason: Switchover db * 15:03 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 15:03 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 15:02 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 15:01 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply * 15:00 eevans@deploy1003: helmfile [staging] START helmfile.d/services/data-gateway: apply * 14:59 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:55 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:55 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:54 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:50 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:50 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:50 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:49 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:49 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2042.codfw.wmnet with reason: host reimage * 14:42 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2042.codfw.wmnet with reason: host reimage * 14:32 Lucas_WMDE: UTC afternoon backport+config window done * 14:32 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298709{{!}}Add translatable messages for WikiProject names (T427804)]], [[gerrit:1298710{{!}}Use translatable messages for WikiProject links (T427804)]], [[gerrit:1297644{{!}}WikiProject links - remove 'text' config (T427804)]] (duration: 31m 57s) * 14:27 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 14:26 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2042.codfw.wmnet with OS trixie * 14:26 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 14:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2042: Upgrading es2042.codfw.wmnet * 14:25 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2042: Upgrading es2042.codfw.wmnet * 14:25 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:24 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2043 to es4 codfw primary [[phab:T428386|T428386]]', diff saved to https://phabricator.wikimedia.org/P93926 and previous config saved to /var/cache/conftool/dbconfig/20260608-142423-marostegui.json * 14:23 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 14:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1041: repool after maintenance * 14:19 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Continuing with deployment * 14:18 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Backport for [[gerrit:1298709{{!}}Add translatable messages for WikiProject names (T427804)]], [[gerrit:1298710{{!}}Use translatable messages for WikiProject links (T427804)]], [[gerrit:1297644{{!}}WikiProject links - remove 'text' config (T427804)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:11 cgoubert@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=liftwing-openapi-server.* * 14:10 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp6013.* * 14:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:05 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 14:05 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 13:54 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 13:52 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 13:50 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 13:50 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 13:50 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296550{{!}}hCaptcha: Don't show AbuseFilter CAPTCHA for wbsetclaim API (T427608)]] (duration: 08m 31s) * 13:48 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 13:46 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 13:43 cgoubert@dns1004: END - running authdns-update * 13:43 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296550{{!}}hCaptcha: Don't show AbuseFilter CAPTCHA for wbsetclaim API (T427608)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:41 cgoubert@dns1004: START - running authdns-update * 13:41 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296550{{!}}hCaptcha: Don't show AbuseFilter CAPTCHA for wbsetclaim API (T427608)]] * 13:39 urbanecm@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * {{safesubst:SAL entry|1=13:38 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298758{{!}}feat(V2): toggle experiment features based on custom url override (T424646)]], [[gerrit:1298762{{!}}specialCreateAccount: use GECreateAccountExperimentV2 instead of hook (T424646)]], [[gerrit:1298764{{!}}fix: correctly read experiments param on Special:UserLogin]], [[gerrit:1298765{{!}}signup.js: use JS var instead of TestKitchen to show exp}} * 13:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1041: repool after maintenance * 13:38 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 13:38 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:37 urbanecm@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 13:36 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 13:35 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1041.eqiad.wmnet with OS trixie * 13:34 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 13:34 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2041: repool after upgrade * 13:34 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Continuing with deployment * 13:34 urbanecm@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 13:32 urbanecm@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * {{safesubst:SAL entry|1=13:30 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Backport for [[gerrit:1298758{{!}}feat(V2): toggle experiment features based on custom url override (T424646)]], [[gerrit:1298762{{!}}specialCreateAccount: use GECreateAccountExperimentV2 instead of hook (T424646)]], [[gerrit:1298764{{!}}fix: correctly read experiments param on Special:UserLogin]], [[gerrit:1298765{{!}}signup.js: use JS var instead of TestKitchen to show}} * {{safesubst:SAL entry|1=13:29 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298758{{!}}feat(V2): toggle experiment features based on custom url override (T424646)]], [[gerrit:1298762{{!}}specialCreateAccount: use GECreateAccountExperimentV2 instead of hook (T424646)]], [[gerrit:1298764{{!}}fix: correctly read experiments param on Special:UserLogin]], [[gerrit:1298765{{!}}signup.js: use JS var instead of TestKitchen to show expe}} * 13:21 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298418{{!}}NewUserMessage: Add $wgNewUserMessageOnAutoCreateFirstEdit (T426206)]], [[gerrit:1298717{{!}}Replace NewUserMessageOnAutoCreateFirstEdit with wgNewUserMessageOnFirstEdit (T426206)]], [[gerrit:1298734{{!}}Enable wgNewUserMessageOnFirstEdit on incubatorwiki (T426206)]] (duration: 11m 06s) * 13:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1041.eqiad.wmnet with reason: host reimage * 13:17 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Continuing with deployment * 13:12 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 13:12 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Backport for [[gerrit:1298418{{!}}NewUserMessage: Add $wgNewUserMessageOnAutoCreateFirstEdit (T426206)]], [[gerrit:1298717{{!}}Replace NewUserMessageOnAutoCreateFirstEdit with wgNewUserMessageOnFirstEdit (T426206)]], [[gerrit:1298734{{!}}Enable wgNewUserMessageOnFirstEdit on incubatorwiki (T426206)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki * 13:12 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 13:12 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1041.eqiad.wmnet with reason: host reimage * 13:11 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 13:11 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 13:10 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298418{{!}}NewUserMessage: Add $wgNewUserMessageOnAutoCreateFirstEdit (T426206)]], [[gerrit:1298717{{!}}Replace NewUserMessageOnAutoCreateFirstEdit with wgNewUserMessageOnFirstEdit (T426206)]], [[gerrit:1298734{{!}}Enable wgNewUserMessageOnFirstEdit on incubatorwiki (T426206)]] * 12:57 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298767{{!}}Follow-up: Allow CaptchaConsequence to be skipped via hook (T427608)]] (duration: 06m 20s) * 12:57 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1041.eqiad.wmnet with OS trixie * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:56 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1041: Upgrading es1041.eqiad.wmnet * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:55 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1041: Upgrading es1041.eqiad.wmnet * 12:55 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:54 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:53 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 12:53 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1298767{{!}}Follow-up: Allow CaptchaConsequence to be skipped via hook (T427608)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:51 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:51 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1298767{{!}}Follow-up: Allow CaptchaConsequence to be skipped via hook (T427608)]] * 12:49 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:49 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2041: repool after upgrade * 12:49 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:47 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:46 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 12:44 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 12:43 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 12:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 12:41 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 12:40 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2063.codfw.wmnet with OS bullseye * 12:32 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2062.codfw.wmnet with OS bullseye * 12:27 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2041.codfw.wmnet with OS trixie * 12:21 joal@deploy1003: Finished deploy [analytics/refinery@d67c584] (thin): Regular analytics weekly train THIN [analytics/refinery@d67c584f] (duration: 02m 00s) * 12:19 joal@deploy1003: Started deploy [analytics/refinery@d67c584] (thin): Regular analytics weekly train THIN [analytics/refinery@d67c584f] * 12:19 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2063.codfw.wmnet with reason: host reimage * 12:18 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 12:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 12:16 joal@deploy1003: Finished deploy [analytics/refinery@d67c584]: Regular analytics weekly train [analytics/refinery@d67c584f] (duration: 07m 52s) * 12:15 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2063.codfw.wmnet with reason: host reimage * 12:13 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2062.codfw.wmnet with reason: host reimage * 12:09 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2041.codfw.wmnet with reason: host reimage * 12:08 joal@deploy1003: Started deploy [analytics/refinery@d67c584]: Regular analytics weekly train [analytics/refinery@d67c584f] * 12:08 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2062.codfw.wmnet with reason: host reimage * 12:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add eqiad e8 public vlans - ayounsi@cumin1003" * 12:06 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add eqiad e8 public vlans - ayounsi@cumin1003" * 12:03 joal@deploy1003: Finished deploy [analytics/refinery@d67c584] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@d67c584f] (duration: 02m 00s) * 12:03 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2041.codfw.wmnet with reason: host reimage * 12:01 joal@deploy1003: Started deploy [analytics/refinery@d67c584] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@d67c584f] * 12:01 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 12:00 ayounsi@cumin1003: END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) * 12:00 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 12:00 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 12:00 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:57 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2063 * 11:57 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2063 * 11:57 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2063 * 11:57 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2063.codfw.wmnet 52.16.192.10.in-addr.arpa 2.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:56 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2063.codfw.wmnet 52.16.192.10.in-addr.arpa 2.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:56 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:56 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2063 - mvernon@cumin2002" * 11:56 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2063 - mvernon@cumin2002" * 11:51 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:51 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2063 * 11:50 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2063.codfw.wmnet with OS bullseye * 11:50 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2062 * 11:50 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2062 * 11:49 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2062 * 11:49 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2062.codfw.wmnet 123.0.192.10.in-addr.arpa 3.2.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:49 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2062.codfw.wmnet 123.0.192.10.in-addr.arpa 3.2.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:49 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:49 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2062 - mvernon@cumin2002" * 11:49 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2062 - mvernon@cumin2002" * 11:47 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2041.codfw.wmnet with OS trixie * 11:45 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2041: Upgrading es2041.codfw.wmnet * 11:45 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2041: Upgrading es2041.codfw.wmnet * 11:44 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:44 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.major-upgrade (exit_code=97) * 11:44 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:44 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1042: repool after maintenance * 11:43 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:43 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2062 * 11:42 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2062.codfw.wmnet with OS bullseye * 11:30 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298728{{!}}SpecialMediaSearch: Prefer thumb steps over thumb limits (T424032)]] (duration: 17m 39s) * 11:25 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 11:18 Raine: progressively switching shellbox to bookworm (start) * 11:15 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 11:14 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 11:14 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1298728{{!}}SpecialMediaSearch: Prefer thumb steps over thumb limits (T424032)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:13 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 11:12 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 11:12 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1298728{{!}}SpecialMediaSearch: Prefer thumb steps over thumb limits (T424032)]] * 11:02 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be2062 * 11:02 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be2063 * 10:58 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1042: repool after maintenance * 10:58 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:56 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1042.eqiad.wmnet with OS trixie * 10:47 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298721{{!}}GuessedThumbnailInfo: Also allow showing webp originals (T428202)]] (duration: 16m 41s) * 10:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1042.eqiad.wmnet with reason: host reimage * 10:39 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 10:39 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 10:38 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 10:36 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2160.codfw.wmnet * 10:36 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2160.codfw.wmnet * 10:35 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2043: repool after upgrade * 10:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2160.codfw.wmnet with reason: Reboot * 10:34 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1298721{{!}}GuessedThumbnailInfo: Also allow showing webp originals (T428202)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:34 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1042.eqiad.wmnet with reason: host reimage * 10:30 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1298721{{!}}GuessedThumbnailInfo: Also allow showing webp originals (T428202)]] * 10:18 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1042.eqiad.wmnet with OS trixie * 10:18 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:18 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:18 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:18 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:15 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1042: Upgrading es1042.eqiad.wmnet * 10:14 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:14 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:14 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:14 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:13 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1042: Upgrading es1042.eqiad.wmnet * 10:13 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:12 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be2063 * 10:09 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be2062 * 10:07 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:07 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:07 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:06 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 09:52 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply * 09:52 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply * 09:50 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply * 09:49 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply * 09:49 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2043: repool after upgrade * 09:49 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2043.codfw.wmnet with OS trixie * 09:44 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 09:44 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 09:42 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: sync * 09:42 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: sync * 09:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2043.codfw.wmnet with reason: host reimage * 09:27 jelto@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org * 09:23 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2043.codfw.wmnet with reason: host reimage * 09:17 jelto@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org * 09:15 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: sync * 09:15 ozge@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: sync * 09:07 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2043.codfw.wmnet with OS trixie * 09:06 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2043: Upgrading es2043.codfw.wmnet * 09:06 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2043: Upgrading es2043.codfw.wmnet * 09:05 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:41 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1217.eqiad.wmnet with OS trixie * 08:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1217.eqiad.wmnet with reason: host reimage * 08:15 taavi@cumin1003: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database urwikisource ([[phab:T415977|T415977]]) * 08:14 taavi@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database urwikisource ([[phab:T415977|T415977]]) * 08:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1217.eqiad.wmnet with reason: host reimage * 08:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2052: repool after upgrade * 08:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1051: repool after maintenance * 08:03 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Managing sanitization for wikis urwikisource in section s5 * 07:55 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1217.eqiad.wmnet with OS trixie * 07:53 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1217.eqiad.wmnet with reason: reimage * 07:53 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis urwikisource in section s5 * 07:52 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Checking sanitization for wikis urwikisource in section s5 * 07:50 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis urwikisource in section s5 * 07:50 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.sanitize-wiki (exit_code=97) Managing sanitization for wikis urwikisource in section s5 * 07:50 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis urwikisource in section s5 * 07:44 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297681{{!}}Global rollout - Sub-ref deployments to Group 0, Group 1 and frwiki (T425662)]] (duration: 32m 51s) * 07:32 wmde-fisch@deploy1003: wmde-fisch, lilients: Continuing with deployment * 07:29 wmde-fisch@deploy1003: wmde-fisch, lilients: Backport for [[gerrit:1297681{{!}}Global rollout - Sub-ref deployments to Group 0, Group 1 and frwiki (T425662)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:21 elukey: upgrade sudo package on an-* hosts for [[phab:T428384|T428384]] * 07:18 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2052: repool after upgrade * 07:18 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1051: repool after maintenance * 07:17 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:17 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:12 taavi@cumin1003: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database urwikisource ([[phab:T415977|T415977]]) * 07:12 elukey: upgrade exim4 packages on seaborgium for security upgrades * 07:11 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1297681{{!}}Global rollout - Sub-ref deployments to Group 0, Group 1 and frwiki (T425662)]] * 06:36 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1051.eqiad.wmnet with OS trixie * 06:20 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1051.eqiad.wmnet with reason: host reimage * 06:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1051.eqiad.wmnet with reason: host reimage * 06:15 taavi@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database urwikisource ([[phab:T415977|T415977]]) * 05:58 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1051.eqiad.wmnet with OS trixie * 05:54 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2052.codfw.wmnet with OS trixie * 05:44 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool es1051: Upgrading es1051.eqiad.wmnet * 05:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2052.codfw.wmnet with reason: host reimage * 05:35 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2052.codfw.wmnet with reason: host reimage * 05:35 marostegui@dns1004: END - running authdns-update * 05:34 marostegui@dns1004: START - running authdns-update * 05:33 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1051: Upgrading es1051.eqiad.wmnet * 05:33 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:31 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1054 to es3 eqiad primary [[phab:T428050|T428050]]', diff saved to https://phabricator.wikimedia.org/P93895 and previous config saved to /var/cache/conftool/dbconfig/20260608-053156-marostegui.json * 05:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2052.codfw.wmnet with OS trixie * 05:18 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2052: Upgrading es2052.codfw.wmnet * 05:18 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2052: Upgrading es2052.codfw.wmnet * 05:18 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade == 2026-06-07 == * 16:32 elukey: `elukey@cumin1003:~$ sudo cumin 'cp6* and not cp6014* and not cp6010*' "varnish-frontend-restart" -b 1` * 16:29 elukey: restart varnish-frontend on cp6014 == 2026-06-06 == * 09:07 ammarpad@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=hewiki --logwiki=metawiki W.Mechelke Tungsten_Mechelke # [[phab:T428182|T428182]] == 2026-06-05 == * 22:16 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 21:01 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=10 --verbose` (after stopping the other commons scan) * 20:56 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=30 --verbose` (after stopping the other commons scan) * 20:20 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290093{{!}}Enable wmgUseUrlShortenerLegacy on test2wiki (T107188)]] (duration: 10m 02s) * 20:16 krinkle@deploy1003: krinkle: Continuing with deployment * 20:12 krinkle@deploy1003: krinkle: Backport for [[gerrit:1290093{{!}}Enable wmgUseUrlShortenerLegacy on test2wiki (T107188)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:10 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1290093{{!}}Enable wmgUseUrlShortenerLegacy on test2wiki (T107188)]] * 16:45 jgreen@dns1004: END - running authdns-update * 16:44 jgreen@dns1004: START - running authdns-update * 16:17 dzahn@dns1005: END - running authdns-update * 16:17 mutante: DNS - adding new project language "mag" - Magahi - a language spoken in India and Nepal by about 12 million native speakers ([[phab:T428266|T428266]]) * 16:16 dzahn@dns1005: START - running authdns-update * 14:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:38 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:37 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 12:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:30 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:30 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 12:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2202.codfw.wmnet with reason: Reboot * 12:28 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:28 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 12:08 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:07 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 12:07 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:06 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 11:29 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 11:28 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:55 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:54 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:31 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1054: repool after upgrade * 08:08 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 08:07 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 08:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 08:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/kafka-ui: apply * 07:39 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1054: repool after upgrade * 07:38 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 07:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 07:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/kafka-ui: apply * 07:07 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 06:01 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1054.eqiad.wmnet with OS trixie * 05:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1054.eqiad.wmnet with reason: host reimage * 05:37 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1054.eqiad.wmnet with reason: host reimage * 05:22 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1054.eqiad.wmnet with OS trixie * 05:21 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1054: Upgrading es1054.eqiad.wmnet * 05:21 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1054: Upgrading es1054.eqiad.wmnet * 05:20 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 01:55 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1010.eqiad.wmnet with OS trixie * 01:39 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: host reimage * 01:32 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: host reimage * 01:16 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1010.eqiad.wmnet with OS trixie * 00:56 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1007.eqiad.wmnet with OS trixie * 00:40 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1007.eqiad.wmnet with reason: host reimage * 00:33 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1007.eqiad.wmnet with reason: host reimage * 00:17 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1007.eqiad.wmnet with OS trixie * 00:02 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297268{{!}}Redirect unknown wikinews languages to portal (T427126)]] (duration: 07m 02s) == 2026-06-04 == * 23:57 ladsgroup@deploy1003: ladsgroup, pppery: Continuing with deployment * 23:57 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1006.eqiad.wmnet with OS trixie * 23:57 ladsgroup@deploy1003: ladsgroup, pppery: Backport for [[gerrit:1297268{{!}}Redirect unknown wikinews languages to portal (T427126)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:55 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1297268{{!}}Redirect unknown wikinews languages to portal (T427126)]] * 23:40 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1006.eqiad.wmnet with reason: host reimage * 23:36 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1006.eqiad.wmnet with reason: host reimage * 23:20 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1006.eqiad.wmnet with OS trixie * 21:28 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host releases1003.eqiad.wmnet with OS trixie * 21:04 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases1003.eqiad.wmnet with reason: host reimage * 20:58 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on releases1003.eqiad.wmnet with reason: host reimage * 20:50 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5030.* * 20:42 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host releases1003.eqiad.wmnet with OS trixie * 20:27 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1100.eqiad.wmnet,service=(cdn{{!}}ats-be) * 20:26 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6013.drmrs.wmnet,service=(cdn{{!}}ats-be) * 20:20 brett@dns1006: END - running authdns-update * 20:19 brett@dns1006: START - running authdns-update * 20:18 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5030.eqsin.wmnet with OS trixie * 20:10 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296015{{!}}Deploy PRV to 6 wikis (T427851)]] (duration: 07m 39s) * 20:08 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist group2.dblist extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` * 20:06 arlolra@deploy1003: arlolra: Continuing with deployment * 20:04 arlolra@deploy1003: arlolra: Backport for [[gerrit:1296015{{!}}Deploy PRV to 6 wikis (T427851)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:02 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1296015{{!}}Deploy PRV to 6 wikis (T427851)]] * 19:49 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5030.eqsin.wmnet with reason: host reimage * 19:43 cmooney@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5030.eqsin.wmnet with reason: host reimage * 19:15 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5030 * 19:15 cmooney@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5030 * 19:14 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cp5030 * 19:14 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5030.eqsin.wmnet 27.0.132.10.in-addr.arpa 7.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:14 cmooney@cumin1003: START - Cookbook sre.dns.wipe-cache cp5030.eqsin.wmnet 27.0.132.10.in-addr.arpa 7.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:14 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:14 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5030 - cmooney@cumin1003" * 19:13 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5030 - cmooney@cumin1003" * 19:09 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 19:08 cmooney@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5030 * 19:08 cmooney@cumin1003: START - Cookbook sre.hosts.reimage for host cp5030.eqsin.wmnet with OS trixie * 18:51 cmooney@dns2005: END - running authdns-update * 18:50 cmooney@dns2005: START - running authdns-update * 18:43 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:42 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove IPs that had been used for eqsin cr links - cmooney@cumin1003" * 18:40 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove IPs that had been used for eqsin cr links - cmooney@cumin1003" * 18:37 sukhe: sukhe@cp6013:~$ sudo traffic_server -C clear_cache * 18:36 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 18:08 dancy@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.5 refs [[phab:T423914|T423914]] * 17:17 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297751{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297752{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] (duration: 06m 40s) * 17:13 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 17:13 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297751{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297752{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:11 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297751{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297752{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] * 16:55 topranks: shift traffic off cr1-esams et-1/0/1 link to asw1-by27-esams [[phab:T427056|T427056]] * 16:45 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297741{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297742{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] (duration: 13m 58s) * 16:41 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 16:33 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297741{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297742{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:31 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297741{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297742{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] * 16:17 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 16:03 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297740{{!}}hCaptcha: Move ConfirmEditCaptchaClass hook inside hCaptcha block (T428183)]] (duration: 10m 21s) * 16:03 elukey: uploaded spicerack_12.7.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia * 15:59 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 15:55 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297740{{!}}hCaptcha: Move ConfirmEditCaptchaClass hook inside hCaptcha block (T428183)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:53 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297740{{!}}hCaptcha: Move ConfirmEditCaptchaClass hook inside hCaptcha block (T428183)]] * 15:44 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5030.* * 15:41 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2007.codfw.wmnet with OS trixie * 15:39 ladsgroup@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0) * 15:28 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views * 15:24 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297730{{!}}ptwiki: Disable Article Guidance experiment (T426871)]] (duration: 07m 26s) * 15:24 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2007.codfw.wmnet with reason: host reimage * 15:20 sbisson@deploy1003: sbisson: Continuing with deployment * 15:19 sbisson@deploy1003: sbisson: Backport for [[gerrit:1297730{{!}}ptwiki: Disable Article Guidance experiment (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:19 jayme@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2007.codfw.wmnet with reason: host reimage * 15:17 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1297730{{!}}ptwiki: Disable Article Guidance experiment (T426871)]] * 15:13 ladsgroup@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0) * 15:06 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297724{{!}}Revert "Start reading from new file tables on commons"]] (duration: 07m 00s) * 15:05 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views * 15:02 zabe@deploy1003: zabe: Continuing with deployment * 15:01 zabe@deploy1003: zabe: Backport for [[gerrit:1297724{{!}}Revert "Start reading from new file tables on commons"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:59 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1297724{{!}}Revert "Start reading from new file tables on commons"]] * 14:57 zabe@deploy1003: Finished scap sync-world: [[phab:T416548|T416548]] (duration: 05m 10s) * 14:56 jayme@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-main2007.codfw.wmnet with OS trixie * 14:52 zabe@deploy1003: Started scap sync-world: [[phab:T416548|T416548]] * 14:50 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:49 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:43 zabe@deploy1003: sync-world aborted: Backport for [[gerrit:1270513{{!}}Start reading from new file tables on commons (T416548)]] (duration: 03m 58s) * 14:43 zabe@deploy1003: zabe: Continuing with deployment * 14:41 zabe@deploy1003: zabe: Backport for [[gerrit:1270513{{!}}Start reading from new file tables on commons (T416548)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:40 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f1-codfw * 14:40 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device lsw1-f1-codfw * 14:39 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1270513{{!}}Start reading from new file tables on commons (T416548)]] * 14:36 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297711{{!}}hCaptcha: Enable for MobileFrontend in some Group 2 wikis (T425940)]] (duration: 08m 20s) * 14:32 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:30 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297711{{!}}hCaptcha: Enable for MobileFrontend in some Group 2 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:29 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1057: repool after upgrade * 14:28 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297711{{!}}hCaptcha: Enable for MobileFrontend in some Group 2 wikis (T425940)]] * 14:20 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 14:16 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:15 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply * 14:15 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:15 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply * 14:13 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297704{{!}}Use the globalblock-local-status right over globalblock-whitelist (T277942)]], [[gerrit:1296620{{!}}core-Permissions: Stop assigning unused globalblock-whitelist right (T277942)]] (duration: 06m 46s) * 14:10 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 14:08 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:08 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297704{{!}}Use the globalblock-local-status right over globalblock-whitelist (T277942)]], [[gerrit:1296620{{!}}core-Permissions: Stop assigning unused globalblock-whitelist right (T277942)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:07 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:06 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply * 14:06 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297704{{!}}Use the globalblock-local-status right over globalblock-whitelist (T277942)]], [[gerrit:1296620{{!}}core-Permissions: Stop assigning unused globalblock-whitelist right (T277942)]] * 14:06 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:06 tappof: bump space for prometheus k8s-aux in eqiad * 14:05 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply * 14:05 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:04 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply * 13:56 _joe_: transferred requestctl api tokens for all ops to the db ([[phab:T428119|T428119]]) * 13:56 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2050 to es3 codfw primary [[phab:T428050|T428050]]', diff saved to https://phabricator.wikimedia.org/P93878 and previous config saved to /var/cache/conftool/dbconfig/20260604-135631-marostegui.json * 13:56 Dreamy_Jazz: Afternoon UTC backport window done * 13:54 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297700{{!}}Revert "hCaptcha: Provide always challenge sitekey for account creation"]] (duration: 13m 38s) * 13:51 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 13:50 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 13:47 sukhe: sukhe@cp6011:~$ sudo -i varnish-frontend-restart * 13:44 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1057: repool after upgrade * 13:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:43 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297700{{!}}Revert "hCaptcha: Provide always challenge sitekey for account creation"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:41 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1057.eqiad.wmnet with OS trixie * 13:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297700{{!}}Revert "hCaptcha: Provide always challenge sitekey for account creation"]] * 13:38 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297692{{!}}hCaptcha: Provide always challenge sitekey for account creation (T421041)]] (duration: 05m 27s) * 13:38 dreamyjazz@deploy1003: dreamyjazz: Rolling back deployment * 13:36 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: down * 13:35 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297692{{!}}hCaptcha: Provide always challenge sitekey for account creation (T421041)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:33 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297692{{!}}hCaptcha: Provide always challenge sitekey for account creation (T421041)]] * 13:31 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295978{{!}}Update config for WikiProjects linking prototype (T427804)]] (duration: 17m 13s) * 13:26 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Continuing with deployment * 13:25 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1057.eqiad.wmnet with reason: host reimage * 13:17 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1057.eqiad.wmnet with reason: host reimage * 13:16 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Backport for [[gerrit:1295978{{!}}Update config for WikiProjects linking prototype (T427804)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:14 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1295978{{!}}Update config for WikiProjects linking prototype (T427804)]] * 13:13 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:13 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1220: Migration of db1220.eqiad.wmnet completed * 13:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: down * 13:12 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1224', diff saved to https://phabricator.wikimedia.org/P93875 and previous config saved to /var/cache/conftool/dbconfig/20260604-131219-marostegui.json * 13:00 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1057.eqiad.wmnet with OS trixie * 13:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1057: Upgrading es1057.eqiad.wmnet * 12:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1057: Upgrading es1057.eqiad.wmnet * 12:59 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:56 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296557{{!}}wmf-config: Skip CAPTCHA for action=mcrundo (T427612)]] (duration: 08m 30s) * 12:52 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Continuing with deployment * 12:50 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Backport for [[gerrit:1296557{{!}}wmf-config: Skip CAPTCHA for action=mcrundo (T427612)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2050: repool after upgrade * 12:48 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296557{{!}}wmf-config: Skip CAPTCHA for action=mcrundo (T427612)]] * 12:37 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 12:37 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 12:28 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1220: Migration of db1220.eqiad.wmnet completed * 12:20 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1220.eqiad.wmnet with OS trixie * 12:04 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2050: repool after upgrade * 12:04 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 12:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1220.eqiad.wmnet with reason: host reimage * 11:59 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1220.eqiad.wmnet with reason: host reimage * 11:42 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1220.eqiad.wmnet with OS trixie * 11:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2050.codfw.wmnet with OS trixie * 11:40 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1220: Upgrading db1220.eqiad.wmnet * 11:37 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1220: Upgrading db1220.eqiad.wmnet * 11:36 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1179: Migration of db1179.eqiad.wmnet completed * 11:23 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2050.codfw.wmnet with reason: host reimage * 11:16 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2050.codfw.wmnet with reason: host reimage * 11:00 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2050.codfw.wmnet with OS trixie * 11:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2050: Upgrading es2050.codfw.wmnet * 10:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2050: Upgrading es2050.codfw.wmnet * 10:59 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:59 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2057: repool after upgrade * 10:58 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:55 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 10:46 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1179: Migration of db1179.eqiad.wmnet completed * 10:38 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1179.eqiad.wmnet with OS trixie * 10:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1179.eqiad.wmnet with reason: host reimage * 10:16 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply * 10:15 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply * 10:15 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply * 10:15 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/kartotherian: apply * 10:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1179.eqiad.wmnet with reason: host reimage * 10:13 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2057: repool after upgrade * 10:13 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2057.codfw.wmnet with OS trixie * 09:59 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1179.eqiad.wmnet with OS trixie * 09:58 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1179: Upgrading db1179.eqiad.wmnet * 09:58 jynus: redoing m2 backups after grant change [[phab:T411111|T411111]] * 09:57 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1179: Upgrading db1179.eqiad.wmnet * 09:56 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:54 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2057.codfw.wmnet with reason: host reimage * 09:53 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 09:49 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2057.codfw.wmnet with reason: host reimage * 09:39 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:39 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Migration of db1224.eqiad.wmnet completed * 09:38 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 09:37 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 09:36 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 09:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/kafka-ui: apply * 09:33 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2057.codfw.wmnet with OS trixie * 09:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2057: Upgrading es2057.codfw.wmnet * 09:32 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2057: Upgrading es2057.codfw.wmnet * 09:31 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:26 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=30 --sleep=60 --verbose` * 09:25 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist "group0.dblist + group1.dblist - mediamoderation-continuous-scan.dblist" extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` * 08:54 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Introduce pluggable authentication - oblivian@cumin1003" * 08:54 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Introduce pluggable authentication - oblivian@cumin1003 * 08:53 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Migration of db1224.eqiad.wmnet completed * 08:53 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Introduce pluggable authentication - oblivian@cumin1003 * 08:53 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Introduce pluggable authentication - oblivian@cumin1003" * 08:29 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 08:29 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 08:24 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 08:24 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 08:21 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 08:21 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1224.eqiad.wmnet with OS trixie * 08:21 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 08:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1224.eqiad.wmnet with reason: host reimage * 08:02 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2249.codfw.wmnet with reason: upgrade * 08:00 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1224.eqiad.wmnet with reason: host reimage * 07:53 marostegui: Install mariadb 10.11.17 on db2249 [[phab:T427345|T427345]] * 07:43 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1224.eqiad.wmnet with OS trixie * 07:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1224: Upgrading db1224.eqiad.wmnet * 07:41 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1224: Upgrading db1224.eqiad.wmnet * 07:41 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:39 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:39 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1255: Migration of db1255.eqiad.wmnet completed * 07:34 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297536{{!}}hCaptcha risk scores: VE plugin to collect risk scores for block notices (T426943)]], [[gerrit:1297200{{!}}hCaptcha: Render a fresh mobile widget for each captcha attempt (T425929)]], [[gerrit:1297173{{!}}hCaptcha: Enable risk-score collection for users blocked by IP blocks (T424629)]] (duration: 08m 56s) * 07:29 kharlan@deploy1003: kharlan, harroyo-wmf: Continuing with deployment * 07:27 kharlan@deploy1003: kharlan, harroyo-wmf: Backport for [[gerrit:1297536{{!}}hCaptcha risk scores: VE plugin to collect risk scores for block notices (T426943)]], [[gerrit:1297200{{!}}hCaptcha: Render a fresh mobile widget for each captcha attempt (T425929)]], [[gerrit:1297173{{!}}hCaptcha: Enable risk-score collection for users blocked by IP blocks (T424629)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwd * 07:25 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297536{{!}}hCaptcha risk scores: VE plugin to collect risk scores for block notices (T426943)]], [[gerrit:1297200{{!}}hCaptcha: Render a fresh mobile widget for each captcha attempt (T425929)]], [[gerrit:1297173{{!}}hCaptcha: Enable risk-score collection for users blocked by IP blocks (T424629)]] * 07:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2191: Migration of db2191.codfw.wmnet completed * 07:12 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297550{{!}}Revert "EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion"]] (duration: 06m 45s) * 07:08 kharlan@deploy1003: kharlan: Continuing with deployment * 07:08 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297550{{!}}Revert "EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:06 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297550{{!}}Revert "EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion"]] * 07:04 otto@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297260{{!}}EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] (duration: 399m 30s) * 07:03 otto@deploy1003: otto: Rolling back deployment * 06:53 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1255: Migration of db1255.eqiad.wmnet completed * 06:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1255.eqiad.wmnet with OS trixie * 06:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2191: Migration of db2191.codfw.wmnet completed * 06:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1255.eqiad.wmnet with reason: host reimage * 06:32 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2191.codfw.wmnet with OS trixie * 06:31 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1255.eqiad.wmnet with reason: host reimage * 06:16 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1255.eqiad.wmnet with OS trixie * 06:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2191.codfw.wmnet with reason: host reimage * 06:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1255: Upgrading db1255.eqiad.wmnet * 06:12 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1255: Upgrading db1255.eqiad.wmnet * 06:12 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2191.codfw.wmnet with reason: host reimage * 06:04 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db1255 [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93836 and previous config saved to /var/cache/conftool/dbconfig/20260604-060428-cwilliams.json * 06:03 cwilliams@dns1004: END - running authdns-update * 06:02 cwilliams@dns1004: START - running authdns-update * 05:54 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db1258 to x3 primary and set section read-write [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93835 and previous config saved to /var/cache/conftool/dbconfig/20260604-055429-cwilliams.json * 05:53 cwilliams@cumin1003: dbctl commit (dc=all): 'Set x3 eqiad as read-only for maintenance - [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93834 and previous config saved to /var/cache/conftool/dbconfig/20260604-055346-cwilliams.json * 05:53 cezmunsta: Starting x3 eqiad failover from db1255 to db1258 - [[phab:T427895|T427895]] * 05:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2191.codfw.wmnet with OS trixie * 05:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2191: Upgrading db2191.codfw.wmnet * 05:50 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2191: Upgrading db2191.codfw.wmnet * 05:50 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db1258 with weight 0 [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93833 and previous config saved to /var/cache/conftool/dbconfig/20260604-055021-cwilliams.json * 05:50 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:50 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Primary switchover x3 [[phab:T427895|T427895]] * 05:48 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 05:46 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2191 [[phab:T428120|T428120]]', diff saved to https://phabricator.wikimedia.org/P93832 and previous config saved to /var/cache/conftool/dbconfig/20260604-054614-marostegui.json * 05:45 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db2215 to x1 primary [[phab:T428120|T428120]]', diff saved to https://phabricator.wikimedia.org/P93831 and previous config saved to /var/cache/conftool/dbconfig/20260604-054528-marostegui.json * 05:44 marostegui: Starting x1 codfw failover from db2191 to db2215 - [[phab:T428120|T428120]] * 05:27 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: Primary switchover x1 [[phab:T428120|T428120]] * 05:27 marostegui@cumin1003: dbctl commit (dc=all): 'Set db2215 with weight 0 [[phab:T428120|T428120]]', diff saved to https://phabricator.wikimedia.org/P93830 and previous config saved to /var/cache/conftool/dbconfig/20260604-052722-marostegui.json * 05:19 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 03:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93829 and previous config saved to /var/cache/conftool/dbconfig/20260604-034546-fceratto.json * 03:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P93828 and previous config saved to /var/cache/conftool/dbconfig/20260604-033538-fceratto.json * 03:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P93827 and previous config saved to /var/cache/conftool/dbconfig/20260604-032531-fceratto.json * 03:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93826 and previous config saved to /var/cache/conftool/dbconfig/20260604-031523-fceratto.json * 03:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1263 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93825 and previous config saved to /var/cache/conftool/dbconfig/20260604-030710-fceratto.json * 03:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1263.eqiad.wmnet with reason: Maintenance * 03:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93824 and previous config saved to /var/cache/conftool/dbconfig/20260604-030642-fceratto.json * 02:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P93823 and previous config saved to /var/cache/conftool/dbconfig/20260604-025634-fceratto.json * 02:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P93822 and previous config saved to /var/cache/conftool/dbconfig/20260604-024627-fceratto.json * 02:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93821 and previous config saved to /var/cache/conftool/dbconfig/20260604-023619-fceratto.json * 02:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1262 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93820 and previous config saved to /var/cache/conftool/dbconfig/20260604-022809-fceratto.json * 02:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Maintenance * 02:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93819 and previous config saved to /var/cache/conftool/dbconfig/20260604-022742-fceratto.json * 02:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P93818 and previous config saved to /var/cache/conftool/dbconfig/20260604-021734-fceratto.json * 02:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P93817 and previous config saved to /var/cache/conftool/dbconfig/20260604-020726-fceratto.json * 01:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93816 and previous config saved to /var/cache/conftool/dbconfig/20260604-015718-fceratto.json * 01:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1261 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93815 and previous config saved to /var/cache/conftool/dbconfig/20260604-014909-fceratto.json * 01:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1261.eqiad.wmnet with reason: Maintenance * 01:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93814 and previous config saved to /var/cache/conftool/dbconfig/20260604-014841-fceratto.json * 01:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P93813 and previous config saved to /var/cache/conftool/dbconfig/20260604-013833-fceratto.json * 01:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P93812 and previous config saved to /var/cache/conftool/dbconfig/20260604-012826-fceratto.json * 01:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93811 and previous config saved to /var/cache/conftool/dbconfig/20260604-011818-fceratto.json * 01:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1260 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93810 and previous config saved to /var/cache/conftool/dbconfig/20260604-011005-fceratto.json * 01:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1260.eqiad.wmnet with reason: Maintenance * 01:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93809 and previous config saved to /var/cache/conftool/dbconfig/20260604-010937-fceratto.json * 00:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P93808 and previous config saved to /var/cache/conftool/dbconfig/20260604-005929-fceratto.json * 00:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P93807 and previous config saved to /var/cache/conftool/dbconfig/20260604-004922-fceratto.json * 00:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93806 and previous config saved to /var/cache/conftool/dbconfig/20260604-003914-fceratto.json * 00:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1252 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93805 and previous config saved to /var/cache/conftool/dbconfig/20260604-002851-fceratto.json * 00:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1252.eqiad.wmnet with reason: Maintenance * 00:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93804 and previous config saved to /var/cache/conftool/dbconfig/20260604-002821-fceratto.json * 00:26 otto@deploy1003: otto: Backport for [[gerrit:1297260{{!}}EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 00:24 otto@deploy1003: Started scap sync-world: Backport for [[gerrit:1297260{{!}}EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] * 00:18 Amir1: mwscript-k8s --follow --dblist=all -- extensions/timeline/maintenance/DeleteOldTimelineFiles.php --date {{Gerrit|20210101000000}} * 00:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P93803 and previous config saved to /var/cache/conftool/dbconfig/20260604-001813-fceratto.json * 00:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P93802 and previous config saved to /var/cache/conftool/dbconfig/20260604-000805-fceratto.json == 2026-06-03 == * 23:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93801 and previous config saved to /var/cache/conftool/dbconfig/20260603-235758-fceratto.json * 23:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93800 and previous config saved to /var/cache/conftool/dbconfig/20260603-234935-fceratto.json * 23:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1248.eqiad.wmnet with reason: Maintenance * 23:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93799 and previous config saved to /var/cache/conftool/dbconfig/20260603-234907-fceratto.json * 23:42 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296561{{!}}Add a maintenance script to delete old files]], [[gerrit:1296560{{!}}Add a maintenance script to delete old files]] (duration: 07m 09s) * 23:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P93798 and previous config saved to /var/cache/conftool/dbconfig/20260603-233859-fceratto.json * 23:37 ladsgroup@deploy1003: ladsgroup, reedy: Continuing with deployment * 23:36 ladsgroup@deploy1003: ladsgroup, reedy: Backport for [[gerrit:1296561{{!}}Add a maintenance script to delete old files]], [[gerrit:1296560{{!}}Add a maintenance script to delete old files]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:34 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1296561{{!}}Add a maintenance script to delete old files]], [[gerrit:1296560{{!}}Add a maintenance script to delete old files]] * 23:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P93797 and previous config saved to /var/cache/conftool/dbconfig/20260603-232852-fceratto.json * 23:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 23:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 23:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93796 and previous config saved to /var/cache/conftool/dbconfig/20260603-231844-fceratto.json * 23:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93795 and previous config saved to /var/cache/conftool/dbconfig/20260603-231031-fceratto.json * 23:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1247.eqiad.wmnet with reason: Maintenance * 23:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93794 and previous config saved to /var/cache/conftool/dbconfig/20260603-231001-fceratto.json * 22:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P93793 and previous config saved to /var/cache/conftool/dbconfig/20260603-225953-fceratto.json * 22:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P93792 and previous config saved to /var/cache/conftool/dbconfig/20260603-224945-fceratto.json * 22:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93791 and previous config saved to /var/cache/conftool/dbconfig/20260603-223937-fceratto.json * 22:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1244 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93790 and previous config saved to /var/cache/conftool/dbconfig/20260603-223116-fceratto.json * 22:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1244.eqiad.wmnet with reason: Maintenance * 22:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93789 and previous config saved to /var/cache/conftool/dbconfig/20260603-223048-fceratto.json * 22:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P93788 and previous config saved to /var/cache/conftool/dbconfig/20260603-222041-fceratto.json * 22:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P93787 and previous config saved to /var/cache/conftool/dbconfig/20260603-221034-fceratto.json * 22:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93786 and previous config saved to /var/cache/conftool/dbconfig/20260603-220026-fceratto.json * 21:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1243 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93785 and previous config saved to /var/cache/conftool/dbconfig/20260603-215110-fceratto.json * 21:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1243.eqiad.wmnet with reason: Maintenance * 21:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93784 and previous config saved to /var/cache/conftool/dbconfig/20260603-215053-fceratto.json * 21:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P93783 and previous config saved to /var/cache/conftool/dbconfig/20260603-214046-fceratto.json * 21:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P93782 and previous config saved to /var/cache/conftool/dbconfig/20260603-213038-fceratto.json * 21:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93781 and previous config saved to /var/cache/conftool/dbconfig/20260603-212030-fceratto.json * 21:12 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1242 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93779 and previous config saved to /var/cache/conftool/dbconfig/20260603-211206-fceratto.json * 21:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1242.eqiad.wmnet with reason: Maintenance * 21:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93778 and previous config saved to /var/cache/conftool/dbconfig/20260603-211138-fceratto.json * 21:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P93774 and previous config saved to /var/cache/conftool/dbconfig/20260603-210130-fceratto.json * 20:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P93773 and previous config saved to /var/cache/conftool/dbconfig/20260603-205122-fceratto.json * 20:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93772 and previous config saved to /var/cache/conftool/dbconfig/20260603-204115-fceratto.json * 20:33 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297228{{!}}Attribution research don't use testKitchen compatibility layer (T417050)]] (duration: 06m 41s) * 20:32 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1241 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93771 and previous config saved to /var/cache/conftool/dbconfig/20260603-203254-fceratto.json * 20:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1241.eqiad.wmnet with reason: Maintenance * 20:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93770 and previous config saved to /var/cache/conftool/dbconfig/20260603-203227-fceratto.json * 20:29 cjming@deploy1003: cjming: Continuing with deployment * 20:29 cjming@deploy1003: cjming: Backport for [[gerrit:1297228{{!}}Attribution research don't use testKitchen compatibility layer (T417050)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:26 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1297228{{!}}Attribution research don't use testKitchen compatibility layer (T417050)]] * 20:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P93769 and previous config saved to /var/cache/conftool/dbconfig/20260603-202219-fceratto.json * 20:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P93766 and previous config saved to /var/cache/conftool/dbconfig/20260603-201211-fceratto.json * 20:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93765 and previous config saved to /var/cache/conftool/dbconfig/20260603-200203-fceratto.json * 19:59 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 19:59 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 19:59 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 19:59 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 19:53 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93764 and previous config saved to /var/cache/conftool/dbconfig/20260603-195341-fceratto.json * 19:53 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1238.eqiad.wmnet with reason: Maintenance * 19:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93763 and previous config saved to /var/cache/conftool/dbconfig/20260603-195313-fceratto.json * 19:47 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5032.* * 19:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P93762 and previous config saved to /var/cache/conftool/dbconfig/20260603-194306-fceratto.json * 19:39 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5032.* * 19:37 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5032.* * 19:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P93761 and previous config saved to /var/cache/conftool/dbconfig/20260603-193258-fceratto.json * 19:26 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 19:25 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 19:25 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 19:25 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 19:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93760 and previous config saved to /var/cache/conftool/dbconfig/20260603-192250-fceratto.json * 19:22 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 19:22 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 19:14 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93759 and previous config saved to /var/cache/conftool/dbconfig/20260603-191437-fceratto.json * 19:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1024-1025].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 19:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1221.eqiad.wmnet with reason: Maintenance * 19:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93758 and previous config saved to /var/cache/conftool/dbconfig/20260603-191348-fceratto.json * 19:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P93757 and previous config saved to /var/cache/conftool/dbconfig/20260603-190340-fceratto.json * 18:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P93756 and previous config saved to /var/cache/conftool/dbconfig/20260603-185331-fceratto.json * 18:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93755 and previous config saved to /var/cache/conftool/dbconfig/20260603-184324-fceratto.json * 18:34 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1199 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93754 and previous config saved to /var/cache/conftool/dbconfig/20260603-183455-fceratto.json * 18:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance * 18:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93753 and previous config saved to /var/cache/conftool/dbconfig/20260603-183427-fceratto.json * 18:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P93752 and previous config saved to /var/cache/conftool/dbconfig/20260603-182420-fceratto.json * 18:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P93751 and previous config saved to /var/cache/conftool/dbconfig/20260603-181412-fceratto.json * 18:10 dancy@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.5 refs [[phab:T423914|T423914]] * 18:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93750 and previous config saved to /var/cache/conftool/dbconfig/20260603-180404-fceratto.json * 17:57 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5032.* * 17:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93749 and previous config saved to /var/cache/conftool/dbconfig/20260603-175544-fceratto.json * 17:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance * 17:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93748 and previous config saved to /var/cache/conftool/dbconfig/20260603-175342-fceratto.json * 17:52 hashar: contint1003: sudo puppet agent --disable "Prevent Jenkins from coming back" * 17:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P93747 and previous config saved to /var/cache/conftool/dbconfig/20260603-174334-fceratto.json * 17:38 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:37 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 17:37 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:36 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:36 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:35 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:35 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:35 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:34 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:34 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:33 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P93746 and previous config saved to /var/cache/conftool/dbconfig/20260603-173327-fceratto.json * 17:33 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:32 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:29 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5032.* * 17:26 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host sretest2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 17:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93745 and previous config saved to /var/cache/conftool/dbconfig/20260603-172319-fceratto.json * 17:18 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: Stopping before sync operations * 17:17 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: Started scap sync-world: No-deploy scap run to verify scap config change * 17:17 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:15 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:15 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1253 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93744 and previous config saved to /var/cache/conftool/dbconfig/20260603-171521-fceratto.json * 17:15 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:15 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance * 17:14 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93743 and previous config saved to /var/cache/conftool/dbconfig/20260603-171452-fceratto.json * 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:13 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:12 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:10 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:09 ayounsi@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2012.wikimedia.org with OS trixie * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P93742 and previous config saved to /var/cache/conftool/dbconfig/20260603-170444-fceratto.json * 17:04 swfrench@deploy1003: Stopping before sync operations * 17:03 swfrench@deploy1003: Started scap sync-world: No-deploy scap run to verify clean state before config change * 16:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P93741 and previous config saved to /var/cache/conftool/dbconfig/20260603-165436-fceratto.json * 16:53 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:53 hashar: Restarting CI Jenkins one last time # [[phab:T418521|T418521]] * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:44 btullis@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295922{{!}}Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] (duration: 07m 16s) * 16:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93740 and previous config saved to /var/cache/conftool/dbconfig/20260603-164428-fceratto.json * 16:43 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:43 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:42 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:41 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:40 btullis@deploy1003: btullis: Continuing with deployment * 16:39 btullis@deploy1003: btullis: Backport for [[gerrit:1295922{{!}}Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93739 and previous config saved to /var/cache/conftool/dbconfig/20260603-163726-fceratto.json * 16:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance * 16:37 btullis@deploy1003: Started scap sync-world: Backport for [[gerrit:1295922{{!}}Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] * 16:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93738 and previous config saved to /var/cache/conftool/dbconfig/20260603-163658-fceratto.json * 16:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P93737 and previous config saved to /var/cache/conftool/dbconfig/20260603-162650-fceratto.json * 16:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P93736 and previous config saved to /var/cache/conftool/dbconfig/20260603-161643-fceratto.json * 16:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93735 and previous config saved to /var/cache/conftool/dbconfig/20260603-160635-fceratto.json * 16:04 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:59 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93734 and previous config saved to /var/cache/conftool/dbconfig/20260603-155928-fceratto.json * 15:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance * 15:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93733 and previous config saved to /var/cache/conftool/dbconfig/20260603-155859-fceratto.json * 15:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 15:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P93732 and previous config saved to /var/cache/conftool/dbconfig/20260603-154852-fceratto.json * 15:46 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:46 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2012.wikimedia.org with OS trixie * 15:40 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:40 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 15:40 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 15:40 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 15:39 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 15:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P93731 and previous config saved to /var/cache/conftool/dbconfig/20260603-153844-fceratto.json * 15:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93729 and previous config saved to /var/cache/conftool/dbconfig/20260603-152836-fceratto.json * 15:25 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host sretest2012 * 15:25 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2012 * 15:25 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host sretest2012 * 15:25 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2012 * 15:24 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:23 mutante: disabling jenkins on CI servers for maintenance * 15:23 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host sretest2012 * 15:23 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2012 * 15:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 15:21 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1202 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93728 and previous config saved to /var/cache/conftool/dbconfig/20260603-152129-fceratto.json * 15:21 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance * 15:21 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:21 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding sretest2012 to codfw - jhancock@cumin2002" * 15:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 15:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93727 and previous config saved to /var/cache/conftool/dbconfig/20260603-152102-fceratto.json * 15:20 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding sretest2012 to codfw - jhancock@cumin2002" * 15:18 brouberol@dns1004: END - running authdns-update * 15:18 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:16 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 15:16 brouberol@dns1004: START - running authdns-update * 15:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P93726 and previous config saved to /var/cache/conftool/dbconfig/20260603-151055-fceratto.json * 15:01 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P93725 and previous config saved to /var/cache/conftool/dbconfig/20260603-150047-fceratto.json * 14:57 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 14:52 cmooney@cumin1003: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox * 14:51 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93723 and previous config saved to /var/cache/conftool/dbconfig/20260603-145039-fceratto.json * 14:48 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297137{{!}}Revert "MultimediaViewer: enable image carousel as a beta feature on Wikipedias"]] (duration: 06m 46s) * 14:47 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 14:46 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:46 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:43 mlitn@deploy1003: mlitn: Continuing with deployment * 14:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93722 and previous config saved to /var/cache/conftool/dbconfig/20260603-144334-fceratto.json * 14:43 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:43 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance * 14:43 mlitn@deploy1003: mlitn: Backport for [[gerrit:1297137{{!}}Revert "MultimediaViewer: enable image carousel as a beta feature on Wikipedias"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93721 and previous config saved to /var/cache/conftool/dbconfig/20260603-144306-fceratto.json * 14:41 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:41 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:41 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1297137{{!}}Revert "MultimediaViewer: enable image carousel as a beta feature on Wikipedias"]] * 14:39 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:39 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:39 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:39 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:38 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:35 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 14:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 14:34 sgimeno@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297130{{!}}editor: make redesigned anon warning the default experience (T424595)]] (duration: 10m 45s) * 14:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P93719 and previous config saved to /var/cache/conftool/dbconfig/20260603-143259-fceratto.json * 14:30 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:28 sgimeno@deploy1003: sgimeno: Continuing with deployment * 14:25 sgimeno@deploy1003: sgimeno: Backport for [[gerrit:1297130{{!}}editor: make redesigned anon warning the default experience (T424595)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:24 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:24 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:23 sgimeno@deploy1003: Started scap sync-world: Backport for [[gerrit:1297130{{!}}editor: make redesigned anon warning the default experience (T424595)]] * 14:23 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P93717 and previous config saved to /var/cache/conftool/dbconfig/20260603-142251-fceratto.json * 14:22 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:22 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:21 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:21 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:21 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:20 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:20 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:20 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:20 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:19 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:19 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:16 vriley@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:16 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:16 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:13 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:12 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93716 and previous config saved to /var/cache/conftool/dbconfig/20260603-141242-fceratto.json * 14:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:11 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:11 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:10 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mc2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:10 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host mc2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:10 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:09 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:08 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:07 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:05 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296631{{!}}translate: adding separate read/write endpoints (T425377)]] (duration: 13m 06s) * 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1191 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93715 and previous config saved to /var/cache/conftool/dbconfig/20260603-140537-fceratto.json * 14:05 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance * 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93714 and previous config saved to /var/cache/conftool/dbconfig/20260603-140507-fceratto.json * 14:01 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:58 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:58 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:58 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 13:56 dcausse@deploy1003: atsuko, dcausse: Rolling back deployment * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T426633|T426633]])', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20260603-133440-fceratto.json * 13:29 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:29 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2186: Migration of db2186.codfw.wmnet completed * 13:28 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295910{{!}}hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]] (duration: 07m 36s) * 13:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1174 ([[phab:T426633|T426633]])', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20260603-132638-fceratto.json * 13:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance * 13:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93710 and previous config saved to /var/cache/conftool/dbconfig/20260603-132605-fceratto.json * 13:25 sukhe: sudo cumin 'A:lvs or A:liberica' 'disable-puppet "merging CR 1282764"' * 13:23 kharlan@deploy1003: kharlan: Continuing with deployment * 13:22 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295910{{!}}hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:20 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295910{{!}}hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]] * 13:18 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296649{{!}}hCaptcha: Roll out to all except enwiki for mobile apps. (T426048)]] (duration: 07m 46s) * 13:16 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20260603-131556-fceratto.json * 13:15 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:13 kharlan@deploy1003: dbrant, kharlan: Continuing with deployment * 13:12 kharlan@deploy1003: dbrant, kharlan: Backport for [[gerrit:1296649{{!}}hCaptcha: Roll out to all except enwiki for mobile apps. (T426048)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296649{{!}}hCaptcha: Roll out to all except enwiki for mobile apps. (T426048)]] * 13:09 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:09 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add codfw d3 and e5 public vlans - ayounsi@cumin1003" * 13:09 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add codfw d3 and e5 public vlans - ayounsi@cumin1003" * 13:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P93708 and previous config saved to /var/cache/conftool/dbconfig/20260603-130548-fceratto.json * 13:05 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 12:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93706 and previous config saved to /var/cache/conftool/dbconfig/20260603-125540-fceratto.json * 12:51 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297110{{!}}ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] (duration: 07m 44s) * 12:49 jgreen@dns1004: END - running authdns-update * 12:47 jgreen@dns1004: START - running authdns-update * 12:46 jiji@deploy1003: jiji: Continuing with deployment * 12:46 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93705 and previous config saved to /var/cache/conftool/dbconfig/20260603-124624-fceratto.json * 12:46 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance * 12:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93704 and previous config saved to /var/cache/conftool/dbconfig/20260603-124556-fceratto.json * 12:45 jiji@deploy1003: jiji: Backport for [[gerrit:1297110{{!}}ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:43 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2186: Migration of db2186.codfw.wmnet completed * 12:43 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1297110{{!}}ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] * 12:41 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1067.eqiad.wmnet with OS bullseye * 12:38 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1292364{{!}}Update hCaptcha checks to retrieve API parameters from $_REQUEST (T427105)]] (duration: 11m 15s) * 12:36 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2186.codfw.wmnet with OS trixie * 12:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P93702 and previous config saved to /var/cache/conftool/dbconfig/20260603-123548-fceratto.json * 12:34 dreamyjazz@deploy1003: somerandomdeveloper, dreamyjazz: Continuing with deployment * 12:31 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1066.eqiad.wmnet with OS bullseye * 12:29 dreamyjazz@deploy1003: somerandomdeveloper, dreamyjazz: Backport for [[gerrit:1292364{{!}}Update hCaptcha checks to retrieve API parameters from $_REQUEST (T427105)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:27 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1292364{{!}}Update hCaptcha checks to retrieve API parameters from $_REQUEST (T427105)]] * 12:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P93701 and previous config saved to /var/cache/conftool/dbconfig/20260603-122541-fceratto.json * 12:22 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1067.eqiad.wmnet with reason: host reimage * 12:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2186.codfw.wmnet with reason: host reimage * 12:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93700 and previous config saved to /var/cache/conftool/dbconfig/20260603-121533-fceratto.json * 12:13 mvernon@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ms-be1066.eqiad.wmnet with reason: host reimage * 12:13 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2186.codfw.wmnet with reason: host reimage * 12:11 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1067.eqiad.wmnet with reason: host reimage * 12:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93699 and previous config saved to /var/cache/conftool/dbconfig/20260603-120732-fceratto.json * 12:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 12:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance * 12:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93698 and previous config saved to /var/cache/conftool/dbconfig/20260603-120634-fceratto.json * 12:03 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1066.eqiad.wmnet with reason: host reimage * 11:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P93697 and previous config saved to /var/cache/conftool/dbconfig/20260603-115626-fceratto.json * 11:54 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2186.codfw.wmnet with OS trixie * 11:54 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be1067 * 11:54 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1067 * 11:52 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1067 * 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be1067.eqiad.wmnet 96.48.64.10.in-addr.arpa 6.9.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:52 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be1067.eqiad.wmnet 96.48.64.10.in-addr.arpa 6.9.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1067 - mvernon@cumin2002" * 11:52 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1067 - mvernon@cumin2002" * 11:48 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2186: Upgrading db2186.codfw.wmnet * 11:48 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2186: Upgrading db2186.codfw.wmnet * 11:48 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:47 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:46 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be1067 * 11:46 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be1067.eqiad.wmnet with OS bullseye * 11:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P93695 and previous config saved to /var/cache/conftool/dbconfig/20260603-114618-fceratto.json * 11:46 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be1066 * 11:46 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1066 * 11:45 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1066 * 11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be1066.eqiad.wmnet 117.32.64.10.in-addr.arpa 7.1.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:45 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be1066.eqiad.wmnet 117.32.64.10.in-addr.arpa 7.1.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1066 - mvernon@cumin2002" * 11:45 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1066 - mvernon@cumin2002" * 11:43 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply * 11:41 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:40 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be1066 * 11:40 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be1066.eqiad.wmnet with OS bullseye * 11:39 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be1067 * 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93693 and previous config saved to /var/cache/conftool/dbconfig/20260603-113611-fceratto.json * 11:33 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:33 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2196: Migration of db2196.codfw.wmnet completed * 11:29 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93691 and previous config saved to /var/cache/conftool/dbconfig/20260603-112909-fceratto.json * 11:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance * 11:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance * 11:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93690 and previous config saved to /var/cache/conftool/dbconfig/20260603-112838-fceratto.json * 11:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:20 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:20 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:20 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P93689 and previous config saved to /var/cache/conftool/dbconfig/20260603-111831-fceratto.json * 11:14 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:09 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply * 11:09 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply * 11:08 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P93687 and previous config saved to /var/cache/conftool/dbconfig/20260603-110823-fceratto.json * 11:07 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be1066 * 11:07 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply * 11:06 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply * 11:05 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply * 11:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:01 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:01 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:00 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289895{{!}}Update UserInfoCard to be enabled by default for certain user groups (T426021)]] (duration: 07m 37s) * 11:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:59 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply * 10:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:59 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply * 10:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:58 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply * 10:58 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93685 and previous config saved to /var/cache/conftool/dbconfig/20260603-105815-fceratto.json * 10:58 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:56 mszwarc@deploy1003: mszwarc: Continuing with deployment * 10:55 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1289895{{!}}Update UserInfoCard to be enabled by default for certain user groups (T426021)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:54 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply * 10:54 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply * 10:53 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply * 10:53 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1289895{{!}}Update UserInfoCard to be enabled by default for certain user groups (T426021)]] * 10:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:50 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1198 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93684 and previous config saved to /var/cache/conftool/dbconfig/20260603-105006-fceratto.json * 10:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance * 10:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93683 and previous config saved to /var/cache/conftool/dbconfig/20260603-104939-fceratto.json * 10:45 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:45 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:44 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2196: Migration of db2196.codfw.wmnet completed * 10:44 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:41 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 10:40 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 10:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P93681 and previous config saved to /var/cache/conftool/dbconfig/20260603-103931-fceratto.json * 10:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1053: repool after upgrade * 10:37 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2196.codfw.wmnet with OS trixie * 10:36 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297090{{!}}hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] (duration: 12m 03s) * 10:32 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P93679 and previous config saved to /var/cache/conftool/dbconfig/20260603-102924-fceratto.json * 10:26 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297090{{!}}hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:24 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297090{{!}}hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] * 10:22 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be1067 * 10:21 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be1066 * 10:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2196.codfw.wmnet with reason: host reimage * 10:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93677 and previous config saved to /var/cache/conftool/dbconfig/20260603-101916-fceratto.json * 10:15 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2013.codfw.wmnet * 10:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2196.codfw.wmnet with reason: host reimage * 10:11 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93676 and previous config saved to /var/cache/conftool/dbconfig/20260603-101105-fceratto.json * 10:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1189.eqiad.wmnet with reason: Maintenance * 10:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93675 and previous config saved to /var/cache/conftool/dbconfig/20260603-101037-fceratto.json * 10:10 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2013.codfw.wmnet * 10:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P93673 and previous config saved to /var/cache/conftool/dbconfig/20260603-100029-fceratto.json * 09:59 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2196.codfw.wmnet with OS trixie * 09:57 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2196: Upgrading db2196.codfw.wmnet * 09:57 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2196: Upgrading db2196.codfw.wmnet * 09:57 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:52 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1053: repool after upgrade * 09:52 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:52 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:52 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:52 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:51 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:51 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:51 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P93670 and previous config saved to /var/cache/conftool/dbconfig/20260603-095022-fceratto.json * 09:49 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:49 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:48 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1053.eqiad.wmnet with OS trixie * 09:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:43 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2013.codfw.wmnet * 09:41 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on es1053.eqiad.wmnet with reason: host reimage * 09:41 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1053.eqiad.wmnet with reason: host reimage * 09:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93669 and previous config saved to /var/cache/conftool/dbconfig/20260603-094014-fceratto.json * 09:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2215: Migration of db2215.codfw.wmnet completed * 09:38 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2013.codfw.wmnet * 09:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93667 and previous config saved to /var/cache/conftool/dbconfig/20260603-093146-fceratto.json * 09:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance * 09:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93666 and previous config saved to /var/cache/conftool/dbconfig/20260603-093119-fceratto.json * 09:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1211: Migration of db1211.eqiad.wmnet completed * 09:27 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297069{{!}}hCaptcha: Collect risk score for blocked account creations (T427784)]] (duration: 07m 26s) * 09:25 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1053.eqiad.wmnet with OS trixie * 09:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add public1-b3-codfw gateway IPs - ayounsi@cumin1003" * 09:24 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add public1-b3-codfw gateway IPs - ayounsi@cumin1003" * 09:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1053: Upgrading es1053.eqiad.wmnet * 09:23 kharlan@deploy1003: kharlan: Continuing with deployment * 09:22 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1053: Upgrading es1053.eqiad.wmnet * 09:22 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:21 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297069{{!}}hCaptcha: Collect risk score for blocked account creations (T427784)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:21 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply * 09:21 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2054: repool after upgrade * 09:21 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply * 09:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P93661 and previous config saved to /var/cache/conftool/dbconfig/20260603-092111-fceratto.json * 09:20 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 09:20 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297069{{!}}hCaptcha: Collect risk score for blocked account creations (T427784)]] * 09:14 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297065{{!}}Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] (duration: 07m 06s) * 09:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P93659 and previous config saved to /var/cache/conftool/dbconfig/20260603-091104-fceratto.json * 09:10 kharlan@deploy1003: kharlan: Continuing with deployment * 09:09 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297065{{!}}Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:07 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297065{{!}}Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] * 09:06 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 09:06 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297064{{!}}Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 10m 54s) * 09:05 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 09:04 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 09:01 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003 - [[phab:T422043|T422043]]" * 09:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93656 and previous config saved to /var/cache/conftool/dbconfig/20260603-090056-fceratto.json * 09:00 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003 - [[phab:T422043|T422043]]" * 09:00 ayounsi@cumin1003: END (ERROR) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=97) generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003" * 09:00 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003" * 08:59 kharlan@deploy1003: kharlan: Continuing with deployment * 08:59 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297064{{!}}Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:55 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297064{{!}}Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] * 08:53 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296635{{!}}Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 11m 43s) * 08:52 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2215: Migration of db2215.codfw.wmnet completed * 08:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet * 08:52 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet * 08:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb[1022-1023].eqiad.wmnet * 08:51 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb[1022-1023].eqiad.wmnet * 08:50 kharlan@deploy1003: kharlan: Rolling back deployment * 08:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93652 and previous config saved to /var/cache/conftool/dbconfig/20260603-084846-fceratto.json * 08:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance * 08:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93651 and previous config saved to /var/cache/conftool/dbconfig/20260603-084819-fceratto.json * 08:47 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296635{{!}}Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2215.codfw.wmnet with OS trixie * 08:45 jiji@cumin1003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check docker-registry: maintenance * 08:45 jiji@cumin1003: START - Cookbook sre.discovery.service-route check docker-registry: maintenance * 08:43 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1211: Migration of db1211.eqiad.wmnet completed * 08:41 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296635{{!}}Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] * 08:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1211.eqiad.wmnet with OS trixie * 08:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93649 and previous config saved to /var/cache/conftool/dbconfig/20260603-083811-fceratto.json * 08:37 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296632{{!}}Image Browsing: add accessible labels to carousel elements (T407793)]] (duration: 32m 11s) * 08:36 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2054: repool after upgrade * 08:35 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool es2054.codfw.wmnet: After reimage * 08:35 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2054.codfw.wmnet: After reimage * 08:35 jiji@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:34 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 08:34 jiji@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:33 jiji@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:33 jiji@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:31 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:31 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:31 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2054.codfw.wmnet with OS trixie * 08:30 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:29 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 08:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2215.codfw.wmnet with reason: host reimage * 08:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93647 and previous config saved to /var/cache/conftool/dbconfig/20260603-082804-fceratto.json * 08:25 mszwarc@deploy1003: mlitn, mszwarc: Continuing with deployment * 08:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1211.eqiad.wmnet with reason: host reimage * 08:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1049: repool after upgrade * 08:22 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2215.codfw.wmnet with reason: host reimage * 08:22 mszwarc@deploy1003: mlitn, mszwarc: Backport for [[gerrit:1296632{{!}}Image Browsing: add accessible labels to carousel elements (T407793)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:18 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1211.eqiad.wmnet with reason: host reimage * 08:18 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 08:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93645 and previous config saved to /var/cache/conftool/dbconfig/20260603-081756-fceratto.json * 08:17 jiji@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 08:17 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 08:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 08:14 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2054.codfw.wmnet with reason: host reimage * 08:08 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2054.codfw.wmnet with reason: host reimage * 08:05 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296632{{!}}Image Browsing: add accessible labels to carousel elements (T407793)]] * {{safesubst:SAL entry|1=08:04 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296580{{!}}Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703{{!}}jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713{{!}}conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627{{!}}Add missing lazy img to carousel (T427821)]], [[gerrit:1295968{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T426799)]}} * 08:03 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93643 and previous config saved to /var/cache/conftool/dbconfig/20260603-080346-fceratto.json * 08:03 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1211.eqiad.wmnet with OS trixie * 08:03 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance * 08:03 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2215.codfw.wmnet with OS trixie * 08:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1211: Upgrading db1211.eqiad.wmnet * 08:02 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2215: Upgrading db2215.codfw.wmnet * 08:01 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:01 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1211: Upgrading db1211.eqiad.wmnet * 08:01 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2215: Upgrading db2215.codfw.wmnet * 08:01 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:01 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:01 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1157: Repooling * 08:01 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1157: Repooling * 08:00 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 07:57 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1022-1023].eqiad.wmnet with reason: Reimaging upstream server * 07:57 mszwarc@deploy1003: anzx, mlitn, mfossati, mszwarc: Continuing with deployment * 07:56 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Reimaging upstream server * {{safesubst:SAL entry|1=07:54 mszwarc@deploy1003: anzx, mlitn, mfossati, mszwarc: Backport for [[gerrit:1296580{{!}}Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703{{!}}jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713{{!}}conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627{{!}}Add missing lazy img to carousel (T427821)]], [[gerrit:1295968{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T42}} * 07:52 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2231: repool after maintenance * 07:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2054.codfw.wmnet with OS trixie * 07:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2054: Upgrading es2054.codfw.wmnet * 07:50 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2054: Upgrading es2054.codfw.wmnet * 07:50 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:50 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296580{{!}}Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703{{!}}jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713{{!}}conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627{{!}}Add missing lazy img to carousel (T427821)]], [[gerrit:1295968{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T426799)]] * 07:48 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296516{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] (duration: 32m 13s) * 07:44 marostegui@dns1004: END - running authdns-update * 07:43 marostegui@dns1004: START - running authdns-update * 07:42 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1056 to es2 eqiad primary [[phab:T427875|T427875]]', diff saved to https://phabricator.wikimedia.org/P93637 and previous config saved to /var/cache/conftool/dbconfig/20260603-074250-marostegui.json * 07:37 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1049: repool after upgrade * 07:37 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:35 mszwarc@deploy1003: mszwarc, stran: Continuing with deployment * 07:35 mszwarc@deploy1003: mszwarc, stran: Backport for [[gerrit:1296516{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:32 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1049.eqiad.wmnet with OS trixie * 07:16 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296516{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] * 07:14 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1049.eqiad.wmnet with reason: host reimage * 07:07 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1049.eqiad.wmnet with reason: host reimage * 07:07 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2231: repool after maintenance * 07:04 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 06:57 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2231.codfw.wmnet with OS trixie * 06:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1049.eqiad.wmnet with OS trixie * 06:46 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1049: Upgrading es1049.eqiad.wmnet * 06:46 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2056 to es2 codfw primary [[phab:T427875|T427875]]', diff saved to https://phabricator.wikimedia.org/P93632 and previous config saved to /var/cache/conftool/dbconfig/20260603-064623-marostegui.json * 06:45 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1049: Upgrading es1049.eqiad.wmnet * 06:45 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:44 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1056: repool after upgrade * 06:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2231.codfw.wmnet with reason: host reimage * 06:36 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2231.codfw.wmnet with reason: host reimage * 06:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2231.codfw.wmnet with OS trixie * 06:09 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2231: Upgrading db2231.codfw.wmnet * 06:09 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2231: Upgrading db2231.codfw.wmnet * 06:09 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:59 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1056: repool after upgrade * 05:59 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 05:55 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1056.eqiad.wmnet with OS trixie * 05:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1056.eqiad.wmnet with reason: host reimage * 05:33 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1056.eqiad.wmnet with reason: host reimage * 05:18 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1056.eqiad.wmnet with OS trixie * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1056: Upgrading es1056.eqiad.wmnet * 05:17 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1056: Upgrading es1056.eqiad.wmnet * 05:16 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade == 2026-06-02 == * 22:21 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296689{{!}}hCaptcha: Correct inaccurate comment]] (duration: 06m 27s) * 22:18 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 22:18 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 22:17 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 22:17 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296689{{!}}hCaptcha: Correct inaccurate comment]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:15 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296689{{!}}hCaptcha: Correct inaccurate comment]] * 22:13 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296551{{!}}hCaptcha: Enable for badlogin on group0 wikis (T426875)]] (duration: 08m 31s) * 22:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 22:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 22:09 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 22:07 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296551{{!}}hCaptcha: Enable for badlogin on group0 wikis (T426875)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:05 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296551{{!}}hCaptcha: Enable for badlogin on group0 wikis (T426875)]] * 20:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93621 and previous config saved to /var/cache/conftool/dbconfig/20260602-203945-fceratto.json * 20:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93620 and previous config saved to /var/cache/conftool/dbconfig/20260602-202937-fceratto.json * 20:27 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1054.eqiad.wmnet * 20:27 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:27 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1054.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:26 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1054.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:20 jiji@cumin1003: START - Cookbook sre.dns.netbox * 20:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93619 and previous config saved to /var/cache/conftool/dbconfig/20260602-201929-fceratto.json * 20:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93618 and previous config saved to /var/cache/conftool/dbconfig/20260602-200922-fceratto.json * 20:03 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1054.eqiad.wmnet * 19:48 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1053.eqiad.wmnet * 19:48 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:48 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1053.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 19:37 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1053.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 19:09 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93617 and previous config saved to /var/cache/conftool/dbconfig/20260602-190907-fceratto.json * 19:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance * 19:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93616 and previous config saved to /var/cache/conftool/dbconfig/20260602-190811-fceratto.json * 19:05 dancy@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.5 refs [[phab:T423914|T423914]] * 18:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P93615 and previous config saved to /var/cache/conftool/dbconfig/20260602-185804-fceratto.json * 18:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P93614 and previous config saved to /var/cache/conftool/dbconfig/20260602-184757-fceratto.json * 18:38 jiji@cumin1003: START - Cookbook sre.dns.netbox * 18:38 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:38 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93612 and previous config saved to /var/cache/conftool/dbconfig/20260602-183749-fceratto.json * 18:37 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:37 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:33 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1053.eqiad.wmnet * 18:30 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1259 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93611 and previous config saved to /var/cache/conftool/dbconfig/20260602-183023-fceratto.json * 18:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance * 18:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93610 and previous config saved to /var/cache/conftool/dbconfig/20260602-182956-fceratto.json * 18:27 mutante: gerrit delete unused plugin projects: barricade, WikimediaBlocks and WikimediaWebSessions * 18:26 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1052.eqiad.wmnet * 18:26 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:26 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1052.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 18:25 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1052.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 18:25 dancy: Train is blocked at testwikis on https://phabricator.wikimedia.org/T427935 * 18:21 Daimona: Running query from [[phab:T427962|T427962]]#11978299 in x1.wikishared * 18:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P93609 and previous config saved to /var/cache/conftool/dbconfig/20260602-181949-fceratto.json * 18:16 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296615{{!}}feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614{{!}}feat(cleanMentorList): Add a feature flag (T427386)]] (duration: 34m 09s) * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 18:12 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 18:12 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 18:12 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 18:10 jiji@cumin1003: START - Cookbook sre.dns.netbox * 18:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P93608 and previous config saved to /var/cache/conftool/dbconfig/20260602-180941-fceratto.json * 18:08 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 18:07 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 18:06 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 18:06 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 18:05 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:05 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:05 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 18:05 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 18:04 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 18:02 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 18:02 swfrench-wmf: reverting shellbox to 2026-05-20-192555 due to errors in shellbox-syntaxhighlight * 18:02 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 18:01 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 18:01 urbanecm@deploy1003: urbanecm: Continuing with deployment * 18:01 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1296615{{!}}feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614{{!}}feat(cleanMentorList): Add a feature flag (T427386)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:00 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1052.eqiad.wmnet * 17:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93607 and previous config saved to /var/cache/conftool/dbconfig/20260602-175933-fceratto.json * 17:58 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:57 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:56 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1051.eqiad.wmnet * 17:56 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:56 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1051.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:55 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1051.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:53 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93605 and previous config saved to /var/cache/conftool/dbconfig/20260602-175227-fceratto.json * 17:52 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance * 17:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93604 and previous config saved to /var/cache/conftool/dbconfig/20260602-175157-fceratto.json * 17:51 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:51 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:50 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:50 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:50 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:49 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:49 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:48 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:48 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:47 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:44 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:42 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:42 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:42 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P93603 and previous config saved to /var/cache/conftool/dbconfig/20260602-174150-fceratto.json * 17:41 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1296615{{!}}feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614{{!}}feat(cleanMentorList): Add a feature flag (T427386)]] * 17:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P93602 and previous config saved to /var/cache/conftool/dbconfig/20260602-173143-fceratto.json * 17:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93601 and previous config saved to /var/cache/conftool/dbconfig/20260602-172135-fceratto.json * 17:14 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93600 and previous config saved to /var/cache/conftool/dbconfig/20260602-171422-fceratto.json * 17:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance * 17:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93599 and previous config saved to /var/cache/conftool/dbconfig/20260602-171354-fceratto.json * 17:04 jiji@cumin1003: START - Cookbook sre.dns.netbox * 17:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P93598 and previous config saved to /var/cache/conftool/dbconfig/20260602-170344-fceratto.json * 16:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P93597 and previous config saved to /var/cache/conftool/dbconfig/20260602-165336-fceratto.json * 16:49 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1051.eqiad.wmnet * 16:48 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1050.eqiad.wmnet * 16:48 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:48 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1050.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:47 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1050.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93596 and previous config saved to /var/cache/conftool/dbconfig/20260602-164328-fceratto.json * 16:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93595 and previous config saved to /var/cache/conftool/dbconfig/20260602-163622-fceratto.json * 16:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance * 16:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93594 and previous config saved to /var/cache/conftool/dbconfig/20260602-163550-fceratto.json * 16:34 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:34 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1072.eqiad.wmnet with OS trixie * 16:30 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:29 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:27 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2006.codfw.wmnet with OS trixie * 16:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P93593 and previous config saved to /var/cache/conftool/dbconfig/20260602-162542-fceratto.json * 16:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P93591 and previous config saved to /var/cache/conftool/dbconfig/20260602-161534-fceratto.json * 16:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1072.eqiad.wmnet with reason: host reimage * 16:10 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1071.eqiad.wmnet with OS trixie * 16:10 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296624{{!}}Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 06m 40s) * 16:09 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2006.codfw.wmnet with reason: host reimage * 16:05 kharlan@deploy1003: kharlan: Continuing with deployment * 16:05 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1072.eqiad.wmnet with reason: host reimage * 16:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93590 and previous config saved to /var/cache/conftool/dbconfig/20260602-160527-fceratto.json * 16:05 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2006.codfw.wmnet with reason: host reimage * 16:05 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296624{{!}}Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:03 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296624{{!}}Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] * 15:59 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295909{{!}}hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] (duration: 09m 48s) * 15:59 kharlan@deploy1003: kharlan: Rolling back deployment * 15:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93589 and previous config saved to /var/cache/conftool/dbconfig/20260602-155817-fceratto.json * 15:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance * 15:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93588 and previous config saved to /var/cache/conftool/dbconfig/20260602-155749-fceratto.json * 15:54 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1071.eqiad.wmnet with reason: host reimage * 15:53 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1072.eqiad.wmnet with OS trixie * 15:51 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1070.eqiad.wmnet with OS trixie * 15:51 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295909{{!}}hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:50 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1071.eqiad.wmnet with reason: host reimage * 15:49 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295909{{!}}hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] * 15:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P93587 and previous config saved to /var/cache/conftool/dbconfig/20260602-154742-fceratto.json * 15:47 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296558{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] (duration: 07m 24s) * 15:43 kharlan@deploy1003: kharlan: Continuing with deployment * 15:42 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296558{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:40 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296558{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] * 15:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P93586 and previous config saved to /var/cache/conftool/dbconfig/20260602-153734-fceratto.json * 15:37 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1071.eqiad.wmnet with OS trixie * 15:36 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1069.eqiad.wmnet with OS trixie * 15:35 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1070.eqiad.wmnet with reason: host reimage * 15:32 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:32 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:31 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1070.eqiad.wmnet with reason: host reimage * 15:30 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:29 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93585 and previous config saved to /var/cache/conftool/dbconfig/20260602-152726-fceratto.json * 15:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2158: Repooling * {{safesubst:SAL entry|1=15:22 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295502{{!}}Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106{{!}}Remove unused 'writeapi' right]], [[gerrit:1296566{{!}}Clean up bot password configuration]], [[gerrit:1296563{{!}}Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574{{!}}cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582{{!}}U}} * 15:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1069.eqiad.wmnet with reason: host reimage * 15:20 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93583 and previous config saved to /var/cache/conftool/dbconfig/20260602-152026-fceratto.json * 15:20 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance * 15:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93582 and previous config saved to /var/cache/conftool/dbconfig/20260602-151958-fceratto.json * 15:19 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:19 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:18 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1070.eqiad.wmnet with OS trixie * 15:18 dreamyjazz@deploy1003: matmarex, anzx, dreamyjazz: Continuing with deployment * 15:18 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 15:17 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:17 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:15 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1069.eqiad.wmnet with reason: host reimage * {{safesubst:SAL entry|1=15:15 dreamyjazz@deploy1003: matmarex, anzx, dreamyjazz: Backport for [[gerrit:1295502{{!}}Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106{{!}}Remove unused 'writeapi' right]], [[gerrit:1296566{{!}}Clean up bot password configuration]], [[gerrit:1296563{{!}}Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574{{!}}cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582}} * 15:14 jiji@cumin1003: START - Cookbook sre.dns.netbox * {{safesubst:SAL entry|1=15:13 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1295502{{!}}Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106{{!}}Remove unused 'writeapi' right]], [[gerrit:1296566{{!}}Clean up bot password configuration]], [[gerrit:1296563{{!}}Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574{{!}}cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582{{!}}Us}} * 15:12 jayme@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-main2006.codfw.wmnet with OS trixie * 15:12 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1068.eqiad.wmnet with OS trixie * 15:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P93580 and previous config saved to /var/cache/conftool/dbconfig/20260602-150951-fceratto.json * 15:09 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296514{{!}}[Growth] Set wgGEMentorshipCleanupEnabled to false on all wikis (T427386)]] (duration: 06m 22s) * 15:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1167: Repooling after Icing wait-for-green timeout * 15:06 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1050.eqiad.wmnet * 15:06 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1049.eqiad.wmnet * 15:06 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:06 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1049.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:05 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1049.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:02 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1296514{{!}}[Growth] Set wgGEMentorshipCleanupEnabled to false on all wikis (T427386)]] * 15:02 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1069.eqiad.wmnet with OS trixie * 15:01 jiji@cumin1003: START - Cookbook sre.dns.netbox * 14:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P93578 and previous config saved to /var/cache/conftool/dbconfig/20260602-145943-fceratto.json * 14:54 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage * 14:52 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:52 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:52 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1049.eqiad.wmnet * 14:51 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1067.eqiad.wmnet with OS trixie * 14:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:50 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage * 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93575 and previous config saved to /var/cache/conftool/dbconfig/20260602-144935-fceratto.json * 14:42 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for pc2021.codfw.wmnet * 14:42 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for pc2021.codfw.wmnet * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2250.codfw.wmnet * 14:41 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2250.codfw.wmnet * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2158.codfw.wmnet * 14:41 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2158.codfw.wmnet * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc2021: Repooling * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 14:41 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache * 14:41 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc2021: Repooling * 14:41 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93573 and previous config saved to /var/cache/conftool/dbconfig/20260602-144110-fceratto.json * 14:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance * 14:41 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2158: Repooling * 14:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93571 and previous config saved to /var/cache/conftool/dbconfig/20260602-144043-fceratto.json * 14:38 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:38 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:38 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:37 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1048.eqiad.wmnet * 14:37 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:37 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1048.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 14:37 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1068.eqiad.wmnet with OS trixie * 14:36 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1066.eqiad.wmnet with OS trixie * 14:34 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage * 14:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P93569 and previous config saved to /var/cache/conftool/dbconfig/20260602-143035-fceratto.json * 14:30 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage * 14:25 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1048.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 14:21 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1167: Repooling after Icing wait-for-green timeout * 14:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage * 14:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P93566 and previous config saved to /var/cache/conftool/dbconfig/20260602-142027-fceratto.json * 14:17 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1067.eqiad.wmnet with OS trixie * 14:17 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 14:17 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1167.eqiad.wmnet * 14:17 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1167.eqiad.wmnet * 14:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1065.eqiad.wmnet with OS trixie * 14:15 jayme@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2006.codfw.wmnet with OS trixie * 14:14 jiji@cumin1003: START - Cookbook sre.dns.netbox * 14:13 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage * 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93564 and previous config saved to /var/cache/conftool/dbconfig/20260602-141019-fceratto.json * 14:09 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments userOptions.php --delete --nowarn growthexperiments-homepage-variant # [[phab:T417621|T417621]] * 14:09 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1048.eqiad.wmnet * 14:08 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments userOptions.php --delete growthexperiments-homepage-variant # [[phab:T417621|T417621]] * 14:05 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 14:01 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93563 and previous config saved to /var/cache/conftool/dbconfig/20260602-140140-fceratto.json * 14:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 14:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1156.eqiad.wmnet with reason: Maintenance * 14:01 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1066.eqiad.wmnet with OS trixie * 14:00 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage * 14:00 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 14:00 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 14:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93562 and previous config saved to /var/cache/conftool/dbconfig/20260602-140022-fceratto.json * 14:00 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1064.eqiad.wmnet with OS trixie * 13:56 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage * 13:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1167.eqiad.wmnet with OS trixie * 13:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P93561 and previous config saved to /var/cache/conftool/dbconfig/20260602-135015-fceratto.json * 13:47 topranks: revert all config to normal on cr1-codfw and ssw1-a1-codfw * 13:43 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1065.eqiad.wmnet with OS trixie * 13:42 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1064.eqiad.wmnet with reason: host reimage * 13:40 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1063.eqiad.wmnet with OS trixie * 13:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P93560 and previous config saved to /var/cache/conftool/dbconfig/20260602-134007-fceratto.json * 13:38 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1167.eqiad.wmnet with reason: host reimage * 13:35 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs1002.eqiad.wmnet with OS trixie * 13:35 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs1003.eqiad.wmnet with OS trixie * 13:34 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:34 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:32 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1064.eqiad.wmnet with reason: host reimage * 13:31 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1167.eqiad.wmnet with reason: host reimage * 13:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93559 and previous config saved to /var/cache/conftool/dbconfig/20260602-132959-fceratto.json * 13:27 slyngshede@dns1004: END - running authdns-update * 13:25 slyngshede@dns1004: START - running authdns-update * 13:24 topranks: increase OSPF cost on ssw1-a1-codfw et-0/0/4 towards lsw1-a5-codfw [[phab:T427301|T427301]] * 13:23 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1063.eqiad.wmnet with reason: host reimage * 13:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93558 and previous config saved to /var/cache/conftool/dbconfig/20260602-132314-fceratto.json * 13:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1210.eqiad.wmnet with reason: Maintenance * 13:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93557 and previous config saved to /var/cache/conftool/dbconfig/20260602-132246-fceratto.json * 13:20 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1064.eqiad.wmnet with OS trixie * 13:19 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 13:19 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1062.eqiad.wmnet with OS trixie * 13:18 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1063.eqiad.wmnet with reason: host reimage * 13:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2049: repool after upgrade * 13:17 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 13:16 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1167.eqiad.wmnet with OS trixie * 13:15 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 13:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1167: Upgrading db1167.eqiad.wmnet * 13:13 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1167: Upgrading db1167.eqiad.wmnet * 13:13 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:12 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 13:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P93554 and previous config saved to /var/cache/conftool/dbconfig/20260602-131238-fceratto.json * 13:12 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 13:12 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 13:11 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 13:07 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1003.eqiad.wmnet with OS trixie * 13:07 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1002.eqiad.wmnet with OS trixie * 13:06 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1063.eqiad.wmnet with OS trixie * 13:04 jayme@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-main2006.codfw.wmnet with OS trixie * 13:04 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:04 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:03 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1022-1023].eqiad.wmnet with reason: Reimaging upstream servers * 13:03 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1001.eqiad.wmnet with OS trixie * 13:03 topranks: increase OSPF cost on ssw1-a1-codfw et-0/0/2 towards lsw1-a3-codfw [[phab:T427301|T427301]] * 13:03 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1062.eqiad.wmnet with reason: host reimage * 13:02 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Reimaging upstream servers * 13:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P93553 and previous config saved to /var/cache/conftool/dbconfig/20260602-130230-fceratto.json * 12:59 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1062.eqiad.wmnet with reason: host reimage * 12:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:57 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:57 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2161: Migration of db2161.codfw.wmnet completed * 12:54 topranks: shutdown sub-interfaces on cr1-codfw et-1/1/5 for row A/B vlans [[phab:T427301|T427301]] * 12:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 12:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93550 and previous config saved to /var/cache/conftool/dbconfig/20260602-125223-fceratto.json * 12:50 topranks: enable bgp graceful-shutdown in overlay on ssw1-a1-codfw [[phab:T427301|T427301]] * 12:49 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc1061.eqiad.wmnet with OS trixie * 12:48 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt * 12:48 ayounsi@cumin1003: START - Cookbook sre.hosts.remove-downtime for lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt * 12:47 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1062.eqiad.wmnet with OS trixie * 12:45 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93548 and previous config saved to /var/cache/conftool/dbconfig/20260602-124541-fceratto.json * 12:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance * 12:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93547 and previous config saved to /var/cache/conftool/dbconfig/20260602-124512-fceratto.json * 12:43 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc1060.eqiad.wmnet with OS trixie * 12:42 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 12:42 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mc1061.eqiad.wmnet with reason: host reimage * 12:42 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1061.eqiad.wmnet with reason: host reimage * 12:41 topranks: enable bgp graceful-shutdown in underlay on ssw1-a1-codfw [[phab:T427301|T427301]] * 12:35 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mc1060.eqiad.wmnet with reason: host reimage * 12:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P93545 and previous config saved to /var/cache/conftool/dbconfig/20260602-123505-fceratto.json * 12:33 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 12:33 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1060.eqiad.wmnet with reason: host reimage * 12:31 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2049: repool after upgrade * 12:31 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 12:29 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1061.eqiad.wmnet with OS trixie * 12:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2049.codfw.wmnet with OS trixie * 12:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P93542 and previous config saved to /var/cache/conftool/dbconfig/20260602-122459-fceratto.json * 12:24 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1059.eqiad.wmnet with OS trixie * 12:21 XioNoX: reboot lsw1-a3-codfw for software upgrade - [[phab:T427301|T427301]] * 12:20 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1060.eqiad.wmnet with OS trixie * 12:20 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 12:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1058.eqiad.wmnet with OS trixie * 12:17 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 12:16 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296532{{!}}hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533{{!}}hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] (duration: 09m 02s) * 12:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93539 and previous config saved to /var/cache/conftool/dbconfig/20260602-121451-fceratto.json * 12:11 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 12:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2049.codfw.wmnet with reason: host reimage * 12:11 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt with reason: Switch maintenance * 12:10 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2161: Migration of db2161.codfw.wmnet completed * 12:09 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Switch maintenance * 12:09 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296532{{!}}hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533{{!}}hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:08 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1200 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93537 and previous config saved to /var/cache/conftool/dbconfig/20260602-120755-fceratto.json * 12:07 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1059.eqiad.wmnet with reason: host reimage * 12:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance * 12:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93536 and previous config saved to /var/cache/conftool/dbconfig/20260602-120728-fceratto.json * 12:07 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 12:07 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296532{{!}}hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533{{!}}hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] * 12:05 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2049.codfw.wmnet with reason: host reimage * 12:04 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1058.eqiad.wmnet with reason: host reimage * 12:02 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1059.eqiad.wmnet with reason: host reimage * 12:01 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2161.codfw.wmnet with OS trixie * 12:00 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1058.eqiad.wmnet with reason: host reimage * 11:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P93535 and previous config saved to /var/cache/conftool/dbconfig/20260602-115721-fceratto.json * 11:55 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 11:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:55 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 11:53 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 11:53 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 11:53 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:50 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1059.eqiad.wmnet with OS trixie * 11:49 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1057.eqiad.wmnet with OS trixie * 11:49 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2049.codfw.wmnet with OS trixie * 11:48 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2049: Upgrading es2049.codfw.wmnet * 11:48 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2049: Upgrading es2049.codfw.wmnet * 11:47 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:47 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1058.eqiad.wmnet with OS trixie * 11:47 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2056: repool after upgrade * 11:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P93532 and previous config saved to /var/cache/conftool/dbconfig/20260602-114713-fceratto.json * 11:45 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1056.eqiad.wmnet with OS trixie * 11:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2161.codfw.wmnet with reason: host reimage * 11:40 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2161.codfw.wmnet with reason: host reimage * 11:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93531 and previous config saved to /var/cache/conftool/dbconfig/20260602-113705-fceratto.json * 11:33 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1057.eqiad.wmnet with reason: host reimage * 11:30 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1185 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93529 and previous config saved to /var/cache/conftool/dbconfig/20260602-113019-fceratto.json * 11:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance * 11:29 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1056.eqiad.wmnet with reason: host reimage * 11:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1161: Repooling * 11:26 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1161: Repooling * 11:23 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2161.codfw.wmnet with OS trixie * 11:22 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1057.eqiad.wmnet with reason: host reimage * 11:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2161: Upgrading db2161.codfw.wmnet * 11:21 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2161: Upgrading db2161.codfw.wmnet * 11:21 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1056.eqiad.wmnet with reason: host reimage * 11:21 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P93527 and previous config saved to /var/cache/conftool/dbconfig/20260602-111954-fceratto.json * 11:15 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2161 [[phab:T427892|T427892]]', diff saved to https://phabricator.wikimedia.org/P93525 and previous config saved to /var/cache/conftool/dbconfig/20260602-111511-cwilliams.json * 11:12 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2165 to s8 primary [[phab:T427892|T427892]]', diff saved to https://phabricator.wikimedia.org/P93524 and previous config saved to /var/cache/conftool/dbconfig/20260602-111200-cwilliams.json * 11:10 cezmunsta: Starting s8 codfw failover from db2161 to db2165 - [[phab:T427892|T427892]] * 11:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P93523 and previous config saved to /var/cache/conftool/dbconfig/20260602-110947-fceratto.json * 11:09 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1057.eqiad.wmnet with OS trixie * 11:09 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1056.eqiad.wmnet with OS trixie * 11:04 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2165 with weight 0 [[phab:T427892|T427892]]', diff saved to https://phabricator.wikimedia.org/P93522 and previous config saved to /var/cache/conftool/dbconfig/20260602-110420-cwilliams.json * 11:03 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s8 [[phab:T427892|T427892]] * 11:02 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2056: repool after upgrade * 11:01 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93520 and previous config saved to /var/cache/conftool/dbconfig/20260602-105939-fceratto.json * 10:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93519 and previous config saved to /var/cache/conftool/dbconfig/20260602-105239-fceratto.json * 10:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 10:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance * 10:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93518 and previous config saved to /var/cache/conftool/dbconfig/20260602-105202-fceratto.json * 10:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2056.codfw.wmnet with OS trixie * 10:42 moritzm: installing busybox security updates * 10:42 claime: Enabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 10:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P93517 and previous config saved to /var/cache/conftool/dbconfig/20260602-104154-fceratto.json * 10:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P93516 and previous config saved to /var/cache/conftool/dbconfig/20260602-103146-fceratto.json * 10:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2056.codfw.wmnet with reason: host reimage * 10:27 claime: Disabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 10:25 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2056.codfw.wmnet with reason: host reimage * 10:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93515 and previous config saved to /var/cache/conftool/dbconfig/20260602-102139-fceratto.json * 10:09 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2056.codfw.wmnet with OS trixie * 10:08 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2056: Upgrading es2056.codfw.wmnet * 10:08 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2056: Upgrading es2056.codfw.wmnet * 10:08 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:06 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply * 10:06 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply * 09:56 claime: Enabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 09:46 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on cumin2003.codfw.wmnet with reason: in setup * 09:45 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1187: Pooling * 09:37 claime: Running puppet on cp6010 and cp6011 - [[phab:T422937|T422937]] * 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow2004.codfw.wmnet to plain * 09:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93511 and previous config saved to /var/cache/conftool/dbconfig/20260602-093716-fceratto.json * 09:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance * 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow2004.codfw.wmnet to plain * 09:34 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of rpki2003.codfw.wmnet to plain * 09:34 claime: Disabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 09:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of rpki2003.codfw.wmnet to plain * 09:32 moritzm: temporarily remove ganeti2045 from the codfw cluster [[phab:T427357|T427357]] * 09:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1055.eqiad.wmnet with OS trixie * 09:15 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1187: Pooling * 09:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1055.eqiad.wmnet with reason: host reimage * 09:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1187 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93508 and previous config saved to /var/cache/conftool/dbconfig/20260602-091126-fceratto.json * 09:09 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1055.eqiad.wmnet with reason: host reimage * 09:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1187 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93506 and previous config saved to /var/cache/conftool/dbconfig/20260602-090432-fceratto.json * 09:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1187.eqiad.wmnet with reason: Maintenance * 08:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2250.codfw.wmnet with reason: rack A3 maintenance * 08:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:56 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1055.eqiad.wmnet with OS trixie * 08:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:54 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:54 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:53 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 08:52 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 08:51 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 08:50 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 08:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 08:41 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:39 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:37 urbanecm: Reset user email of Barras@votewiki to the one of Barras@SUL * 08:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance * 08:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93505 and previous config saved to /var/cache/conftool/dbconfig/20260602-083033-fceratto.json * 08:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:29 slyngs: IDP, new configuration in preparation for webauthn * 08:20 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P93504 and previous config saved to /var/cache/conftool/dbconfig/20260602-082026-fceratto.json * 08:19 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 08:18 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 08:18 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:17 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296488{{!}}Revert "translate: adding separate read/write endpoints" (T425377)]] (duration: 03m 33s) * 08:16 atsuko@deploy1003: atsuko: Rolling back deployment * 08:16 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2053: repool after upgrade * 08:15 atsuko@deploy1003: atsuko: Backport for [[gerrit:1296488{{!}}Revert "translate: adding separate read/write endpoints" (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:13 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1296488{{!}}Revert "translate: adding separate read/write endpoints" (T425377)]] * 08:11 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:10 marostegui: Install mariadb 10.11.17 on es2053 [[phab:T427345|T427345]] * 08:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P93502 and previous config saved to /var/cache/conftool/dbconfig/20260602-081018-fceratto.json * 08:09 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:09 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: Depool for rack maintenance * 08:03 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296262{{!}}translate: fixing missed variable in credentials formatting closure (T425377)]] (duration: 14m 47s) * 08:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93499 and previous config saved to /var/cache/conftool/dbconfig/20260602-080011-fceratto.json * 07:59 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 07:59 atsuko@deploy1003: atsuko: Rolling back deployment * 07:58 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 07:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1181 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93498 and previous config saved to /var/cache/conftool/dbconfig/20260602-075759-fceratto.json * 07:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1181.eqiad.wmnet with reason: Maintenance * 07:57 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling * 07:50 atsuko@deploy1003: atsuko: Backport for [[gerrit:1296262{{!}}translate: fixing missed variable in credentials formatting closure (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:49 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1296262{{!}}translate: fixing missed variable in credentials formatting closure (T425377)]] * 07:48 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1181: Pooling * 07:47 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1181: Pooling * 07:44 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1181: Reboot * 07:43 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1181: Reboot * 07:42 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1181.eqiad.wmnet with reason: Reboot * 07:41 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 07:41 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:41 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1181: Migration of db1181.eqiad.wmnet completed * 07:40 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294949{{!}}translate: adding separate read/write endpoints (T425377)]] (duration: 21m 01s) * 07:39 atsuko@deploy1003: atsuko: Rolling back deployment * 07:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93490 and previous config saved to /var/cache/conftool/dbconfig/20260602-073904-fceratto.json * 07:32 XioNoX: pfw1-eqiad# delete protocols bgp group Production family inet6 - [[phab:T423384|T423384]] * 07:30 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2053: repool after upgrade * 07:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2158.codfw.wmnet with reason: rack A3 maintenance * 07:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93487 and previous config saved to /var/cache/conftool/dbconfig/20260602-072856-fceratto.json * 07:28 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2158: rack A3 maintenance * 07:28 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2158: rack A3 maintenance * 07:27 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on pc2021.codfw.wmnet with reason: rack A3 maintenance * 07:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc2021: rack A3 maintenance * 07:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 07:25 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache * 07:25 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool pc2021: rack A3 maintenance * 07:23 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2241: Depool for rack maintenance * 07:23 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2241.codfw.wmnet * 07:23 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2241.codfw.wmnet * 07:21 atsuko@deploy1003: atsuko: Backport for [[gerrit:1294949{{!}}translate: adding separate read/write endpoints (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:20 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2053.codfw.wmnet with OS trixie * 07:19 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1294949{{!}}translate: adding separate read/write endpoints (T425377)]] * 07:15 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2241.codfw.wmnet with reason: Depool for rack maintenance * 07:14 marostegui: Install mariadb 10.11.17 on db2186 [[phab:T427345|T427345]] * 07:12 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: Depool for rack maintenance * 07:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2186.codfw.wmnet with reason: upgrade * 07:12 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2241: Depool for rack maintenance * 07:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2053.codfw.wmnet with reason: host reimage * 06:59 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2053.codfw.wmnet with reason: host reimage * 06:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93478 and previous config saved to /var/cache/conftool/dbconfig/20260602-065533-fceratto.json * 06:55 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1181: Migration of db1181.eqiad.wmnet completed * 06:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 06:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1181.eqiad.wmnet with OS trixie * 06:43 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2053.codfw.wmnet with OS trixie * 06:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2053: Upgrading es2053.codfw.wmnet * 06:41 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2053: Upgrading es2053.codfw.wmnet * 06:41 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:37 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 06:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 06:36 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 06:36 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1052: repool after upgrade * 06:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1181.eqiad.wmnet with reason: host reimage * 06:24 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1181.eqiad.wmnet with reason: host reimage * 06:22 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 06:21 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 06:16 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 06:15 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 06:08 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1181.eqiad.wmnet with OS trixie * 06:05 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1181: Upgrading db1181.eqiad.wmnet * 06:05 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1181: Upgrading db1181.eqiad.wmnet * 06:04 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:02 marostegui@dns1004: END - running authdns-update * 06:01 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1181 [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93473 and previous config saved to /var/cache/conftool/dbconfig/20260602-060157-marostegui.json * 06:01 marostegui@dns1004: START - running authdns-update * 06:00 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db1236 to s7 primary and set section read-write [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93472 and previous config saved to /var/cache/conftool/dbconfig/20260602-060041-marostegui.json * 06:00 marostegui@cumin1003: dbctl commit (dc=all): 'Set s7 eqiad as read-only for maintenance - [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93471 and previous config saved to /var/cache/conftool/dbconfig/20260602-060018-marostegui.json * 06:00 marostegui: Starting s7 eqiad failover from db1181 to db1236 - [[phab:T426088|T426088]] * 05:51 marostegui@cumin1003: dbctl commit (dc=all): 'Set db1236 with weight 0 [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93470 and previous config saved to /var/cache/conftool/dbconfig/20260602-055153-marostegui.json * 05:51 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 [[phab:T426088|T426088]] * 05:50 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1052: repool after upgrade * 05:50 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 05:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1052.eqiad.wmnet with OS trixie * 05:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1052.eqiad.wmnet with reason: host reimage * 05:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:22 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1052.eqiad.wmnet with reason: host reimage * 05:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:07 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1052.eqiad.wmnet with OS trixie * 05:06 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1052: Upgrading es1052.eqiad.wmnet * 05:06 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1052: Upgrading es1052.eqiad.wmnet * 05:05 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 04:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 04:49 ryankemper: [[phab:T425007|T425007]] (k8s) created 4 wdqs namespaces on `dse-k8s-codfw`'s `admin_ng` ns: `wdqs-[internal,external]` & `wdqs-[internal,external]-next`; certs issued * 04:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 04:40 ryankemper@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 04:36 ryankemper@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 04:05 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.2 (duration: 05m 33s) == 2026-06-01 == * 23:27 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295963{{!}}Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962{{!}}Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] (duration: 07m 17s) * 23:23 jdlrobson@deploy1003: mfossati, jdlrobson: Continuing with deployment * 23:22 jdlrobson@deploy1003: mfossati, jdlrobson: Backport for [[gerrit:1295963{{!}}Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962{{!}}Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:20 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1295963{{!}}Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962{{!}}Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] * 23:15 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296022{{!}}Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028{{!}}styles: Limit selector to badge client pref (T427407)]] (duration: 09m 33s) * 23:11 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 23:07 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1296022{{!}}Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028{{!}}styles: Limit selector to badge client pref (T427407)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:06 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1296022{{!}}Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028{{!}}styles: Limit selector to badge client pref (T427407)]] * 23:04 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6015.* * 22:36 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296024{{!}}Add maintenance script to scrape SVG render files]] (duration: 06m 22s) * 22:32 reedy@deploy1003: reedy: Continuing with deployment * 22:31 reedy@deploy1003: reedy: Backport for [[gerrit:1296024{{!}}Add maintenance script to scrape SVG render files]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:30 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1296024{{!}}Add maintenance script to scrape SVG render files]] * 22:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 22:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 22:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 21:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 21:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 21:51 sbassett: Deployed updated mitigation for [[phab:T326691|T326691]] * 21:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 21:35 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 21:35 maryum: Deployed security fix for [[phab:T427611|T427611]] * 21:35 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 21:33 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 21:32 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 21:27 maryum: Deployed security fix for [[phab:T427235|T427235]] * 21:13 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296002{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009{{!}}Redirect Special:AccountRecovery to the shared domain (T427692)]] (duration: 09m 20s) * 21:09 catrope@deploy1003: catrope, arlolra: Continuing with deployment * 21:09 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 21:09 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 21:08 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 21:07 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 21:07 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 21:06 catrope@deploy1003: catrope, arlolra: Backport for [[gerrit:1296002{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009{{!}}Redirect Special:AccountRecovery to the shared domain (T427692)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:04 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1296002{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009{{!}}Redirect Special:AccountRecovery to the shared domain (T427692)]] * 20:53 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 20:37 ryankemper@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wdqs1015.eqiad.wmnet with reason: [[phab:T427852|T427852]] hw failure * 20:26 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1285412{{!}}Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531{{!}}Enable AbuseFilter block action on nlwiki (T427384)]] (duration: 07m 48s) * 20:22 catrope@deploy1003: sfaci, xxblackburnxx, catrope: Continuing with deployment * 20:20 catrope@deploy1003: sfaci, xxblackburnxx, catrope: Backport for [[gerrit:1285412{{!}}Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531{{!}}Enable AbuseFilter block action on nlwiki (T427384)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:18 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1285412{{!}}Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531{{!}}Enable AbuseFilter block action on nlwiki (T427384)]] * 20:12 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295504{{!}}passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] (duration: 07m 37s) * 20:08 catrope@deploy1003: catrope: Continuing with deployment * 20:07 catrope@deploy1003: catrope: Backport for [[gerrit:1295504{{!}}passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:05 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1295504{{!}}passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] * 19:48 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 19:47 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 19:47 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 19:46 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 19:46 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 19:45 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 19:01 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: sync * 19:00 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: sync * 18:24 otto@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295950{{!}}mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] (duration: 06m 42s) * 18:20 otto@deploy1003: otto: Continuing with deployment * 18:19 otto@deploy1003: otto: Backport for [[gerrit:1295950{{!}}mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:17 otto@deploy1003: Started scap sync-world: Backport for [[gerrit:1295950{{!}}mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] * 18:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 18:05 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 18:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd2001.codfw.wmnet to plain * 18:02 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 18:02 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd2001.codfw.wmnet to plain * 18:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2003.codfw.wmnet to plain * 18:01 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply * 18:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2003.codfw.wmnet to plain * 17:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 17:58 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 17:53 jasmine@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2006.codfw.wmnet with OS trixie * 17:42 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295976{{!}}nlwiki: change to Wikipedia 25 logo (T424519)]] (duration: 07m 29s) * 17:37 samtar@deploy1003: chlod, samtar: Continuing with deployment * 17:36 samtar@deploy1003: chlod, samtar: Backport for [[gerrit:1295976{{!}}nlwiki: change to Wikipedia 25 logo (T424519)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:34 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1295976{{!}}nlwiki: change to Wikipedia 25 logo (T424519)]] * 17:20 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1236: Update * 17:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd2001.codfw.wmnet to drbd * 17:04 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling * 17:04 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 17:04 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1180: Pooling * 17:03 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 17:03 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling * 17:03 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 16:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd2001.codfw.wmnet to drbd * 16:58 Amir1: drop flaggedrevs tables on wikinews wikis ([[phab:T423577|T423577]]) * 16:57 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 16:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93462 and previous config saved to /var/cache/conftool/dbconfig/20260601-165717-fceratto.json * 16:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93460 and previous config saved to /var/cache/conftool/dbconfig/20260601-164709-fceratto.json * 16:42 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling * 16:37 ryankemper@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,cluster=wdqs-main,service=wdqs-main,name=wdqs1015.eqiad.wmnet * 16:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93458 and previous config saved to /var/cache/conftool/dbconfig/20260601-163701-fceratto.json * 16:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:35 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1236.eqiad.wmnet * 16:35 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1236.eqiad.wmnet * 16:35 ryankemper@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,cluster=wdqs,service=wdqs-main,name=wdqs1015.eqiad.wmnet * 16:34 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update * 16:34 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1236: Update * 16:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1236.eqiad.wmnet with reason: Kernel update [[phab:T426633|T426633]] * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:30 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1236.eqiad.wmnet * 16:30 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1236.eqiad.wmnet * 16:30 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update * 16:29 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1236: Update * 16:29 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update * 16:29 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2003.codfw.wmnet to drbd * 16:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93455 and previous config saved to /var/cache/conftool/dbconfig/20260601-162653-fceratto.json * 16:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 16:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1209: Migration of db1209.eqiad.wmnet completed * 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1236.eqiad.wmnet with reason: Kernel update [[phab:T426633|T426633]] * 16:09 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1236: Update * 16:09 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1236: Update * 16:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:06 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 16:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2003.codfw.wmnet to drbd * 16:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 16:03 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 16:02 moritzm: temporarily remove ganeti2027 from the codfw cluster [[phab:T427357|T427357]] * 15:56 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:56 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.depool (exit_code=97) depool db1224: Pooling * 15:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2005.codfw.wmnet with OS bullseye * 15:53 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1224: Pooling * 15:51 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 15:49 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling * 15:49 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:48 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet * 15:45 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling * 15:44 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2005.codfw.wmnet with reason: host reimage * 15:40 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:40 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling * 15:40 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet * 15:40 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet * 15:40 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet * 15:40 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet * 15:39 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 15:39 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 15:39 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1209: Migration of db1209.eqiad.wmnet completed * 15:39 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:38 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:38 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling * 15:37 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2005.codfw.wmnet with reason: host reimage * 15:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 15:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1209.eqiad.wmnet with OS trixie * 15:28 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295802{{!}}hCaptcha: Raise SiteVerify error threshold to 100]] (duration: 06m 15s) * 15:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93446 and previous config saved to /var/cache/conftool/dbconfig/20260601-152638-fceratto.json * 15:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 15:26 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:25 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet * 15:25 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet * 15:25 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling * 15:25 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:24 kharlan@deploy1003: kharlan: Continuing with deployment * 15:24 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295802{{!}}hCaptcha: Raise SiteVerify error threshold to 100]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:22 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host testvm2005.codfw.wmnet with OS bullseye * 15:22 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295802{{!}}hCaptcha: Raise SiteVerify error threshold to 100]] * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:20 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295946{{!}}hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] (duration: 08m 24s) * 15:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:16 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 15:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1209.eqiad.wmnet with reason: host reimage * 15:14 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1295946{{!}}hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:13 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:12 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1295946{{!}}hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] * 15:10 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1209.eqiad.wmnet with reason: host reimage * 15:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93445 and previous config saved to /var/cache/conftool/dbconfig/20260601-151024-fceratto.json * 15:08 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-reboot (exit_code=0) rolling reboot on A:sessionstore * 15:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93443 and previous config saved to /var/cache/conftool/dbconfig/20260601-150017-fceratto.json * 14:55 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1209.eqiad.wmnet with OS trixie * 14:52 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 14:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1209: Upgrading db1209.eqiad.wmnet * 14:52 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 14:52 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1209: Upgrading db1209.eqiad.wmnet * 14:52 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 14:51 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:51 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 14:50 atsuko@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 14:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93441 and previous config saved to /var/cache/conftool/dbconfig/20260601-145010-fceratto.json * 14:49 atsuko@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 14:49 atsuko@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 14:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:42 atsuko@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 14:41 atsuko@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93440 and previous config saved to /var/cache/conftool/dbconfig/20260601-144002-fceratto.json * 14:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:30 ladsgroup@deploy1003: Synchronized portals: Deploy portals ([[phab:T421797|T421797]]) (duration: 02m 43s) * 14:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:27 ladsgroup@deploy1003: Synchronized portals/wikipedia.org/assets: Deploy portals ([[phab:T421797|T421797]]) (duration: 06m 10s) * 14:25 sukhe@dns1004: END - running authdns-update * 14:23 sukhe@dns1004: START - running authdns-update * 14:22 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 14:21 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:16 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:12 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:12 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:11 Lucas_WMDE: UTC afternoon backport+config window done * 14:10 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295918{{!}}Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] (duration: 11m 06s) * 14:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:05 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, codenamenoreste: Continuing with deployment * 14:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, codenamenoreste: Backport for [[gerrit:1295918{{!}}Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:01 eevans@cumin1003: START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:sessionstore * 13:58 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1295918{{!}}Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] * 13:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:52 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1265.eqiad.wmnet with OS trixie * 13:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93439 and previous config saved to /var/cache/conftool/dbconfig/20260601-133947-fceratto.json * 13:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 13:37 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage * 13:35 atsukoito: restarted pybal.service on lvs2013 * 13:31 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage * 13:31 atsukoito: restarted pybal.service on lvs2014 * 13:24 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs-test2001.codfw.wmnet * 13:24 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:22 atsukoito: restarted pybal.service on lvs1019 * 13:22 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in eqiad/ml-serve-eqiad: maintenance * 13:21 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in eqiad/ml-serve-eqiad: maintenance * 13:20 atsukoito: restarted pybal.service on lvs1020 * 13:20 Msz2001: UTC afternoon backpot+config window done * 13:20 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295875{{!}}Add SetGlobalPreference maintenance script (T427476)]] (duration: 06m 22s) * 13:19 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs-test2001.codfw.wmnet * 13:18 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1265.eqiad.wmnet with OS trixie * 13:18 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:16 mszwarc@deploy1003: mszwarc: Continuing with deployment * 13:15 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1295875{{!}}Add SetGlobalPreference maintenance script (T427476)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:14 atsukoito: sudo cumin 'A:lvs-low-traffic-eqiad' 'systemctl restart pybal.service' * 13:14 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1295875{{!}}Add SetGlobalPreference maintenance script (T427476)]] * 13:12 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295536{{!}}swwiki: Enable the Visual Editor on the project namespace (T427117)]] (duration: 10m 06s) * 13:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93438 and previous config saved to /var/cache/conftool/dbconfig/20260601-130949-fceratto.json * 13:08 mszwarc@deploy1003: codenamenoreste, mszwarc: Continuing with deployment * 13:07 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 13:06 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . * 13:05 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 13:04 mszwarc@deploy1003: codenamenoreste, mszwarc: Backport for [[gerrit:1295536{{!}}swwiki: Enable the Visual Editor on the project namespace (T427117)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:04 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 13:04 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 13:03 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 13:02 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1295536{{!}}swwiki: Enable the Visual Editor on the project namespace (T427117)]] * 12:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93437 and previous config saved to /var/cache/conftool/dbconfig/20260601-125941-fceratto.json * 12:56 dpogorzelski@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=inference,name=eqiad * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'readability' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'logo-detection' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'edit-check' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . * 12:52 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:50 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:49 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93436 and previous config saved to /var/cache/conftool/dbconfig/20260601-124934-fceratto.json * 12:48 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:47 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:46 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:44 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:43 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:42 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:41 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93435 and previous config saved to /var/cache/conftool/dbconfig/20260601-123926-fceratto.json * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:29 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:28 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 12:28 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:27 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:27 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2005.codfw.wmnet to plain * 12:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2005.codfw.wmnet to plain * 12:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet * 12:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 12:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2005.codfw.wmnet to drbd * 12:20 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:17 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:15 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in eqiad/ml-serve-eqiad: maintenance * 12:15 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in eqiad/ml-serve-eqiad: maintenance * 12:11 dpogorzelski@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=inference,name=eqiad * 12:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2005.codfw.wmnet to drbd * 12:05 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet * 12:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 12:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2027.codfw.wmnet * 12:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 11:59 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in eqiad/ml-serve-eqiad: maintenance * 11:59 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in eqiad/ml-serve-eqiad: maintenance * 11:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93434 and previous config saved to /var/cache/conftool/dbconfig/20260601-113911-fceratto.json * 11:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 11:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93433 and previous config saved to /var/cache/conftool/dbconfig/20260601-113843-fceratto.json * 11:37 moritzm: installing Exim security updates * 11:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:32 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:32 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:32 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93432 and previous config saved to /var/cache/conftool/dbconfig/20260601-112835-fceratto.json * 11:25 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 11:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:22 moritzm: installing imagemagick security updates * 11:22 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:22 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:22 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 11:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93430 and previous config saved to /var/cache/conftool/dbconfig/20260601-111827-fceratto.json * 11:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:14 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 11:12 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 11:10 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93429 and previous config saved to /var/cache/conftool/dbconfig/20260601-110820-fceratto.json * 11:04 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 11:01 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1055: repool after upgrade * 11:01 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93427 and previous config saved to /var/cache/conftool/dbconfig/20260601-110121-fceratto.json * 11:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance * 10:54 marostegui@dns1004: END - running authdns-update * 10:52 marostegui@dns1004: START - running authdns-update * 10:48 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1050 to es1 eqiad primary [[phab:T427032|T427032]]', diff saved to https://phabricator.wikimedia.org/P93425 and previous config saved to /var/cache/conftool/dbconfig/20260601-104837-marostegui.json * 10:47 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2055 to es1 codfw primary [[phab:T427032|T427032]]', diff saved to https://phabricator.wikimedia.org/P93424 and previous config saved to /var/cache/conftool/dbconfig/20260601-104739-marostegui.json * 10:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1177: Migration of db1177.eqiad.wmnet completed * 10:40 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet * 10:34 kamila@cumin1003: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet * 10:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93421 and previous config saved to /var/cache/conftool/dbconfig/20260601-103316-fceratto.json * 10:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93418 and previous config saved to /var/cache/conftool/dbconfig/20260601-102308-fceratto.json * 10:16 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1055: repool after upgrade * 10:15 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1055.eqiad.wmnet with OS trixie * 10:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93415 and previous config saved to /var/cache/conftool/dbconfig/20260601-101300-fceratto.json * 10:09 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 10:07 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 10:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93414 and previous config saved to /var/cache/conftool/dbconfig/20260601-100252-fceratto.json * 10:00 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1177: Migration of db1177.eqiad.wmnet completed * 09:58 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1055.eqiad.wmnet with reason: host reimage * 09:56 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 09:54 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 09:53 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1055.eqiad.wmnet with reason: host reimage * 09:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1177.eqiad.wmnet with OS trixie * 09:51 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 09:50 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 09:39 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1055.eqiad.wmnet with OS trixie * 09:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1055: Upgrading es1055.eqiad.wmnet * 09:38 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1055: Upgrading es1055.eqiad.wmnet * 09:37 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1177.eqiad.wmnet with reason: host reimage * 09:31 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1177.eqiad.wmnet with reason: host reimage * 09:17 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1177.eqiad.wmnet with OS trixie * 09:15 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 09:14 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 09:13 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 09:12 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 09:12 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1177: Upgrading db1177.eqiad.wmnet * 09:11 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1177: Upgrading db1177.eqiad.wmnet * 09:11 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93410 and previous config saved to /var/cache/conftool/dbconfig/20260601-090237-fceratto.json * 09:02 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93409 and previous config saved to /var/cache/conftool/dbconfig/20260601-090209-fceratto.json * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P93408 and previous config saved to /var/cache/conftool/dbconfig/20260601-085202-fceratto.json * 08:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P93407 and previous config saved to /var/cache/conftool/dbconfig/20260601-084154-fceratto.json * 08:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93406 and previous config saved to /var/cache/conftool/dbconfig/20260601-083146-fceratto.json * 08:24 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93405 and previous config saved to /var/cache/conftool/dbconfig/20260601-082442-fceratto.json * 08:24 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1168.eqiad.wmnet with reason: Maintenance * 07:58 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295454{{!}}Disable the creation of synthetic main refs in production (T427484)]] (duration: 11m 26s) * 07:56 XioNoX: add no_p2p term to pfw1-codfw BGP_fundraising_export - [[phab:T423384|T423384]] * 07:52 wmde-fisch@deploy1003: lilients, wmde-fisch: Continuing with deployment * 07:51 wmde-fisch@deploy1003: lilients, wmde-fisch: Backport for [[gerrit:1295454{{!}}Disable the creation of synthetic main refs in production (T427484)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:47 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1295454{{!}}Disable the creation of synthetic main refs in production (T427484)]] * 07:45 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294826{{!}}Update VE core submodule to master (9cf5524e7) (T424232)]] (duration: 31m 34s) * 07:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 07:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 07:32 wmde-fisch@deploy1003: wmde-fisch: Continuing with deployment * 07:31 wmde-fisch@deploy1003: wmde-fisch: Backport for [[gerrit:1294826{{!}}Update VE core submodule to master (9cf5524e7) (T424232)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet * 07:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet * 07:13 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1294826{{!}}Update VE core submodule to master (9cf5524e7) (T424232)]] * 06:48 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 06:47 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. == 2026-05-31 == * 02:06 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 30s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-30 == * 16:21 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:21 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:21 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:21 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 06:39 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 06:39 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 06:39 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 06:38 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 27s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-29 == * 23:39 aokoth@cumin1003: END (PASS) - Cookbook sre.vrts.upgrade (exit_code=0) on VRTS host vrts1003.eqiad.wmnet * 23:37 aokoth@cumin1003: START - Cookbook sre.vrts.upgrade on VRTS host vrts1003.eqiad.wmnet * 21:42 catrope@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 21:41 catrope@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 17:40 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295487{{!}}Hide experiment if not active and no assigned group]] (duration: 06m 54s) * 17:35 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 17:34 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1295487{{!}}Hide experiment if not active and no assigned group]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:33 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1295487{{!}}Hide experiment if not active and no assigned group]] * 16:30 jgreen@dns1004: END - running authdns-update * 16:28 jgreen@dns1004: START - running authdns-update * 16:13 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:12 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 15:28 dancy@deploy1003: Installation of scap version "4.267.0" completed for 2 hosts * 15:26 dancy@deploy1003: Installing scap version "4.267.0" for 2 host(s) * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:15 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295466{{!}}GlobalPreferencesHandler: Cast auto-reveal expiry to int (T427625)]] (duration: 07m 58s) * 14:11 kharlan@deploy1003: kharlan: Continuing with deployment * 14:09 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295466{{!}}GlobalPreferencesHandler: Cast auto-reveal expiry to int (T427625)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:07 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295466{{!}}GlobalPreferencesHandler: Cast auto-reveal expiry to int (T427625)]] * 13:53 moritzm: imported OpenJDK 21 21.0.11+10-1~deb12u1 to component/jdk21 (backport of latest Java 21 security release for Bookworm) * 12:09 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader1006.wikimedia.org * 12:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1006.wikimedia.org with OS trixie * 11:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1006.wikimedia.org with reason: host reimage * 11:47 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1006.wikimedia.org with reason: host reimage * 11:36 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader1006.wikimedia.org with OS trixie * 11:15 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:15 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:13 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader1006.wikimedia.org on all recursors * 11:12 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader1006.wikimedia.org on all recursors * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:06 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:00 jmm@cumin2002: START - Cookbook sre.dns.netbox * 11:00 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader1006.wikimedia.org * 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader1005.wikimedia.org * 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1005.wikimedia.org with OS trixie * 10:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1005.wikimedia.org with reason: host reimage * 10:40 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2212: Pooling * 10:37 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1005.wikimedia.org with reason: host reimage * 10:27 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader1005.wikimedia.org with OS trixie * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 10:01 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:59 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:55 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 09:50 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 09:49 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:45 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:44 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup2014.codfw.wmnet with OS bookworm * 09:33 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:20 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup2014.codfw.wmnet with reason: host reimage * 09:12 jynus@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on backup2014.codfw.wmnet with reason: host reimage * 09:10 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 09:10 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 09:03 jelto@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM etherpad2002.codfw.wmnet * 08:59 jelto@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM etherpad2002.codfw.wmnet * 08:59 jelto: gnt-instance modify -B memory=4g,vcpus=1 etherpad2002.codfw.wmnet - [[phab:T427588|T427588]] * 08:54 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2014.codfw.wmnet with OS bookworm * 08:51 jelto@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM etherpad1004.eqiad.wmnet * 08:50 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams-internal: apply * 08:50 jynus@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host backup2014.codfw.wmnet with OS bookworm * 08:49 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams-internal: apply * 08:47 jelto@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM etherpad1004.eqiad.wmnet * 08:46 jelto: gnt-instance modify -B memory=4g,vcpus=1 etherpad1004.eqiad.wmnet - [[phab:T427588|T427588]] * 08:42 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2212: Pooling * 08:42 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 08:39 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2212: Pooling * 08:39 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 08:38 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams-internal: apply * 08:37 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams-internal: apply * 08:37 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams-internal: apply * 08:36 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams-internal: apply * 08:33 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2014.codfw.wmnet with OS bookworm * 08:31 jynus@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host backup2014.codfw.wmnet with OS bookworm * 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader1005.wikimedia.org on all recursors * 08:21 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader1005.wikimedia.org on all recursors * 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 08:21 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 08:18 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 08:17 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 08:16 jmm@cumin2002: START - Cookbook sre.dns.netbox * 08:16 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader1005.wikimedia.org * 08:05 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2212: Pooling * 07:59 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 07:59 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 07:54 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 07:54 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2212.codfw.wmnet * 07:54 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2212.codfw.wmnet * 07:22 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2014.codfw.wmnet with OS bookworm * 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader2006.wikimedia.org * 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader2006.wikimedia.org with OS trixie * 06:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader2006.wikimedia.org with reason: host reimage * 06:53 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader2006.wikimedia.org with reason: host reimage * 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader2006.wikimedia.org with OS trixie * 06:32 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:32 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader2006.wikimedia.org on all recursors * 06:31 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader2006.wikimedia.org on all recursors * 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:31 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:27 jmm@cumin2002: START - Cookbook sre.dns.netbox * 06:27 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader2006.wikimedia.org * 03:01 vriley@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts db1224.eqiad.wmnet * 03:00 vriley@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1224.eqiad.wmnet * 03:00 vriley@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1224.eqiad.wmnet * 02:56 vriley@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1224.eqiad.wmnet * 01:47 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5032.eqsin.wmnet with OS trixie * 01:18 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5032.eqsin.wmnet with reason: host reimage * 01:14 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5032.eqsin.wmnet with reason: host reimage * 00:31 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cp5032.eqsin.wmnet with OS trixie * 00:29 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host cp5032.eqsin.wmnet * 00:23 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 00:22 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply * 00:21 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 00:21 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply == 2026-05-28 == * 23:07 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 23:07 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new ae1.522 interface - pt1979@cumin2002" * 23:07 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new ae1.522 interface - pt1979@cumin2002" * 23:02 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 22:34 andrewbogott: reprepro includedeb trixie-wikimedia /home/andrew/magnum-cluster-api_0.36.6-1~wmf13u2_amd64.deb * 22:31 logmsgbot: dreamyjazz Deployed security patch for [[phab:T426388|T426388]] * 21:33 maryum: Deployed security fix for [[phab:T426867|T426867]] * 21:21 alexsanford: Deployed security fix for [[phab:T426889|T426889]] * 21:07 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host cp5032.eqsin.wmnet * 21:04 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "setup new eqsin vlan - pt1979@cumin2002 - [[phab:T427393|T427393]]" * 21:04 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "setup new eqsin vlan - pt1979@cumin2002 - [[phab:T427393|T427393]]" * 20:48 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295066{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T420336 T427098 T427354 T427082)]], [[gerrit:1295067{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T427082)]] (duration: 07m 34s) * 20:44 arlolra@deploy1003: arlolra: Continuing with deployment * 20:43 arlolra@deploy1003: arlolra: Backport for [[gerrit:1295066{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T420336 T427098 T427354 T427082)]], [[gerrit:1295067{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T427082)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:41 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1295066{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T420336 T427098 T427354 T427082)]], [[gerrit:1295067{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T427082)]] * 20:34 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293805{{!}}Deploy PRV to 7 wikis (T427331)]] (duration: 07m 20s) * 20:30 arlolra@deploy1003: arlolra: Continuing with deployment * 20:29 arlolra@deploy1003: arlolra: Backport for [[gerrit:1293805{{!}}Deploy PRV to 7 wikis (T427331)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:27 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1293805{{!}}Deploy PRV to 7 wikis (T427331)]] * 20:22 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291996{{!}}Replace deprecated Hooks::getInstance (T426981)]], [[gerrit:1294393{{!}}Permissions: Create wmf-officeit group on officewiki]], [[gerrit:1294229{{!}}Deploy IRS Direct Reporting feature to enwiki (T427369)]], [[gerrit:1295039{{!}}Add 2FA enforcement demotion config for phase 2 groups (T423119)]] (duration: 09m 07s) * 20:18 stran@deploy1003: alexsanford, stran, catrope, dreamyjazz: Continuing with deployment * 20:14 stran@deploy1003: alexsanford, stran, catrope, dreamyjazz: Backport for [[gerrit:1291996{{!}}Replace deprecated Hooks::getInstance (T426981)]], [[gerrit:1294393{{!}}Permissions: Create wmf-officeit group on officewiki]], [[gerrit:1294229{{!}}Deploy IRS Direct Reporting feature to enwiki (T427369)]], [[gerrit:1295039{{!}}Add 2FA enforcement demotion config for phase 2 groups (T423119)]] synced to the testservers (see https://wikitech. * 20:13 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5032.eqsin.wmnet with OS trixie * 20:13 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1291996{{!}}Replace deprecated Hooks::getInstance (T426981)]], [[gerrit:1294393{{!}}Permissions: Create wmf-officeit group on officewiki]], [[gerrit:1294229{{!}}Deploy IRS Direct Reporting feature to enwiki (T427369)]], [[gerrit:1295039{{!}}Add 2FA enforcement demotion config for phase 2 groups (T423119)]] * 19:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1018.eqiad.wmnet * 19:27 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1018.eqiad.wmnet * 19:09 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1018.eqiad.wmnet with reason: Kernel reboot * 19:09 brett: Stopping pybal/puppet/downtiming lvs1018.eqiad.wmnet for reboot * 19:05 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1019.eqiad.wmnet * 19:05 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1019.eqiad.wmnet * 18:52 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cp5032.eqsin.wmnet with OS trixie * 18:51 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:51 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change cp5032 IP - pt1979@cumin2002" * 18:51 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change cp5032 IP - pt1979@cumin2002" * 18:47 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 18:40 mutante: planet1003/planet2003 - apt-get upgrade - all pending package upgrades * 18:35 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: Kernel reboot * 18:34 brett: Stopping pybal/puppet/downtiming lvs1019.eqiad.wmnet for reboot and BIOS update/memory self-healing - [[phab:T426109|T426109]] * 18:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs2011.codfw.wmnet * 18:25 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs2011.codfw.wmnet * 18:19 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2011.codfw.wmnet with reason: Kernel reboot * 18:19 brett: Stopping pybal/puppet/downtiming lvs2011.codfw.wmnet for reboot * 18:09 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs2013.codfw.wmnet * 18:06 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs2013.codfw.wmnet * 18:00 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2013.codfw.wmnet with reason: Kernel reboot * 17:57 brett: Stopping pybal/puppet/downtiming lvs2013.codfw.wmnet for reboot * 17:19 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 16:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93393 and previous config saved to /var/cache/conftool/dbconfig/20260528-164514-fceratto.json * 16:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P93392 and previous config saved to /var/cache/conftool/dbconfig/20260528-163507-fceratto.json * 16:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P93391 and previous config saved to /var/cache/conftool/dbconfig/20260528-162459-fceratto.json * 16:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db1224.eqiad.wmnet with reason: unreachable [[phab:T427535|T427535]] * 16:17 swfrench-wmf: reprepro include xdebug_3.4.4-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:17 swfrench-wmf: reprepro include wikidiff2_1.14.1-2+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:17 swfrench-wmf: reprepro include php-yaml_2.2.4-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-xhprof_2.3.10-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-wmerrors_2.0.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-uuid_1.3.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-redis_6.2.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 swfrench-wmf: reprepro include php-pcov_1.0.12-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 swfrench-wmf: reprepro include php-memcached_3.3.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 16:15 swfrench-wmf: reprepro include php-luasandbox_4.1.2-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 16:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93390 and previous config saved to /var/cache/conftool/dbconfig/20260528-161452-fceratto.json * 16:14 swfrench-wmf: reprepro include php-imagick_3.7.0-13+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:14 swfrench-wmf: reprepro include php-excimer_1.2.5-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:09 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:09 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:06 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1251 ([[phab:T426633|T426633]])', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20260528-160646-fceratto.json * 16:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1251.eqiad.wmnet with reason: Maintenance * 16:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93388 and previous config saved to /var/cache/conftool/dbconfig/20260528-160613-fceratto.json * 15:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P93387 and previous config saved to /var/cache/conftool/dbconfig/20260528-155605-fceratto.json * 15:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P93386 and previous config saved to /var/cache/conftool/dbconfig/20260528-154557-fceratto.json * 15:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93385 and previous config saved to /var/cache/conftool/dbconfig/20260528-153550-fceratto.json * 15:27 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1235 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93384 and previous config saved to /var/cache/conftool/dbconfig/20260528-152736-fceratto.json * 15:27 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1235.eqiad.wmnet with reason: Maintenance * 15:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93383 and previous config saved to /var/cache/conftool/dbconfig/20260528-152708-fceratto.json * 15:20 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp5032.eqsin.wmnet with reason: Testing reimaging on new subnet * 15:18 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5032.* * 15:17 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P93382 and previous config saved to /var/cache/conftool/dbconfig/20260528-151701-fceratto.json * 15:17 jhathaway: dmarc ingress test on mx-in1001 * 15:14 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:14 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P93381 and previous config saved to /var/cache/conftool/dbconfig/20260528-150653-fceratto.json * 14:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93380 and previous config saved to /var/cache/conftool/dbconfig/20260528-145646-fceratto.json * 14:56 moritzm: installing nginx security updates * 14:49 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1234 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93379 and previous config saved to /var/cache/conftool/dbconfig/20260528-144936-fceratto.json * 14:49 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 14:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1234.eqiad.wmnet with reason: Maintenance * 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93378 and previous config saved to /var/cache/conftool/dbconfig/20260528-144909-fceratto.json * 14:48 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader2005.wikimedia.org * 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader2005.wikimedia.org with OS trixie * 14:47 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 14:39 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2189.codfw.wmnet * 14:39 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2189.codfw.wmnet * 14:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P93377 and previous config saved to /var/cache/conftool/dbconfig/20260528-143901-fceratto.json * 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader2005.wikimedia.org with reason: host reimage * 14:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P93376 and previous config saved to /var/cache/conftool/dbconfig/20260528-142854-fceratto.json * 14:28 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:28 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader2005.wikimedia.org with reason: host reimage * 14:27 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:19 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294998{{!}}ImageContentLookup: Fix issue created by strict types (T427505)]], [[gerrit:1295001{{!}}Enable hCaptcha for VisualEditor in group 1 (T425940)]] (duration: 11m 29s) * 14:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93375 and previous config saved to /var/cache/conftool/dbconfig/20260528-141846-fceratto.json * 14:15 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1232 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93374 and previous config saved to /var/cache/conftool/dbconfig/20260528-141029-fceratto.json * 14:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1232.eqiad.wmnet with reason: Maintenance * 14:10 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader2005.wikimedia.org with OS trixie * 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93373 and previous config saved to /var/cache/conftool/dbconfig/20260528-141001-fceratto.json * 14:09 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1294998{{!}}ImageContentLookup: Fix issue created by strict types (T427505)]], [[gerrit:1295001{{!}}Enable hCaptcha for VisualEditor in group 1 (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:08 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294998{{!}}ImageContentLookup: Fix issue created by strict types (T427505)]], [[gerrit:1295001{{!}}Enable hCaptcha for VisualEditor in group 1 (T425940)]] * 14:00 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down * 13:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P93371 and previous config saved to /var/cache/conftool/dbconfig/20260528-135951-fceratto.json * 13:58 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp6015.drmrs.wmnet,service=(cdn{{!}}ats-be) * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:55 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader2005.wikimedia.org on all recursors * 13:55 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader2005.wikimedia.org on all recursors * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:55 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P93370 and previous config saved to /var/cache/conftool/dbconfig/20260528-134944-fceratto.json * 13:40 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 13:40 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 13:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93369 and previous config saved to /var/cache/conftool/dbconfig/20260528-133936-fceratto.json * 13:39 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 13:38 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 13:36 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294986{{!}}Image Carousel: check candidate pages (T427336)]] (duration: 06m 40s) * 13:34 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 13:33 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 13:32 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93368 and previous config saved to /var/cache/conftool/dbconfig/20260528-133230-fceratto.json * 13:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance * 13:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93367 and previous config saved to /var/cache/conftool/dbconfig/20260528-133202-fceratto.json * 13:31 mlitn@deploy1003: mlitn: Continuing with deployment * 13:31 mlitn@deploy1003: mlitn: Backport for [[gerrit:1294986{{!}}Image Carousel: check candidate pages (T427336)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:29 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1294986{{!}}Image Carousel: check candidate pages (T427336)]] * 13:22 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 13:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P93366 and previous config saved to /var/cache/conftool/dbconfig/20260528-132155-fceratto.json * 13:21 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 13:17 elukey: clean up a lof ot stale Kafka ACLs on Kafka Jumbo - Details in [[phab:T425528|T425528]] * 13:14 jmm@cumin2002: START - Cookbook sre.dns.netbox * 13:14 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader2005.wikimedia.org * 13:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P93365 and previous config saved to /var/cache/conftool/dbconfig/20260528-131147-fceratto.json * 13:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93364 and previous config saved to /var/cache/conftool/dbconfig/20260528-130139-fceratto.json * 12:54 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1218 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93363 and previous config saved to /var/cache/conftool/dbconfig/20260528-125439-fceratto.json * 12:54 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1218.eqiad.wmnet with reason: Maintenance * 12:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93362 and previous config saved to /var/cache/conftool/dbconfig/20260528-125412-fceratto.json * 12:48 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 12:48 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 12:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P93361 and previous config saved to /var/cache/conftool/dbconfig/20260528-124404-fceratto.json * 12:44 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 12:43 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 12:39 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 12:38 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 12:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P93360 and previous config saved to /var/cache/conftool/dbconfig/20260528-123357-fceratto.json * 12:25 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest1006.eqiad.wmnet with OS trixie * 12:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93359 and previous config saved to /var/cache/conftool/dbconfig/20260528-122349-fceratto.json * 12:15 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93358 and previous config saved to /var/cache/conftool/dbconfig/20260528-121551-fceratto.json * 12:15 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance * 12:15 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie * 12:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93357 and previous config saved to /var/cache/conftool/dbconfig/20260528-121523-fceratto.json * 12:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P93356 and previous config saved to /var/cache/conftool/dbconfig/20260528-120515-fceratto.json * 12:02 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest1006.eqiad.wmnet with OS trixie * 12:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 12:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 12:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 12:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 11:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P93355 and previous config saved to /var/cache/conftool/dbconfig/20260528-115508-fceratto.json * 11:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93354 and previous config saved to /var/cache/conftool/dbconfig/20260528-114500-fceratto.json * 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93353 and previous config saved to /var/cache/conftool/dbconfig/20260528-113635-fceratto.json * 11:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 11:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance * 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93352 and previous config saved to /var/cache/conftool/dbconfig/20260528-113559-fceratto.json * 11:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P93351 and previous config saved to /var/cache/conftool/dbconfig/20260528-112551-fceratto.json * 11:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P93350 and previous config saved to /var/cache/conftool/dbconfig/20260528-111543-fceratto.json * 11:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93349 and previous config saved to /var/cache/conftool/dbconfig/20260528-110536-fceratto.json * 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1195 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93348 and previous config saved to /var/cache/conftool/dbconfig/20260528-105820-fceratto.json * 10:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie * 10:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1195.eqiad.wmnet with reason: Maintenance * 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93347 and previous config saved to /var/cache/conftool/dbconfig/20260528-105753-fceratto.json * 10:56 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 10:55 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 10:55 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 10:55 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 10:50 moritzm: update trixie netboot image for 13.5 point release [[phab:T427072|T427072]] * 10:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P93346 and previous config saved to /var/cache/conftool/dbconfig/20260528-104745-fceratto.json * 10:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P93345 and previous config saved to /var/cache/conftool/dbconfig/20260528-103738-fceratto.json * 10:29 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P13724 # [[phab:T406971|T406971]] * 10:28 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P14223 # [[phab:T422264|T422264]] * 10:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93344 and previous config saved to /var/cache/conftool/dbconfig/20260528-102730-fceratto.json * 10:26 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P1748 # [[phab:T422392|T422392]] * 10:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93343 and previous config saved to /var/cache/conftool/dbconfig/20260528-101900-fceratto.json * 10:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance * 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93342 and previous config saved to /var/cache/conftool/dbconfig/20260528-101829-fceratto.json * 10:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P93341 and previous config saved to /var/cache/conftool/dbconfig/20260528-100822-fceratto.json * 09:59 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290687{{!}}stream: webrequest.page_view (T426092 T426091)]] (duration: 06m 41s) * 09:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P93340 and previous config saved to /var/cache/conftool/dbconfig/20260528-095814-fceratto.json * 09:55 javiermonton@deploy1003: javiermonton: Continuing with deployment * 09:54 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1290687{{!}}stream: webrequest.page_view (T426092 T426091)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:52 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1290687{{!}}stream: webrequest.page_view (T426092 T426091)]] * 09:48 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294243{{!}}Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937{{!}}CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] (duration: 07m 37s) * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93339 and previous config saved to /var/cache/conftool/dbconfig/20260528-094807-fceratto.json * 09:44 dreamyjazz@deploy1003: dreamyjazz, stran: Continuing with deployment * 09:44 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:43 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:43 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:43 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:42 dreamyjazz@deploy1003: dreamyjazz, stran: Backport for [[gerrit:1294243{{!}}Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937{{!}}CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294243{{!}}Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937{{!}}CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] * 09:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93338 and previous config saved to /var/cache/conftool/dbconfig/20260528-093920-fceratto.json * 09:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: Maintenance * 09:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93337 and previous config saved to /var/cache/conftool/dbconfig/20260528-093849-fceratto.json * 09:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P93336 and previous config saved to /var/cache/conftool/dbconfig/20260528-092842-fceratto.json * 09:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance * 09:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93335 and previous config saved to /var/cache/conftool/dbconfig/20260528-092239-fceratto.json * 09:22 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pki-root1001.eqiad.wmnet * 09:22 elukey@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:22 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki-root1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1003" * 09:22 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki-root1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1003" * 09:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:18 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 09:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P93334 and previous config saved to /var/cache/conftool/dbconfig/20260528-091834-fceratto.json * 09:18 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 09:18 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 09:17 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1165: Reboot completed * 09:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 09:17 elukey@cumin1003: START - Cookbook sre.dns.netbox * 09:14 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 09:13 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:13 elukey@cumin1003: START - Cookbook sre.hosts.decommission for hosts pki-root1001.eqiad.wmnet * 09:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P93332 and previous config saved to /var/cache/conftool/dbconfig/20260528-091231-fceratto.json * 09:09 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:09 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93331 and previous config saved to /var/cache/conftool/dbconfig/20260528-090826-fceratto.json * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P93329 and previous config saved to /var/cache/conftool/dbconfig/20260528-090224-fceratto.json * 09:02 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Deploying to prod (duration: 02m 31s) * 09:01 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2216 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93328 and previous config saved to /var/cache/conftool/dbconfig/20260528-090114-fceratto.json * 09:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2216.codfw.wmnet with reason: Maintenance * 09:00 joal@deploy1003: Finished deploy [analytics/refinery@878cb24] (thin): Regular analytics weekly train THIN - 2[analytics/refinery@878cb24a] (duration: 02m 08s) * 08:59 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Deploying to prod * 08:58 joal@deploy1003: Started deploy [analytics/refinery@878cb24] (thin): Regular analytics weekly train THIN - 2[analytics/refinery@878cb24a] * 08:57 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Testing on backup host (duration: 00m 53s) * 08:56 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Testing on backup host * 08:56 joal@deploy1003: Finished deploy [analytics/refinery@878cb24]: Regular analytics weekly train - 2 [analytics/refinery@878cb24a] (duration: 06m 54s) * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93327 and previous config saved to /var/cache/conftool/dbconfig/20260528-085216-fceratto.json * 08:50 XioNoX: cr1-codfw# delete protocols bgp group fundraising family inet6 - [[phab:T423384|T423384]] * 08:49 joal@deploy1003: Started deploy [analytics/refinery@878cb24]: Regular analytics weekly train - 2 [analytics/refinery@878cb24a] * 08:49 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294925{{!}}hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] (duration: 09m 20s) * 08:49 joal@deploy1003: Finished deploy [analytics/refinery@878cb24] (hadoop-test): Regular analytics weekly train TEST -2 [analytics/refinery@878cb24a] (duration: 02m 00s) * 08:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93326 and previous config saved to /var/cache/conftool/dbconfig/20260528-084906-fceratto.json * 08:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1209.eqiad.wmnet with reason: Maintenance * 08:48 slyngshede@dns1004: END - running authdns-update * 08:47 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1165: Reboot completed * 08:47 joal@deploy1003: Started deploy [analytics/refinery@878cb24] (hadoop-test): Regular analytics weekly train TEST -2 [analytics/refinery@878cb24a] * 08:47 slyngs: Upgrade IDP to CAS 7.3.7.1 * 08:46 slyngshede@dns1004: START - running authdns-update * 08:45 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 08:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93324 and previous config saved to /var/cache/conftool/dbconfig/20260528-084149-fceratto.json * 08:41 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1294925{{!}}hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294925{{!}}hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] * 08:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet * 08:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet * 08:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93323 and previous config saved to /var/cache/conftool/dbconfig/20260528-083504-fceratto.json * 08:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1025].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 08:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1165.eqiad.wmnet with reason: Maintenance * 08:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93322 and previous config saved to /var/cache/conftool/dbconfig/20260528-083331-fceratto.json * 08:24 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1209: Test * 08:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P93320 and previous config saved to /var/cache/conftool/dbconfig/20260528-082324-fceratto.json * 08:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2189: repool after crash * 08:17 slyngshede@dns1004: END - running authdns-update * 08:16 slyngshede@dns1004: START - running authdns-update * 08:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P93318 and previous config saved to /var/cache/conftool/dbconfig/20260528-081316-fceratto.json * 08:10 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 08:09 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1209: Test * 08:05 hashar@deploy1003: Finished deploy [integration/docroot@2a51016]: build: update dependencies + eslint fix in comment. f021d3f..2a51016 (duration: 00m 13s) * 08:05 hashar@deploy1003: Started deploy [integration/docroot@2a51016]: build: update dependencies + eslint fix in comment. f021d3f..2a51016 * 08:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93315 and previous config saved to /var/cache/conftool/dbconfig/20260528-080309-fceratto.json * 07:56 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93314 and previous config saved to /var/cache/conftool/dbconfig/20260528-075631-fceratto.json * 07:56 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020,1022-1023].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 07:56 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance * 07:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93313 and previous config saved to /var/cache/conftool/dbconfig/20260528-075521-fceratto.json * 07:47 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab replica * 07:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93311 and previous config saved to /var/cache/conftool/dbconfig/20260528-074513-fceratto.json * 07:37 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2189: repool after crash * 07:36 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab replica * 07:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93309 and previous config saved to /var/cache/conftool/dbconfig/20260528-073506-fceratto.json * 07:34 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica * 07:29 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294808{{!}}Don't run the click intent experiment on mobile (T426743)]] (duration: 06m 29s) * 07:25 wmde-fisch@deploy1003: thiemowmde, wmde-fisch: Continuing with deployment * 07:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93308 and previous config saved to /var/cache/conftool/dbconfig/20260528-072458-fceratto.json * 07:24 wmde-fisch@deploy1003: thiemowmde, wmde-fisch: Backport for [[gerrit:1294808{{!}}Don't run the click intent experiment on mobile (T426743)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:24 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica * 07:23 tgr@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwikisource --logwiki=metawiki Ioed Renamed_user_4232d41570b9e8f46ef150e5e360e446 # [[phab:T427459|T427459]] * 07:22 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1294808{{!}}Don't run the click intent experiment on mobile (T426743)]] * 07:20 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1270986{{!}}Update wikimania wordmark for 2026 (T413331)]] (duration: 06m 54s) * 07:18 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93307 and previous config saved to /var/cache/conftool/dbconfig/20260528-071836-fceratto.json * 07:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1264.eqiad.wmnet with reason: Maintenance * 07:16 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1167: Reboot completed * 07:16 wmde-fisch@deploy1003: wmde-fisch, robertsky: Continuing with deployment * 07:15 wmde-fisch@deploy1003: wmde-fisch, robertsky: Backport for [[gerrit:1270986{{!}}Update wikimania wordmark for 2026 (T413331)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:13 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1270986{{!}}Update wikimania wordmark for 2026 (T413331)]] * 07:11 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289898{{!}}Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] (duration: 07m 15s) * 07:07 wmde-fisch@deploy1003: wmde-fisch, arthurtaylor: Continuing with deployment * 07:06 wmde-fisch@deploy1003: wmde-fisch, arthurtaylor: Backport for [[gerrit:1289898{{!}}Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:04 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1289898{{!}}Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] * 06:43 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1167: Reboot completed * 06:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93303 and previous config saved to /var/cache/conftool/dbconfig/20260528-064217-fceratto.json * 06:33 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93302 and previous config saved to /var/cache/conftool/dbconfig/20260528-063357-fceratto.json * 06:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 06:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance * 06:25 hashar: Restarting CI Jenkins for plugins upgrades * 06:16 fceratto@dns1005: END - running authdns-update * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1209 [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93301 and previous config saved to /var/cache/conftool/dbconfig/20260528-061609-fceratto.json * 06:14 fceratto@dns1005: START - running authdns-update * 06:11 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1193 to s8 primary and set section read-write [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93300 and previous config saved to /var/cache/conftool/dbconfig/20260528-061138-fceratto.json * 06:10 fceratto@cumin1003: dbctl commit (dc=all): 'Set s8 eqiad as read-only for maintenance - [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93299 and previous config saved to /var/cache/conftool/dbconfig/20260528-061048-fceratto.json * 06:10 federico3: Starting s8 eqiad failover from db1209 to db1193 - [[phab:T426095|T426095]] * 06:04 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1193 with weight 0 [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93298 and previous config saved to /var/cache/conftool/dbconfig/20260528-060412-fceratto.json * 06:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s8 [[phab:T426095|T426095]] * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 41s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:53 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 00:53 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new subnet in eqsin - pt1979@cumin2002" * 00:53 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new subnet in eqsin - pt1979@cumin2002" * 00:49 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 00:25 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294470{{!}}Activate conductwiki (T426984)]] (duration: 07m 12s) * 00:21 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 00:20 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1294470{{!}}Activate conductwiki (T426984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 00:18 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1294470{{!}}Activate conductwiki (T426984)]] * 00:12 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294438{{!}}Init conductwiki (T426984)]] (duration: 07m 25s) * 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 00:08 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 00:06 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1294438{{!}}Init conductwiki (T426984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 00:04 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1294438{{!}}Init conductwiki (T426984)]] * 00:04 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] == 2026-05-27 == * 23:13 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294432{{!}}Exclude more content from selection (T426308)]], [[gerrit:1285523{{!}}Remove MinervaNightMode config after skin cleanup (T426689)]] (duration: 08m 42s) * 23:09 jdlrobson@deploy1003: jdlrobson, h2o, egardner: Continuing with deployment * 23:06 jdlrobson@deploy1003: jdlrobson, h2o, egardner: Backport for [[gerrit:1294432{{!}}Exclude more content from selection (T426308)]], [[gerrit:1285523{{!}}Remove MinervaNightMode config after skin cleanup (T426689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:04 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294432{{!}}Exclude more content from selection (T426308)]], [[gerrit:1285523{{!}}Remove MinervaNightMode config after skin cleanup (T426689)]] * 22:58 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294435{{!}}passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] (duration: 07m 49s) * 22:55 ladsgroup@cumin1003: END (PASS) - Cookbook sre.mysql.sanitarium_restart (exit_code=0) * 22:54 catrope@deploy1003: catrope: Continuing with deployment * 22:52 catrope@deploy1003: catrope: Backport for [[gerrit:1294435{{!}}passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:50 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1294435{{!}}passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] * 22:46 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294360{{!}}Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322{{!}}Thumbnails are not being optimized in large mode (T427237)]] (duration: 06m 54s) * 22:42 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 22:41 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1294360{{!}}Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322{{!}}Thumbnails are not being optimized in large mode (T427237)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:40 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart * 22:40 ladsgroup@cumin1003: END (FAIL) - Cookbook sre.mysql.sanitarium_restart (exit_code=99) * 22:40 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart * 22:39 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294360{{!}}Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322{{!}}Thumbnails are not being optimized in large mode (T427237)]] * 22:39 ladsgroup@deploy1003: Finished scap sync-world: Add conduct.wikimedia.org ([[phab:T426984|T426984]]) (duration: 07m 16s) * 22:35 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 22:34 ladsgroup@deploy1003: ladsgroup: Add conduct.wikimedia.org ([[phab:T426984|T426984]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:33 ladsgroup@deploy1003: Started scap sync-world: Add conduct.wikimedia.org ([[phab:T426984|T426984]]) * 22:13 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294370{{!}}Carousel only on articles (T427336)]] (duration: 10m 00s) * 22:09 egardner@deploy1003: egardner: Continuing with deployment * 22:05 egardner@deploy1003: egardner: Backport for [[gerrit:1294370{{!}}Carousel only on articles (T427336)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:03 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1294370{{!}}Carousel only on articles (T427336)]] * 21:37 bking@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 15 days, 0:00:00 on relforge[1008-1010].eqiad.wmnet with reason: non-production environment * 21:20 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 21:20 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 21:20 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 21:19 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 21:04 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1288370{{!}}Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819{{!}}Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] (duration: 07m 38s) * 20:59 ebernhardson@deploy1003: matmarex, ebernhardson, pppery: Continuing with deployment * 20:58 ebernhardson@deploy1003: matmarex, ebernhardson, pppery: Backport for [[gerrit:1288370{{!}}Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819{{!}}Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:56 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1288370{{!}}Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819{{!}}Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] * 20:51 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294373{{!}}identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374{{!}}Revert^2 "cirrus: AB test query suggester variants" (T407432)]] (duration: 07m 30s) * 20:47 ebernhardson@deploy1003: ebernhardson: Continuing with deployment * 20:46 ebernhardson@deploy1003: ebernhardson: Backport for [[gerrit:1294373{{!}}identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374{{!}}Revert^2 "cirrus: AB test query suggester variants" (T407432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:44 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294373{{!}}identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374{{!}}Revert^2 "cirrus: AB test query suggester variants" (T407432)]] * 20:43 swfrench-wmf: reprepro include dh-php_5.5+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 20:39 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts lvs1016.eqiad.wmnet * 20:39 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:39 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brett@cumin2002" * 20:38 swfrench-wmf: reprepro include php-defaults_94+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 20:37 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brett@cumin2002" * 20:31 brett@cumin2002: START - Cookbook sre.dns.netbox * 20:27 swfrench-wmf: reprepro include php8.3_8.3.31-1+wmf12u2 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 20:25 brett@cumin2002: START - Cookbook sre.hosts.decommission for hosts lvs1016.eqiad.wmnet * 20:25 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294342{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344{{!}}frwiki: restrict Article Guidance experiment to junior editors (T426871)]] (duration: 08m 11s) * 20:21 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host lvs1016.eqiad.wmnet with OS bullseye * 20:21 sbisson@deploy1003: sbisson: Continuing with deployment * 20:20 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1020.eqiad.wmnet * 20:19 sbisson@deploy1003: sbisson: Backport for [[gerrit:1294342{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344{{!}}frwiki: restrict Article Guidance experiment to junior editors (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be v * 20:17 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294342{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344{{!}}frwiki: restrict Article Guidance experiment to junior editors (T426871)]] * 20:14 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet * 20:05 cmooney@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 12355 * 20:04 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 12355 * 19:51 brett@cumin2002: START - Cookbook sre.hosts.reimage for host lvs1016.eqiad.wmnet with OS bullseye * 19:48 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 19:45 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 19:45 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 19:32 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp6016.drmrs.wmnet,cp[1112,1114].eqiad.wmnet,cp[5024,5031-5032].eqsin.wmnet<nowiki>}</nowiki> and A:cp * 19:32 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5032.eqsin.wmnet * 19:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 19:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 19:01 joal@deploy1003: Finished deploy [analytics/refinery@96cf761] (thin): Regular analytics weekly train THIN [analytics/refinery@96cf761f] (duration: 02m 08s) * 18:59 joal@deploy1003: Started deploy [analytics/refinery@96cf761] (thin): Regular analytics weekly train THIN [analytics/refinery@96cf761f] * 18:58 joal@deploy1003: Finished deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] (duration: 05m 01s) * 18:53 joal@deploy1003: Started deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] * 18:53 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294376{{!}}Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375{{!}}auth: Mark the hidden token field used for reauth as skippable (T427398)]] (duration: 07m 41s) * 18:49 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5031.eqsin.wmnet * 18:49 catrope@deploy1003: catrope: Continuing with deployment * 18:47 catrope@deploy1003: catrope: Backport for [[gerrit:1294376{{!}}Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375{{!}}auth: Mark the hidden token field used for reauth as skippable (T427398)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:45 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1294376{{!}}Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375{{!}}auth: Mark the hidden token field used for reauth as skippable (T427398)]] * 18:40 joal@deploy1003: Finished deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] (duration: 01m 05s) * 18:39 joal@deploy1003: Started deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] * 18:37 joal@deploy1003: Finished deploy [analytics/refinery@96cf761] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96cf761f] (duration: 02m 04s) * 18:35 joal@deploy1003: Started deploy [analytics/refinery@96cf761] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96cf761f] * 18:29 swfrench@deploy1003: Finished scap sync-world: Helmfile-only deployment to clean up unused mesh listeners (duration: 06m 12s) * 18:25 swfrench@deploy1003: swfrench: Continuing with deployment * 18:24 swfrench@deploy1003: swfrench: Helmfile-only deployment to clean up unused mesh listeners synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:23 swfrench@deploy1003: Started scap sync-world: Helmfile-only deployment to clean up unused mesh listeners * 18:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93296 and previous config saved to /var/cache/conftool/dbconfig/20260527-181923-fceratto.json * 18:13 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 18:12 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 18:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 18:11 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 18:11 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 18:10 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 18:10 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 18:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93295 and previous config saved to /var/cache/conftool/dbconfig/20260527-180915-fceratto.json * 18:09 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 18:09 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293776{{!}}ProductionServices: Revert to discovery shellbox listeners]] (duration: 10m 24s) * 18:08 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1017.eqiad.wmnet * 18:08 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1017.eqiad.wmnet * 18:07 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5024.eqsin.wmnet * 18:03 swfrench@deploy1003: swfrench: Continuing with deployment * 18:02 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 18:02 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 18:02 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 18:01 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:01 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 18:01 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 18:00 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 18:00 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293776{{!}}ProductionServices: Revert to discovery shellbox listeners]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:00 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93294 and previous config saved to /var/cache/conftool/dbconfig/20260527-175908-fceratto.json * 17:58 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293776{{!}}ProductionServices: Revert to discovery shellbox listeners]] * 17:55 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93293 and previous config saved to /var/cache/conftool/dbconfig/20260527-174900-fceratto.json * 17:43 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293774{{!}}ProductionServices: Temporarily use shellbox in codfw]] (duration: 15m 01s) * 17:38 swfrench@deploy1003: swfrench: Continuing with deployment * 17:31 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293774{{!}}ProductionServices: Temporarily use shellbox in codfw]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:28 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293774{{!}}ProductionServices: Temporarily use shellbox in codfw]] * 17:25 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp1114.eqiad.wmnet * 17:18 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:15 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:15 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:14 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:05 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293775{{!}}ProductionServices: Temporarily use shellbox in eqiad]] (duration: 08m 44s) * 17:00 swfrench@deploy1003: swfrench: Continuing with deployment * 16:58 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293775{{!}}ProductionServices: Temporarily use shellbox in eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:56 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293775{{!}}ProductionServices: Temporarily use shellbox in eqiad]] * 16:53 atsuko@dns1004: END - running authdns-update * 16:51 atsuko@dns1004: START - running authdns-update * 16:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93292 and previous config saved to /var/cache/conftool/dbconfig/20260527-164846-fceratto.json * 16:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1264.eqiad.wmnet with reason: Maintenance * 16:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93291 and previous config saved to /var/cache/conftool/dbconfig/20260527-164815-fceratto.json * 16:43 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp1112.eqiad.wmnet * 16:41 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1017.eqiad.wmnet with reason: Setting up * 16:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P93290 and previous config saved to /var/cache/conftool/dbconfig/20260527-163808-fceratto.json * 16:37 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2163: Repooling after testing patch * 16:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P93287 and previous config saved to /var/cache/conftool/dbconfig/20260527-162800-fceratto.json * 16:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93285 and previous config saved to /var/cache/conftool/dbconfig/20260527-161753-fceratto.json * 16:14 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 16:13 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 16:13 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 16:12 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 16:11 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93284 and previous config saved to /var/cache/conftool/dbconfig/20260527-161101-fceratto.json * 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: Maintenance * 16:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93283 and previous config saved to /var/cache/conftool/dbconfig/20260527-161034-fceratto.json * 16:10 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 16:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1178: Recovering from failure in cookbook * 16:10 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 16:05 sukhe@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host durum5003.eqsin.wmnet with OS trixie * 16:03 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp6016.drmrs.wmnet * 16:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220', diff saved to https://phabricator.wikimedia.org/P93280 and previous config saved to /var/cache/conftool/dbconfig/20260527-160027-fceratto.json * 15:59 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1017.eqiad.wmnet * 15:53 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2163.codfw.wmnet * 15:53 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2163.codfw.wmnet * 15:52 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs1017.eqiad.wmnet * 15:52 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Repooling after testing patch * 15:52 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp6016.drmrs.wmnet,cp[1112,1114].eqiad.wmnet,cp[5024,5031-5032].eqsin.wmnet<nowiki>}</nowiki> and A:cp * 15:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2163: Testing cookbook * 15:50 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2163: Testing cookbook * 15:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220', diff saved to https://phabricator.wikimedia.org/P93276 and previous config saved to /var/cache/conftool/dbconfig/20260527-155019-fceratto.json * 15:45 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:45 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93274 and previous config saved to /var/cache/conftool/dbconfig/20260527-154011-fceratto.json * 15:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 15:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2163: Migration of db2163.codfw.wmnet completed * 15:32 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Migration of db2163.codfw.wmnet completed * 15:32 cwilliams@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2163: Migration of db2163.codfw.wmnet completed * 15:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1178: Recovering from failure in cookbook * 15:22 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1178.eqiad.wmnet * 15:22 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1178.eqiad.wmnet * 15:19 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 15:19 cdanis: 💙cdanis@cp4047.ulsfo.wmnet ~ 🕦☕ sudo apt install lua5.4-ciderbloom lua5.4-ciderbloom-dbgsym * 15:13 cdanis: 💙cdanis@cp5026.eqsin.wmnet ~ 🕚☕ sudo apt install lua5.4-ciderbloom lua5.4-ciderbloom-dbgsym * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:11 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:11 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:11 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Icinga wait failed during run * 15:10 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:10 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:10 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:09 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:09 cdanis: 💔cdanis@apt1002.wikimedia.org ~ 🕚☕ sudo -i reprepro --component main --restrict cidergrinder update trixie-wikimedia * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:05 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93268 and previous config saved to /var/cache/conftool/dbconfig/20260527-150508-fceratto.json * 15:05 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1220.eqiad.wmnet with reason: Maintenance * 15:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93267 and previous config saved to /var/cache/conftool/dbconfig/20260527-150438-fceratto.json * 14:59 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Migration of db2163.codfw.wmnet completed * 14:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P93264 and previous config saved to /var/cache/conftool/dbconfig/20260527-145430-fceratto.json * 14:54 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 14:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2163.codfw.wmnet with OS trixie * 14:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply * 14:50 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply * 14:46 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290926{{!}}Re-enable ReadingLists QuickSurvey (T426781)]] (duration: 08m 32s) * 14:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1178.eqiad.wmnet with OS trixie * 14:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P93263 and previous config saved to /var/cache/conftool/dbconfig/20260527-144423-fceratto.json * 14:42 aude@deploy1003: aude: Continuing with deployment * 14:40 aude@deploy1003: aude: Backport for [[gerrit:1290926{{!}}Re-enable ReadingLists QuickSurvey (T426781)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:38 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db2189.codfw.wmnet with reason: crashed [[phab:T427376|T427376]] * 14:38 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1290926{{!}}Re-enable ReadingLists QuickSurvey (T426781)]] * 14:35 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290924{{!}}Make logging of title and page ID optional (T426457)]] (duration: 11m 30s) * 14:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93262 and previous config saved to /var/cache/conftool/dbconfig/20260527-143416-fceratto.json * 14:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2163.codfw.wmnet with reason: host reimage * 14:29 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2163.codfw.wmnet with reason: host reimage * 14:29 aude@deploy1003: aude: Continuing with deployment * 14:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1178.eqiad.wmnet with reason: host reimage * 14:27 aude@deploy1003: aude: Backport for [[gerrit:1290924{{!}}Make logging of title and page ID optional (T426457)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:27 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93260 and previous config saved to /var/cache/conftool/dbconfig/20260527-142659-fceratto.json * 14:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1179.eqiad.wmnet with reason: Maintenance * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:23 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1290924{{!}}Make logging of title and page ID optional (T426457)]] * 14:22 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1178.eqiad.wmnet with reason: host reimage * 14:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1033.eqiad.wmnet with reason: Maintenance * 14:18 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294247{{!}}Update Direct Reporting email (T427358)]] (duration: 33m 01s) * 14:10 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2163.codfw.wmnet with OS trixie * 14:09 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1178.eqiad.wmnet with OS trixie * 14:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2163: Upgrading db2163.codfw.wmnet * 14:08 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2163: Upgrading db2163.codfw.wmnet * 14:08 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1178: Upgrading db1178.eqiad.wmnet * 14:07 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1178: Upgrading db1178.eqiad.wmnet * 14:06 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:06 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:06 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:06 stran@deploy1003: stran: Continuing with deployment * 14:02 stran@deploy1003: stran: Backport for [[gerrit:1294247{{!}}Update Direct Reporting email (T427358)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:56 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 13:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2164: Migration of db2164.codfw.wmnet completed * 13:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1192: Migration of db1192.eqiad.wmnet completed * 13:45 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1294247{{!}}Update Direct Reporting email (T427358)]] * 13:40 phuedx@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294217{{!}}ext.wikimediaEvents: Add hoisting error detection test (T427092)]] (duration: 11m 35s) * 13:36 phuedx@deploy1003: phuedx: Continuing with deployment * 13:30 phuedx@deploy1003: phuedx: Backport for [[gerrit:1294217{{!}}ext.wikimediaEvents: Add hoisting error detection test (T427092)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:28 phuedx@deploy1003: Started scap sync-world: Backport for [[gerrit:1294217{{!}}ext.wikimediaEvents: Add hoisting error detection test (T427092)]] * 13:21 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290781{{!}}mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264{{!}}MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] (duration: 13m 23s) * 13:15 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2189: Test * 13:15 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2189: Test * 13:15 mlitn@deploy1003: krinkle, mlitn: Continuing with deployment * 13:13 mlitn@deploy1003: krinkle, mlitn: Backport for [[gerrit:1290781{{!}}mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264{{!}}MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 13:10 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2164: Migration of db2164.codfw.wmnet completed * 13:08 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1290781{{!}}mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264{{!}}MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] * 13:06 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 13:05 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db2212.codfw.wmnet with reason: failed to reboot [[phab:T427388|T427388]] [[phab:T426633|T426633]] * 13:05 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1192: Migration of db1192.eqiad.wmnet completed * 13:01 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2164.codfw.wmnet with OS trixie * 12:57 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1192.eqiad.wmnet with OS trixie * 12:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2164.codfw.wmnet with reason: host reimage * 12:40 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1192.eqiad.wmnet with reason: host reimage * 12:40 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2164.codfw.wmnet with reason: host reimage * 12:35 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1192.eqiad.wmnet with reason: host reimage * 12:28 Amir1: deleting binlogs older than a year * 12:22 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2164.codfw.wmnet with OS trixie * 12:21 cmooney@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 36692 * 12:21 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1192.eqiad.wmnet with OS trixie * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1077 * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1080 * 12:20 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1077 * 12:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2164: Upgrading db2164.codfw.wmnet * 12:20 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 36692 * 12:20 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1080 * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1078 * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1079 * 12:20 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2164: Upgrading db2164.codfw.wmnet * 12:19 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:19 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1079 * 12:19 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1078 * 12:19 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:19 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1192: Upgrading db1192.eqiad.wmnet * 12:19 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:18 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1192: Upgrading db1192.eqiad.wmnet * 12:18 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:15 jclark@cumin1003: START - Cookbook sre.dns.netbox * 12:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2165: Migration of db2165.codfw.wmnet completed * 12:14 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:14 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:14 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:12 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db2189: Test * 12:11 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2189: Test * 12:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1193: Migration of db1193.eqiad.wmnet completed * 12:09 jclark@cumin1003: START - Cookbook sre.dns.netbox * 12:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93243 and previous config saved to /var/cache/conftool/dbconfig/20260527-120452-fceratto.json * 12:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: Maintenance * 12:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93242 and previous config saved to /var/cache/conftool/dbconfig/20260527-120205-fceratto.json * 12:01 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 11:58 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 11:58 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "is everything alright? /cc effie - ayounsi@cumin1003" * 11:58 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "is everything alright? /cc effie - ayounsi@cumin1003" * 11:56 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 11:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P93239 and previous config saved to /var/cache/conftool/dbconfig/20260527-115157-fceratto.json * 11:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P93237 and previous config saved to /var/cache/conftool/dbconfig/20260527-114149-fceratto.json * 11:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93235 and previous config saved to /var/cache/conftool/dbconfig/20260527-113142-fceratto.json * 11:29 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2165: Migration of db2165.codfw.wmnet completed * 11:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1193: Migration of db1193.eqiad.wmnet completed * 11:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93231 and previous config saved to /var/cache/conftool/dbconfig/20260527-112327-fceratto.json * 11:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2188.codfw.wmnet with reason: Maintenance * 11:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93230 and previous config saved to /var/cache/conftool/dbconfig/20260527-112257-fceratto.json * 11:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2165.codfw.wmnet with OS trixie * 11:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1193.eqiad.wmnet with OS trixie * 11:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P93229 and previous config saved to /var/cache/conftool/dbconfig/20260527-111250-fceratto.json * 11:10 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:10 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:08 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:08 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:02 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P93227 and previous config saved to /var/cache/conftool/dbconfig/20260527-110242-fceratto.json * 11:02 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:02 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 11:01 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 11:01 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2165.codfw.wmnet with reason: host reimage * 11:00 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2189', diff saved to https://phabricator.wikimedia.org/P93226 and previous config saved to /var/cache/conftool/dbconfig/20260527-110016-marostegui.json * 10:58 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1193.eqiad.wmnet with reason: host reimage * 10:57 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2165.codfw.wmnet with reason: host reimage * 10:56 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 10:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93225 and previous config saved to /var/cache/conftool/dbconfig/20260527-105235-fceratto.json * 10:52 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1193.eqiad.wmnet with reason: host reimage * 10:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1050: repool after maintenance * 10:45 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2176 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93223 and previous config saved to /var/cache/conftool/dbconfig/20260527-104518-fceratto.json * 10:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2176.codfw.wmnet with reason: Maintenance * 10:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93222 and previous config saved to /var/cache/conftool/dbconfig/20260527-104449-fceratto.json * 10:39 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2165.codfw.wmnet with OS trixie * 10:38 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1193.eqiad.wmnet with OS trixie * 10:36 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1193: Upgrading db1193.eqiad.wmnet * 10:35 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1193: Upgrading db1193.eqiad.wmnet * 10:35 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2165: Upgrading db2165.codfw.wmnet * 10:35 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2165: Upgrading db2165.codfw.wmnet * 10:34 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P93218 and previous config saved to /var/cache/conftool/dbconfig/20260527-103441-fceratto.json * 10:29 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:29 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P93217 and previous config saved to /var/cache/conftool/dbconfig/20260527-102434-fceratto.json * 10:22 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:21 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93215 and previous config saved to /var/cache/conftool/dbconfig/20260527-101426-fceratto.json * 10:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1203: Migration of db1203.eqiad.wmnet completed * 10:10 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2166: Migration of db2166.codfw.wmnet completed * 10:08 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93212 and previous config saved to /var/cache/conftool/dbconfig/20260527-100701-fceratto.json * 10:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance * 10:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93211 and previous config saved to /var/cache/conftool/dbconfig/20260527-100632-fceratto.json * 10:05 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1050: repool after maintenance * 10:04 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:02 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1050.eqiad.wmnet with OS trixie * 09:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P93208 and previous config saved to /var/cache/conftool/dbconfig/20260527-095624-fceratto.json * 09:47 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 09:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P93206 and previous config saved to /var/cache/conftool/dbconfig/20260527-094616-fceratto.json * 09:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1050.eqiad.wmnet with reason: host reimage * 09:43 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 09:41 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1050.eqiad.wmnet with reason: host reimage * 09:38 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 09:38 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 09:37 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 09:37 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 09:36 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 09:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93203 and previous config saved to /var/cache/conftool/dbconfig/20260527-093609-fceratto.json * 09:34 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 09:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93202 and previous config saved to /var/cache/conftool/dbconfig/20260527-092842-fceratto.json * 09:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance * 09:28 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1203: Migration of db1203.eqiad.wmnet completed * 09:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93200 and previous config saved to /var/cache/conftool/dbconfig/20260527-092814-fceratto.json * 09:27 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1050.eqiad.wmnet with OS trixie * 09:26 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1050: Upgrading es1050.eqiad.wmnet * 09:25 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1050: Upgrading es1050.eqiad.wmnet * 09:25 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1050: repool after maintenance * 09:25 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1050: repool after maintenance * 09:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2166: Migration of db2166.codfw.wmnet completed * 09:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2051: repool after maintenance * 09:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1203.eqiad.wmnet with OS trixie * 09:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P93196 and previous config saved to /var/cache/conftool/dbconfig/20260527-091806-fceratto.json * 09:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2166.codfw.wmnet with OS trixie * 09:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P93194 and previous config saved to /var/cache/conftool/dbconfig/20260527-090759-fceratto.json * 09:03 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp3074.* * 09:03 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp3066.* * 09:03 fabfur: repooling cp3074 and cp3066 ([[phab:T419825|T419825]]) * 09:02 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp6015.drmrs.wmnet * 09:02 slyngshede@cumin1003: START - Cookbook sre.hosts.remove-downtime for cp6015.drmrs.wmnet * 09:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1203.eqiad.wmnet with reason: host reimage * 09:02 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: name=cp6015.* * 08:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2166.codfw.wmnet with reason: host reimage * 08:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93193 and previous config saved to /var/cache/conftool/dbconfig/20260527-085751-fceratto.json * 08:55 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1203.eqiad.wmnet with reason: host reimage * 08:54 Emperor: restart swift on ms-fe2011 [[phab:T360913|T360913]] * 08:54 jayme@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:54 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2166.codfw.wmnet with reason: host reimage * 08:54 jayme@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 08:51 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 08:51 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 08:51 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp3066.* * 08:51 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp3074.* * 08:51 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 08:50 fabfur: depooling and installing haproxy-awslc on cp3074 and cp3066 ([[phab:T419825|T419825]]) * 08:50 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93191 and previous config saved to /var/cache/conftool/dbconfig/20260527-085024-fceratto.json * 08:50 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance * 08:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93190 and previous config saved to /var/cache/conftool/dbconfig/20260527-085005-fceratto.json * 08:41 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1203.eqiad.wmnet with OS trixie * 08:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P93189 and previous config saved to /var/cache/conftool/dbconfig/20260527-083957-fceratto.json * 08:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2051: repool after maintenance * 08:37 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 08:36 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1203: Upgrading db1203.eqiad.wmnet * 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1004.wikimedia.org * 08:36 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1203: Upgrading db1203.eqiad.wmnet * 08:36 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:35 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2166.codfw.wmnet with OS trixie * 08:35 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2051.codfw.wmnet with OS trixie * 08:34 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2166: Upgrading db2166.codfw.wmnet * 08:33 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2166: Upgrading db2166.codfw.wmnet * 08:33 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader1004.wikimedia.org * 08:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org * 08:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P93185 and previous config saved to /var/cache/conftool/dbconfig/20260527-082950-fceratto.json * 08:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org * 08:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93184 and previous config saved to /var/cache/conftool/dbconfig/20260527-081942-fceratto.json * 08:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2051.codfw.wmnet with reason: host reimage * 08:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2051.codfw.wmnet with reason: host reimage * 08:11 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 08:11 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2153 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93183 and previous config saved to /var/cache/conftool/dbconfig/20260527-081112-fceratto.json * 08:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance * 08:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93182 and previous config saved to /var/cache/conftool/dbconfig/20260527-081054-fceratto.json * 08:07 jmm@dns1004: END - running authdns-update * 08:05 jmm@dns1004: START - running authdns-update * 08:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248', diff saved to https://phabricator.wikimedia.org/P93181 and previous config saved to /var/cache/conftool/dbconfig/20260527-080046-fceratto.json * 07:59 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2051.codfw.wmnet with OS trixie * 07:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248', diff saved to https://phabricator.wikimedia.org/P93180 and previous config saved to /var/cache/conftool/dbconfig/20260527-075039-fceratto.json * 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1026.eqiad.wmnet * 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1026.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:43 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1026.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2051: Upgrading es2051.codfw.wmnet * 07:42 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2051: Upgrading es2051.codfw.wmnet * 07:41 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93178 and previous config saved to /var/cache/conftool/dbconfig/20260527-074031-fceratto.json * 07:40 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294125{{!}}Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126{{!}}Add script to demote ineligible members of restricted global groups (T425395)]] (duration: 06m 42s) * 07:36 mszwarc@deploy1003: mszwarc: Continuing with deployment * 07:35 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1294125{{!}}Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126{{!}}Add script to demote ineligible members of restricted global groups (T425395)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93177 and previous config saved to /var/cache/conftool/dbconfig/20260527-073504-fceratto.json * 07:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2248.codfw.wmnet with reason: Maintenance * 07:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93176 and previous config saved to /var/cache/conftool/dbconfig/20260527-073434-fceratto.json * 07:33 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1294125{{!}}Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126{{!}}Add script to demote ineligible members of restricted global groups (T425395)]] * 07:28 jmm@cumin2002: START - Cookbook sre.dns.netbox * 07:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P93175 and previous config saved to /var/cache/conftool/dbconfig/20260527-072426-fceratto.json * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) * 07:23 marostegui@cumin1003: Removing pc1014 from zarcillo [[phab:T427190|T427190]] * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1014.eqiad.wmnet * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 07:23 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 07:18 marostegui@cumin1003: START - Cookbook sre.dns.netbox * 07:15 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1026.eqiad.wmnet * 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1025.eqiad.wmnet * 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P93174 and previous config saved to /var/cache/conftool/dbconfig/20260527-071418-fceratto.json * 07:13 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1014.eqiad.wmnet * 07:13 marostegui@cumin1003: START - Cookbook sre.mysql.decommission * 07:13 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org * 07:07 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2055: repool after maintenance * 07:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org * 07:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1003.wikimedia.org * 07:06 jmm@cumin2002: START - Cookbook sre.dns.netbox * 07:06 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1190.eqiad.wmnet with reason: Maintenance on db1190 * 07:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93172 and previous config saved to /var/cache/conftool/dbconfig/20260527-070410-fceratto.json * 07:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader1003.wikimedia.org * 06:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93171 and previous config saved to /var/cache/conftool/dbconfig/20260527-065545-fceratto.json * 06:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2247.codfw.wmnet with reason: Maintenance * 06:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93170 and previous config saved to /var/cache/conftool/dbconfig/20260527-065526-fceratto.json * 06:54 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1025.eqiad.wmnet * 06:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P93168 and previous config saved to /var/cache/conftool/dbconfig/20260527-064519-fceratto.json * 06:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P93166 and previous config saved to /var/cache/conftool/dbconfig/20260527-063511-fceratto.json * 06:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93165 and previous config saved to /var/cache/conftool/dbconfig/20260527-062503-fceratto.json * 06:22 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2055: repool after maintenance * 06:21 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 06:21 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2055.codfw.wmnet with OS trixie * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2246 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93163 and previous config saved to /var/cache/conftool/dbconfig/20260527-061643-fceratto.json * 06:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2246.codfw.wmnet with reason: Maintenance * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93162 and previous config saved to /var/cache/conftool/dbconfig/20260527-061613-fceratto.json * 06:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P93161 and previous config saved to /var/cache/conftool/dbconfig/20260527-060606-fceratto.json * 06:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2055.codfw.wmnet with reason: host reimage * 05:56 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2055.codfw.wmnet with reason: host reimage * 05:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P93160 and previous config saved to /var/cache/conftool/dbconfig/20260527-055558-fceratto.json * 05:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93159 and previous config saved to /var/cache/conftool/dbconfig/20260527-054550-fceratto.json * 05:41 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2055.codfw.wmnet with OS trixie * 05:40 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2055: Upgrading es2055.codfw.wmnet * 05:40 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2055: Upgrading es2055.codfw.wmnet * 05:40 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:38 moritzm: remove ganeti1026 from eqiad Ganeti cluster [[phab:T424680|T424680]] * 05:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2245 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93157 and previous config saved to /var/cache/conftool/dbconfig/20260527-053727-fceratto.json * 05:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: Maintenance * 05:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93156 and previous config saved to /var/cache/conftool/dbconfig/20260527-053708-fceratto.json * 05:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P93155 and previous config saved to /var/cache/conftool/dbconfig/20260527-052700-fceratto.json * 05:26 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1014 from dbctl [[phab:T427270|T427270]]', diff saved to https://phabricator.wikimedia.org/P93154 and previous config saved to /var/cache/conftool/dbconfig/20260527-052624-marostegui.json * 05:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P93153 and previous config saved to /var/cache/conftool/dbconfig/20260527-051653-fceratto.json * 05:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93152 and previous config saved to /var/cache/conftool/dbconfig/20260527-050645-fceratto.json * 04:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2237 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93151 and previous config saved to /var/cache/conftool/dbconfig/20260527-045827-fceratto.json * 04:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: Maintenance * 04:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93150 and previous config saved to /var/cache/conftool/dbconfig/20260527-045759-fceratto.json * 04:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P93149 and previous config saved to /var/cache/conftool/dbconfig/20260527-044751-fceratto.json * 04:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P93148 and previous config saved to /var/cache/conftool/dbconfig/20260527-043744-fceratto.json * 04:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93147 and previous config saved to /var/cache/conftool/dbconfig/20260527-042737-fceratto.json * 04:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2236 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93146 and previous config saved to /var/cache/conftool/dbconfig/20260527-041921-fceratto.json * 04:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2236.codfw.wmnet with reason: Maintenance * 04:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93145 and previous config saved to /var/cache/conftool/dbconfig/20260527-041852-fceratto.json * 04:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P93144 and previous config saved to /var/cache/conftool/dbconfig/20260527-040844-fceratto.json * 03:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P93143 and previous config saved to /var/cache/conftool/dbconfig/20260527-035836-fceratto.json * 03:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93142 and previous config saved to /var/cache/conftool/dbconfig/20260527-034828-fceratto.json * 03:40 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93141 and previous config saved to /var/cache/conftool/dbconfig/20260527-034008-fceratto.json * 03:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2219.codfw.wmnet with reason: Maintenance * 03:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93140 and previous config saved to /var/cache/conftool/dbconfig/20260527-033938-fceratto.json * 03:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P93139 and previous config saved to /var/cache/conftool/dbconfig/20260527-032931-fceratto.json * 03:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P93138 and previous config saved to /var/cache/conftool/dbconfig/20260527-031923-fceratto.json * 03:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93137 and previous config saved to /var/cache/conftool/dbconfig/20260527-030915-fceratto.json * 03:00 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93136 and previous config saved to /var/cache/conftool/dbconfig/20260527-030045-fceratto.json * 03:00 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2210.codfw.wmnet with reason: Maintenance * 03:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93135 and previous config saved to /var/cache/conftool/dbconfig/20260527-030016-fceratto.json * 02:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P93134 and previous config saved to /var/cache/conftool/dbconfig/20260527-025008-fceratto.json * 02:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P93133 and previous config saved to /var/cache/conftool/dbconfig/20260527-024000-fceratto.json * 02:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93132 and previous config saved to /var/cache/conftool/dbconfig/20260527-022953-fceratto.json * 02:21 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93131 and previous config saved to /var/cache/conftool/dbconfig/20260527-022133-fceratto.json * 02:21 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2206.codfw.wmnet with reason: Maintenance * 02:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93130 and previous config saved to /var/cache/conftool/dbconfig/20260527-022100-fceratto.json * 02:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P93129 and previous config saved to /var/cache/conftool/dbconfig/20260527-021053-fceratto.json * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 29s) * 02:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P93128 and previous config saved to /var/cache/conftool/dbconfig/20260527-020045-fceratto.json * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93127 and previous config saved to /var/cache/conftool/dbconfig/20260527-015037-fceratto.json * 01:42 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93126 and previous config saved to /var/cache/conftool/dbconfig/20260527-014204-fceratto.json * 01:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2179.codfw.wmnet with reason: Maintenance * 01:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93125 and previous config saved to /var/cache/conftool/dbconfig/20260527-014134-fceratto.json * 01:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P93124 and previous config saved to /var/cache/conftool/dbconfig/20260527-013126-fceratto.json * 01:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P93123 and previous config saved to /var/cache/conftool/dbconfig/20260527-012119-fceratto.json * 01:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93122 and previous config saved to /var/cache/conftool/dbconfig/20260527-011111-fceratto.json * 01:02 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2172 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93121 and previous config saved to /var/cache/conftool/dbconfig/20260527-010234-fceratto.json * 01:02 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance * 01:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93120 and previous config saved to /var/cache/conftool/dbconfig/20260527-010205-fceratto.json * 00:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P93119 and previous config saved to /var/cache/conftool/dbconfig/20260527-005157-fceratto.json * 00:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P93118 and previous config saved to /var/cache/conftool/dbconfig/20260527-004149-fceratto.json * 00:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93117 and previous config saved to /var/cache/conftool/dbconfig/20260527-003141-fceratto.json * 00:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2155 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93116 and previous config saved to /var/cache/conftool/dbconfig/20260527-002309-fceratto.json * 00:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance * 00:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93115 and previous config saved to /var/cache/conftool/dbconfig/20260527-002228-fceratto.json * 00:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P93114 and previous config saved to /var/cache/conftool/dbconfig/20260527-001220-fceratto.json * 00:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P93113 and previous config saved to /var/cache/conftool/dbconfig/20260527-000209-fceratto.json == 2026-05-26 == * 23:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93112 and previous config saved to /var/cache/conftool/dbconfig/20260526-235201-fceratto.json * 23:44 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93111 and previous config saved to /var/cache/conftool/dbconfig/20260526-234451-fceratto.json * 23:44 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance * 23:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93110 and previous config saved to /var/cache/conftool/dbconfig/20260526-234421-fceratto.json * 23:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P93109 and previous config saved to /var/cache/conftool/dbconfig/20260526-233414-fceratto.json * 23:27 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5026.* * 23:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P93108 and previous config saved to /var/cache/conftool/dbconfig/20260526-232406-fceratto.json * 23:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93107 and previous config saved to /var/cache/conftool/dbconfig/20260526-231358-fceratto.json * 23:07 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5026.* * 23:06 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93106 and previous config saved to /var/cache/conftool/dbconfig/20260526-230650-fceratto.json * 23:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance * 23:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93105 and previous config saved to /var/cache/conftool/dbconfig/20260526-230620-fceratto.json * 22:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P93104 and previous config saved to /var/cache/conftool/dbconfig/20260526-225612-fceratto.json * 22:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P93103 and previous config saved to /var/cache/conftool/dbconfig/20260526-224604-fceratto.json * 22:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93101 and previous config saved to /var/cache/conftool/dbconfig/20260526-223556-fceratto.json * 22:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93100 and previous config saved to /var/cache/conftool/dbconfig/20260526-222848-fceratto.json * 22:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance * 22:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93099 and previous config saved to /var/cache/conftool/dbconfig/20260526-222828-fceratto.json * 22:23 robh@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts cp6015.drmrs.wmnet * 22:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P93098 and previous config saved to /var/cache/conftool/dbconfig/20260526-221819-fceratto.json * 22:10 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1009.eqiad.wmnet with OS trixie * 22:08 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1008.eqiad.wmnet with OS trixie * 22:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P93097 and previous config saved to /var/cache/conftool/dbconfig/20260526-220811-fceratto.json * 22:04 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293701{{!}}MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] (duration: 09m 30s) * 22:03 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1009.eqiad.wmnet with reason: host reimage * 22:00 egardner@deploy1003: egardner, mfossati: Continuing with deployment * 21:59 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1008.eqiad.wmnet with reason: host reimage * 21:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93096 and previous config saved to /var/cache/conftool/dbconfig/20260526-215803-fceratto.json * 21:57 egardner@deploy1003: egardner, mfossati: Backport for [[gerrit:1293701{{!}}MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:56 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp6015.drmrs.wmnet * 21:56 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1010.eqiad.wmnet with OS trixie * 21:56 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp6015.drmrs.wmnet * 21:55 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1293701{{!}}MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] * 21:54 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1009.eqiad.wmnet with reason: host reimage * 21:51 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1008.eqiad.wmnet with reason: host reimage * 21:50 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93095 and previous config saved to /var/cache/conftool/dbconfig/20260526-215043-fceratto.json * 21:50 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance * 21:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93094 and previous config saved to /var/cache/conftool/dbconfig/20260526-215011-fceratto.json * 21:49 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1010.eqiad.wmnet with reason: host reimage * 21:47 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp6015.drmrs.wmnet * 21:44 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1009 * 21:44 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host relforge1009 * 21:43 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host relforge1009 * 21:43 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) relforge1009.eqiad.wmnet 120.48.64.10.in-addr.arpa 0.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:43 bking@cumin2002: START - Cookbook sre.dns.wipe-cache relforge1009.eqiad.wmnet 120.48.64.10.in-addr.arpa 0.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:43 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:42 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1009 - bking@cumin2002" * 21:42 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1010.eqiad.wmnet with reason: host reimage * 21:42 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1009 - bking@cumin2002" * 21:41 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1008 * 21:40 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host relforge1008 * 21:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P93093 and previous config saved to /var/cache/conftool/dbconfig/20260526-214003-fceratto.json * 21:36 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host relforge1008 * 21:36 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) relforge1008.eqiad.wmnet 100.32.64.10.in-addr.arpa 0.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:36 bking@cumin2002: START - Cookbook sre.dns.wipe-cache relforge1008.eqiad.wmnet 100.32.64.10.in-addr.arpa 0.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:36 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:36 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1008 - bking@cumin2002" * 21:36 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1008 - bking@cumin2002" * 21:35 bking@cumin2002: START - Cookbook sre.dns.netbox * 21:32 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1010 * 21:32 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1010 * 21:31 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1010.eqiad.wmnet with OS trixie * 21:31 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1009 * 21:30 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1009.eqiad.wmnet with OS trixie * 21:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P93092 and previous config saved to /var/cache/conftool/dbconfig/20260526-212955-fceratto.json * 21:29 bking@cumin2002: START - Cookbook sre.dns.netbox * 21:29 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1008 * 21:29 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1008.eqiad.wmnet with OS trixie * 21:27 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist "all.dblist - mediamoderation-continuous-scan.dblist - preinstall.dblist" extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` in tmux session - [[phab:T421688|T421688]] * 21:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93091 and previous config saved to /var/cache/conftool/dbconfig/20260526-211948-fceratto.json * 21:19 jhathaway: dmarc ingress test run mx-in1001 * 21:15 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_codfw and A:cp * 21:15 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2057.codfw.wmnet * 21:14 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_codfw and A:cp * 21:14 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2058.codfw.wmnet * 21:12 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2222 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93090 and previous config saved to /var/cache/conftool/dbconfig/20260526-211238-fceratto.json * 21:12 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance * 21:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93089 and previous config saved to /var/cache/conftool/dbconfig/20260526-211207-fceratto.json * 21:06 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 21:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P93088 and previous config saved to /var/cache/conftool/dbconfig/20260526-210159-fceratto.json * 20:55 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on phab2003.codfw.wmnet with reason: WIP * 20:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P93087 and previous config saved to /var/cache/conftool/dbconfig/20260526-205152-fceratto.json * 20:50 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:50 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 20:50 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 20:45 dzahn@cumin2002: START - Cookbook sre.dns.netbox * 20:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93086 and previous config saved to /var/cache/conftool/dbconfig/20260526-204143-fceratto.json * 20:38 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2055.codfw.wmnet * 20:34 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93085 and previous config saved to /var/cache/conftool/dbconfig/20260526-203430-fceratto.json * 20:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance * 20:34 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2056.codfw.wmnet * 20:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93084 and previous config saved to /var/cache/conftool/dbconfig/20260526-203357-fceratto.json * 20:32 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:32 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:32 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:31 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P93083 and previous config saved to /var/cache/conftool/dbconfig/20260526-202349-fceratto.json * 20:18 alexsanford@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293161{{!}}Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794{{!}}Re-enable ReadingLists survey on beta cluster (T426781)]] (duration: 09m 14s) * 20:14 alexsanford@deploy1003: alexsanford, aude: Continuing with deployment * 20:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P93082 and previous config saved to /var/cache/conftool/dbconfig/20260526-201341-fceratto.json * 20:11 alexsanford@deploy1003: alexsanford, aude: Backport for [[gerrit:1293161{{!}}Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794{{!}}Re-enable ReadingLists survey on beta cluster (T426781)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:09 alexsanford@deploy1003: Started scap sync-world: Backport for [[gerrit:1293161{{!}}Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794{{!}}Re-enable ReadingLists survey on beta cluster (T426781)]] * 20:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93081 and previous config saved to /var/cache/conftool/dbconfig/20260526-200333-fceratto.json * 19:59 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2053.codfw.wmnet * 19:58 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2029.codfw.wmnet with OS trixie * 19:57 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2028.codfw.wmnet with OS trixie * 19:56 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2208 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93080 and previous config saved to /var/cache/conftool/dbconfig/20260526-195632-fceratto.json * 19:56 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance * 19:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93079 and previous config saved to /var/cache/conftool/dbconfig/20260526-195557-fceratto.json * 19:55 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2054.codfw.wmnet * 19:51 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:51 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P93078 and previous config saved to /var/cache/conftool/dbconfig/20260526-194549-fceratto.json * 19:45 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 19:44 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:43 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2029 * 19:43 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 * 19:43 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 * 19:43 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 19:40 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb2014.codfw.wmnet with OS trixie * 19:40 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:40 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb2013.codfw.wmnet with OS trixie * 19:40 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:39 brett@cumin2002: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 19:38 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 19:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P93077 and previous config saved to /var/cache/conftool/dbconfig/20260526-193541-fceratto.json * 19:35 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:35 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 19:30 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 19:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93076 and previous config saved to /var/cache/conftool/dbconfig/20260526-192533-fceratto.json * 19:24 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:21 dzahn@cumin2002: START - Cookbook sre.dns.netbox * 19:20 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2051.codfw.wmnet * 19:19 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:19 brett@cumin2002: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 19:18 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93075 and previous config saved to /var/cache/conftool/dbconfig/20260526-191818-fceratto.json * 19:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance * 19:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93074 and previous config saved to /var/cache/conftool/dbconfig/20260526-191748-fceratto.json * 19:16 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2052.codfw.wmnet * 19:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P93073 and previous config saved to /var/cache/conftool/dbconfig/20260526-190740-fceratto.json * 19:07 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb2014.codfw.wmnet with reason: host reimage * 19:03 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb2013.codfw.wmnet with reason: host reimage * 18:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1026.eqiad.wmnet * 18:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P93072 and previous config saved to /var/cache/conftool/dbconfig/20260526-185732-fceratto.json * 18:56 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2014.codfw.wmnet with reason: host reimage * 18:56 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2013.codfw.wmnet with reason: host reimage * 18:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93071 and previous config saved to /var/cache/conftool/dbconfig/20260526-184724-fceratto.json * 18:44 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb2014.codfw.wmnet with OS trixie * 18:43 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb2013.codfw.wmnet with OS trixie * 18:41 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host rdb2014.codfw.wmnet with OS trixie * 18:41 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2049.codfw.wmnet * 18:40 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93070 and previous config saved to /var/cache/conftool/dbconfig/20260526-184009-fceratto.json * 18:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance * 18:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93069 and previous config saved to /var/cache/conftool/dbconfig/20260526-183939-fceratto.json * 18:37 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2050.codfw.wmnet * 18:30 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 18:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P93068 and previous config saved to /var/cache/conftool/dbconfig/20260526-182931-fceratto.json * 18:29 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:29 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_magru-v4 - dzahn@cumin2002" * 18:29 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_magru-v4 - dzahn@cumin2002" * 18:24 dzahn@cumin2002: START - Cookbook sre.dns.netbox * 18:21 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 18:21 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 18:21 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 18:20 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 18:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P93066 and previous config saved to /var/cache/conftool/dbconfig/20260526-181923-fceratto.json * 18:15 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 18:15 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 18:15 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 18:15 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 18:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93065 and previous config saved to /var/cache/conftool/dbconfig/20260526-180915-fceratto.json * 18:02 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93064 and previous config saved to /var/cache/conftool/dbconfig/20260526-180205-fceratto.json * 18:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2159.codfw.wmnet with reason: Maintenance * 18:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93063 and previous config saved to /var/cache/conftool/dbconfig/20260526-180132-fceratto.json * 18:00 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2047.codfw.wmnet * 17:59 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2048.codfw.wmnet * 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:54 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:54 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:54 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P93062 and previous config saved to /var/cache/conftool/dbconfig/20260526-175124-fceratto.json * 17:42 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293779{{!}}Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] (duration: 07m 25s) * 17:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P93060 and previous config saved to /var/cache/conftool/dbconfig/20260526-174117-fceratto.json * 17:39 mvernon@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ms-be2089.codfw.wmnet * 17:37 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 17:37 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:36 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:36 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:36 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1293779{{!}}Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:36 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:34 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1293779{{!}}Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] * 17:33 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:33 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:33 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:33 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93059 and previous config saved to /var/cache/conftool/dbconfig/20260526-173109-fceratto.json * 17:27 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:26 jclark@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:25 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:25 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:25 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:24 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:24 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1001 to eqiad - jclark@cumin1003" * 17:24 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:24 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1001 to eqiad - jclark@cumin1003" * 17:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93058 and previous config saved to /var/cache/conftool/dbconfig/20260526-172332-fceratto.json * 17:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2227.codfw.wmnet with reason: Maintenance * 17:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93057 and previous config saved to /var/cache/conftool/dbconfig/20260526-172303-fceratto.json * 17:21 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2045.codfw.wmnet * 17:20 jclark@cumin1003: START - Cookbook sre.dns.netbox * 17:20 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2046.codfw.wmnet * 17:18 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:17 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:17 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:17 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:17 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:17 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:17 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:17 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:16 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:15 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 17:14 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:14 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:14 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:14 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:13 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:13 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:13 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:13 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:13 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P93056 and previous config saved to /var/cache/conftool/dbconfig/20260526-171255-fceratto.json * 17:11 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:11 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:11 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:11 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:07 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:05 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:05 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:05 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:05 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P93055 and previous config saved to /var/cache/conftool/dbconfig/20260526-170247-fceratto.json * 17:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:57 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:55 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:52 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93054 and previous config saved to /var/cache/conftool/dbconfig/20260526-165240-fceratto.json * 16:50 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:50 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:50 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:50 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:45 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:45 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:45 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:45 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:45 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:44 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:44 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2209 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93053 and previous config saved to /var/cache/conftool/dbconfig/20260526-164421-fceratto.json * 16:44 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:44 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1002 to eqiad - jclark@cumin1003" * 16:44 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2209.codfw.wmnet with reason: Maintenance * 16:44 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1002 to eqiad - jclark@cumin1003" * 16:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93052 and previous config saved to /var/cache/conftool/dbconfig/20260526-164352-fceratto.json * 16:42 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2043.codfw.wmnet * 16:41 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2044.codfw.wmnet * 16:40 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:40 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:40 jclark@cumin1003: START - Cookbook sre.dns.netbox * 16:40 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:40 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:40 brett: reboot lvs 101[345].eqiad.wmnet * 16:39 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:39 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:39 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:39 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:37 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:37 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 16:37 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:37 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:37 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:37 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 16:37 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 16:36 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 16:36 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:36 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:36 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:36 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:35 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 16:34 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 16:34 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:34 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:34 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:34 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:33 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_codfw and A:cp * 16:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P93051 and previous config saved to /var/cache/conftool/dbconfig/20260526-163344-fceratto.json * 16:33 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_codfw and A:cp * 16:31 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:31 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:30 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:30 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P93050 and previous config saved to /var/cache/conftool/dbconfig/20260526-162336-fceratto.json * 16:13 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2089.codfw.wmnet * 16:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93049 and previous config saved to /var/cache/conftool/dbconfig/20260526-161328-fceratto.json * 16:11 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:11 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:10 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:10 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:07 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search,name=eqiad * 16:06 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:06 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:06 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:06 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93047 and previous config saved to /var/cache/conftool/dbconfig/20260526-160450-fceratto.json * 16:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2194.codfw.wmnet with reason: Maintenance * 16:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93046 and previous config saved to /var/cache/conftool/dbconfig/20260526-160420-fceratto.json * 16:03 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:03 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:03 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] (duration: 00m 28s) * 16:02 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] * 16:00 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:00 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:00 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:00 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:55 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:55 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:55 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:55 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] (duration: 00m 22s) * 15:55 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:55 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] * 15:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P93045 and previous config saved to /var/cache/conftool/dbconfig/20260526-155413-fceratto.json * 15:46 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search,name=eqiad * 15:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P93044 and previous config saved to /var/cache/conftool/dbconfig/20260526-154405-fceratto.json * 15:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93043 and previous config saved to /var/cache/conftool/dbconfig/20260526-153357-fceratto.json * 15:30 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:30 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:30 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:30 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93042 and previous config saved to /var/cache/conftool/dbconfig/20260526-152629-fceratto.json * 15:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2190.codfw.wmnet with reason: Maintenance * 15:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93041 and previous config saved to /var/cache/conftool/dbconfig/20260526-152559-fceratto.json * 15:24 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:24 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:24 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:24 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:23 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:22 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:22 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:22 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P93040 and previous config saved to /var/cache/conftool/dbconfig/20260526-151552-fceratto.json * 15:12 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2196: Rack maintenance completed * 15:10 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2196.codfw.wmnet * 15:10 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2196.codfw.wmnet * 15:07 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw * 15:06 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2222: Rack maintenance completed * 15:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P93037 and previous config saved to /var/cache/conftool/dbconfig/20260526-150546-fceratto.json * 15:04 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2221: Rack maintenance completed * 15:04 brennen@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab1004 for [[phab:T427286|T427286]] (duration: 00m 39s) * 15:03 brennen@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab1004 for [[phab:T427286|T427286]] * 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2002 for [[phab:T427286|T427286]] (duration: 00m 45s) * 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2002 for [[phab:T427286|T427286]] * 15:02 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2002.codfw.wmnet with reason: Phabricator deploy * 15:01 bjensen: uploading prometheus-memcached-exporter_0.16.0-1_amd64 on apt1002 * 15:01 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator deploy * 15:00 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2223: switch maintenance * 14:56 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2196: Rack maintenance completed * 14:55 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2221.codfw.wmnet * 14:55 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2221.codfw.wmnet * 14:55 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2222.codfw.wmnet * 14:55 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2222.codfw.wmnet * 14:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93033 and previous config saved to /var/cache/conftool/dbconfig/20260526-145538-fceratto.json * 14:55 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet * 14:54 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1026.eqiad.wmnet * 14:52 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet * 14:52 moritzm: remove ganeti1025 from eqiad Ganeti cluster [[phab:T424680|T424680]] * 14:51 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2030.codfw.wmnet to cluster codfw and group A * 14:51 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2222: Rack maintenance completed * 14:49 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:49 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2221: Rack maintenance completed * 14:49 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2030.codfw.wmnet to cluster codfw and group A * 14:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2029.codfw.wmnet to cluster codfw and group A * 14:47 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2029.codfw.wmnet to cluster codfw and group A * 14:47 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2177 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93030 and previous config saved to /var/cache/conftool/dbconfig/20260526-144718-fceratto.json * 14:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance * 14:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93029 and previous config saved to /var/cache/conftool/dbconfig/20260526-144651-fceratto.json * 14:45 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=wdqs-scholarly,name=codfw * 14:45 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=wdqs-scholarly,name=codfw * 14:43 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw * 14:40 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:40 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2167: Migration of db2167.codfw.wmnet completed * 14:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P93026 and previous config saved to /var/cache/conftool/dbconfig/20260526-143643-fceratto.json * 14:31 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1054.eqiad.wmnet with OS trixie * 14:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P93023 and previous config saved to /var/cache/conftool/dbconfig/20260526-142636-fceratto.json * 14:26 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:25 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:24 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1014: Rack maintenance completed * 14:24 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.parsercache (exit_code=99) * 14:24 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache * 14:24 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc1014: Rack maintenance completed * 14:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1025.eqiad.wmnet * 14:19 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for backup2015.codfw.wmnet,db2197.codfw.wmnet * 14:19 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for backup2015.codfw.wmnet,db2197.codfw.wmnet * 14:18 jynus: restarting mediabackups@codfw after maintenance on a codfw backup media storage server [[phab:T426199|T426199]] * 14:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93021 and previous config saved to /var/cache/conftool/dbconfig/20260526-141628-fceratto.json * 14:16 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:14 fabfur: repooled cp2043 ([[phab:T426199|T426199]]) * 14:14 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2223: switch maintenance * 14:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1054.eqiad.wmnet with reason: host reimage * 14:14 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2043.* * 14:13 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293710{{!}}Site info should output thumblimits as array (T427066)]] (duration: 06m 40s) * 14:12 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:10 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1054.eqiad.wmnet with reason: host reimage * 14:10 fabfur@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2011.codfw.wmnet * 14:10 fabfur@cumin1003: START - Cookbook sre.hosts.remove-downtime for lvs2011.codfw.wmnet * 14:09 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 14:09 fabfur: restoring lvs2011 as primary ([[phab:T426199|T426199]]) * 14:08 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1293710{{!}}Site info should output thumblimits as array (T427066)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:08 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 14:08 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 14:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93017 and previous config saved to /var/cache/conftool/dbconfig/20260526-140748-fceratto.json * 14:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance * 14:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93016 and previous config saved to /var/cache/conftool/dbconfig/20260526-140718-fceratto.json * 14:07 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1293710{{!}}Site info should output thumblimits as array (T427066)]] * 14:05 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.decommission (exit_code=99) * 14:05 marostegui@cumin1003: Removing pc1013 from zarcillo [[phab:T427190|T427190]] * 14:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1013.eqiad.wmnet * 14:04 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:04 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 14:04 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 14:00 marostegui@cumin1003: START - Cookbook sre.dns.netbox * 13:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P93014 and previous config saved to /var/cache/conftool/dbconfig/20260526-135711-fceratto.json * 13:56 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1054.eqiad.wmnet with OS trixie * 13:55 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2167: Migration of db2167.codfw.wmnet completed * 13:53 Amir1: drop flaggedrevs tables on cawikinews ([[phab:T423577|T423577]]) * 13:49 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1013.eqiad.wmnet * 13:49 marostegui@cumin1003: START - Cookbook sre.mysql.decommission * 13:48 Lucas_WMDE: UTC afternoon backport+config window done * 13:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P93012 and previous config saved to /var/cache/conftool/dbconfig/20260526-134703-fceratto.json * 13:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2167.codfw.wmnet with OS trixie * 13:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93011 and previous config saved to /var/cache/conftool/dbconfig/20260526-133656-fceratto.json * 13:36 XioNoX: reboot lsw1-a2-codfw for software upgrade - [[phab:T426199|T426199]] * 13:36 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2223: switch maintenance * 13:35 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2223: switch maintenance * 13:35 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2222: switch maintenance * 13:35 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2222: switch maintenance * 13:35 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2221: switch maintenance * 13:35 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293662{{!}}Enable IRS Direct Reporting on testwiki (T425025)]] (duration: 09m 28s) * 13:34 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2221: switch maintenance * 13:34 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2196: switch maintenance * 13:34 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2196: switch maintenance * 13:31 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 13:30 stran@deploy1003: stran: Continuing with deployment * 13:29 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 13:29 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93006 and previous config saved to /var/cache/conftool/dbconfig/20260526-132927-fceratto.json * 13:29 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2167.codfw.wmnet with reason: host reimage * 13:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance * 13:29 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 34 hosts with reason: Switch maintenance * 13:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93005 and previous config saved to /var/cache/conftool/dbconfig/20260526-132857-fceratto.json * 13:28 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lsw1-a2-codfw,lsw1-a2-codfw IPv6,lsw1-a2-codfw.mgmt with reason: Switch maintenance * 13:27 stran@deploy1003: stran: Backport for [[gerrit:1293662{{!}}Enable IRS Direct Reporting on testwiki (T425025)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:25 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1293662{{!}}Enable IRS Direct Reporting on testwiki (T425025)]] * 13:25 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2167.codfw.wmnet with reason: host reimage * 13:22 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293706{{!}}Disable the `no` language code for translation (T424613)]] (duration: 08m 30s) * 13:22 ladsgroup@dns1004: END - running authdns-update * 13:20 ladsgroup@dns1004: START - running authdns-update * 13:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P93004 and previous config saved to /var/cache/conftool/dbconfig/20260526-131850-fceratto.json * 13:18 lucaswerkmeister-wmde@deploy1003: jhsoby, lucaswerkmeister-wmde: Continuing with deployment * 13:16 lucaswerkmeister-wmde@deploy1003: jhsoby, lucaswerkmeister-wmde: Backport for [[gerrit:1293706{{!}}Disable the `no` language code for translation (T424613)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:14 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1293706{{!}}Disable the `no` language code for translation (T424613)]] * 13:12 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293177{{!}}Instrumentation: log new articles namespace and source (T422146)]] (duration: 07m 09s) * 13:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P93003 and previous config saved to /var/cache/conftool/dbconfig/20260526-130842-fceratto.json * 13:08 sbisson@deploy1003: sbisson: Continuing with deployment * 13:07 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2167.codfw.wmnet with OS trixie * 13:07 sbisson@deploy1003: sbisson: Backport for [[gerrit:1293177{{!}}Instrumentation: log new articles namespace and source (T422146)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:05 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2167: Upgrading db2167.codfw.wmnet * 13:05 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1293177{{!}}Instrumentation: log new articles namespace and source (T422146)]] * 13:04 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2167: Upgrading db2167.codfw.wmnet * 13:04 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:04 kart_: Update Recommendation API to 2026-05-26-074931-production * 13:03 kartik@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 13:00 topranks: deactivate CR BGP to doh2002 to test backup path via doh2001 * 12:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93000 and previous config saved to /var/cache/conftool/dbconfig/20260526-125834-fceratto.json * 12:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2226 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92999 and previous config saved to /var/cache/conftool/dbconfig/20260526-125135-fceratto.json * 12:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance * 12:51 kartik@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92998 and previous config saved to /var/cache/conftool/dbconfig/20260526-125105-fceratto.json * 12:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P92997 and previous config saved to /var/cache/conftool/dbconfig/20260526-124059-fceratto.json * 12:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2003.wikimedia.org * 12:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1214: Migration of db1214.eqiad.wmnet completed * 12:33 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc2003.wikimedia.org * 12:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P92995 and previous config saved to /var/cache/conftool/dbconfig/20260526-123052-fceratto.json * 12:26 fabfur: depooled cp204 for network activity ([[phab:T426199|T426199]]) * 12:26 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2043.* * 12:24 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-a1-codfw,ssw1-a1-codfw IPv6,ssw1-a1-codfw.mgmt with reason: Switch maintenance * 12:24 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mirror1001.wikimedia.org * 12:23 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:23 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:22 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92993 and previous config saved to /var/cache/conftool/dbconfig/20260526-122044-fceratto.json * 12:20 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:19 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mirror1001.wikimedia.org * 12:13 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2225 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92991 and previous config saved to /var/cache/conftool/dbconfig/20260526-121336-fceratto.json * 12:13 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance * 12:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92990 and previous config saved to /var/cache/conftool/dbconfig/20260526-121306-fceratto.json * 12:09 fabfur@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2011.codfw.wmnet with reason: Planned downtime for rack maintenance * 12:08 fabfur: downtime, disable puppet and stop pybal for rack maintenance ([[phab:T426199|T426199]]) * 12:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2181: Migration of db2181.codfw.wmnet completed * 12:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P92987 and previous config saved to /var/cache/conftool/dbconfig/20260526-120258-fceratto.json * 12:01 XioNoX: start ssw1-a1-codfw network maintenance (no impact expected as the spines are redundant) * 11:59 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293167{{!}}hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055{{!}}hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] (duration: 15m 26s) * 11:56 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on backup2015.codfw.wmnet,db2197.codfw.wmnet with reason: network maintenance * 11:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1005.eqiad.wmnet * 11:55 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment * 11:54 jynus: stopping mediabackups@codfw for maintenance on a codfw backup media storage server [[phab:T426199|T426199]] * 11:54 jmm@dns1004: END - running authdns-update * 11:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P92985 and previous config saved to /var/cache/conftool/dbconfig/20260526-115251-fceratto.json * 11:52 jmm@dns1004: START - running authdns-update * 11:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1005.eqiad.wmnet * 11:49 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1214: Migration of db1214.eqiad.wmnet completed * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1004.eqiad.wmnet * 11:47 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet * 11:46 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1293167{{!}}hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055{{!}}hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:45 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1004.eqiad.wmnet * 11:44 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1293167{{!}}hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055{{!}}hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] * 11:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92983 and previous config saved to /var/cache/conftool/dbconfig/20260526-114243-fceratto.json * 11:42 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet * 11:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1214.eqiad.wmnet with OS trixie * 11:35 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293691{{!}}Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] (duration: 06m 46s) * 11:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92981 and previous config saved to /var/cache/conftool/dbconfig/20260526-113542-fceratto.json * 11:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance * 11:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92980 and previous config saved to /var/cache/conftool/dbconfig/20260526-113521-fceratto.json * 11:31 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 11:31 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1293691{{!}}Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1222: Migration of db1222.eqiad.wmnet completed * 11:29 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1293691{{!}}Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] * 11:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P92978 and previous config saved to /var/cache/conftool/dbconfig/20260526-112513-fceratto.json * 11:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1214.eqiad.wmnet with reason: host reimage * 11:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc4 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92977 and previous config saved to /var/cache/conftool/dbconfig/20260526-112326-marostegui.json * 11:22 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2181: Migration of db2181.codfw.wmnet completed * 11:22 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1024 to dbctl [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92975 and previous config saved to /var/cache/conftool/dbconfig/20260526-112215-marostegui.json * 11:20 fceratto@cumin1003: dbctl commit (dc=all): 'Switchover es2042 es2041 for [[phab:T426199|T426199]]', diff saved to https://phabricator.wikimedia.org/P92974 and previous config saved to /var/cache/conftool/dbconfig/20260526-112028-fceratto.json * 11:17 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1214.eqiad.wmnet with reason: host reimage * 11:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P92972 and previous config saved to /var/cache/conftool/dbconfig/20260526-111506-fceratto.json * 11:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2181.codfw.wmnet with OS trixie * 11:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92971 and previous config saved to /var/cache/conftool/dbconfig/20260526-110458-fceratto.json * 11:02 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1214.eqiad.wmnet with OS trixie * 11:00 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293095{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] (duration: 15m 50s) * 11:00 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1214: Upgrading db1214.eqiad.wmnet * 10:59 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1214: Upgrading db1214.eqiad.wmnet * 10:59 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92968 and previous config saved to /var/cache/conftool/dbconfig/20260526-105755-fceratto.json * 10:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance * 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92967 and previous config saved to /var/cache/conftool/dbconfig/20260526-105726-fceratto.json * 10:56 jiji@deploy1003: jiji: Continuing with deployment * 10:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2181.codfw.wmnet with reason: host reimage * 10:51 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2181.codfw.wmnet with reason: host reimage * 10:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P92966 and previous config saved to /var/cache/conftool/dbconfig/20260526-104718-fceratto.json * 10:46 jiji@deploy1003: jiji: Backport for [[gerrit:1293095{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:44 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1293095{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] * 10:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P92964 and previous config saved to /var/cache/conftool/dbconfig/20260526-103711-fceratto.json * 10:36 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2181.codfw.wmnet with OS trixie * 10:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply * 10:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply * 10:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92963 and previous config saved to /var/cache/conftool/dbconfig/20260526-102703-fceratto.json * 10:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1226: Migration of db1226.eqiad.wmnet completed * 10:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2181: Upgrading db2181.codfw.wmnet * 10:24 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2181: Upgrading db2181.codfw.wmnet * 10:24 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92960 and previous config saved to /var/cache/conftool/dbconfig/20260526-101936-fceratto.json * 10:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance * 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92959 and previous config saved to /var/cache/conftool/dbconfig/20260526-101842-fceratto.json * 10:16 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-codfw@codfw * 10:16 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 10:15 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 10:10 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293668{{!}}hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] (duration: 06m 42s) * 10:09 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-codfw@codfw * 10:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92957 and previous config saved to /var/cache/conftool/dbconfig/20260526-100834-fceratto.json * 10:06 kharlan@deploy1003: kharlan: Continuing with deployment * 10:05 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293668{{!}}hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:03 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293668{{!}}hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] * 10:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2195: Migration of db2195.codfw.wmnet completed * 10:01 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>kubestage200*<nowiki>}</nowiki> and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw) * 10:01 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet * 10:01 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet * 10:00 jmm@cumin2002: END (PASS) - Cookbook sre.netbox.restart-reboot (exit_code=0) rolling reboot on A:netbox * 09:58 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 09:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92955 and previous config saved to /var/cache/conftool/dbconfig/20260526-095827-fceratto.json * 09:58 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 09:58 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 09:57 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 09:56 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-eqiad@eqiad * 09:56 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:55 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 09:55 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:55 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:55 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet * 09:54 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet * 09:54 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet * 09:54 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet * 09:53 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>kubestage100*<nowiki>}</nowiki> and (A:wikikube-staging-master-eqiad or A:wikikube-staging-worker-eqiad) * 09:53 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet * 09:53 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet * 09:52 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-eqiad@eqiad * 09:52 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293665{{!}}hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] (duration: 08m 07s) * 09:51 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2043.* * 09:51 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2044.* * 09:48 fabfur: repooling cp2043 and cp2044 (haproxy-awslc) ([[phab:T419825|T419825]]) * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92953 and previous config saved to /var/cache/conftool/dbconfig/20260526-094819-fceratto.json * 09:47 kharlan@deploy1003: kharlan: Continuing with deployment * 09:46 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet * 09:45 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293665{{!}}hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:44 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs3009.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:44 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293665{{!}}hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] * 09:41 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet * 09:41 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet * 09:41 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet * 09:41 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92951 and previous config saved to /var/cache/conftool/dbconfig/20260526-094115-fceratto.json * 09:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2229.codfw.wmnet with reason: Maintenance * 09:41 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs3009.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92950 and previous config saved to /var/cache/conftool/dbconfig/20260526-094045-fceratto.json * 09:40 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1226: Migration of db1226.eqiad.wmnet completed * 09:39 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-codfw@codfw * 09:39 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:38 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:34 fabfur: depooling cp2044 to install haproxy-awslc ([[phab:T419825|T419825]]) * 09:34 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet * 09:34 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet * 09:34 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2044.* * 09:33 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet * 09:33 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet * 09:33 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet * 09:33 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2043.* * 09:32 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293661{{!}}hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] (duration: 06m 52s) * 09:32 fabfur: depooling cp2043 to install haproxy-awslc ([[phab:T419825|T419825]]) * 09:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1226.eqiad.wmnet with OS trixie * 09:30 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-codfw@codfw * 09:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P92947 and previous config saved to /var/cache/conftool/dbconfig/20260526-093031-fceratto.json * 09:29 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet * 09:29 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet * 09:29 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet * 09:28 kharlan@deploy1003: kharlan: Continuing with deployment * 09:28 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs3008.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:28 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293661{{!}}hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:27 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet * 09:26 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet * 09:26 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet * 09:26 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet * 09:26 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293661{{!}}hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] * 09:25 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs3008.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:25 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs3010.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet * 09:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet * 09:21 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs3010.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:20 fabfur: start rebooting esams liberica instances ([[phab:T426563|T426563]]) * 09:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P92946 and previous config saved to /var/cache/conftool/dbconfig/20260526-092024-fceratto.json * 09:20 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet * 09:16 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2195: Migration of db2195.codfw.wmnet completed * 09:15 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet * 09:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1226.eqiad.wmnet with reason: host reimage * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>kubestage100*<nowiki>}</nowiki> and (A:wikikube-staging-master-eqiad or A:wikikube-staging-worker-eqiad) * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>kubestage200*<nowiki>}</nowiki> and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw) * 09:14 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293658{{!}}Fix TypeError in Mandatory2FAChecker (T427251)]] (duration: 06m 47s) * 09:10 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1226.eqiad.wmnet with reason: host reimage * 09:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92944 and previous config saved to /var/cache/conftool/dbconfig/20260526-091016-fceratto.json * 09:09 mszwarc@deploy1003: mszwarc: Continuing with deployment * 09:09 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1293658{{!}}Fix TypeError in Mandatory2FAChecker (T427251)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2195.codfw.wmnet with OS trixie * 09:07 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293658{{!}}Fix TypeError in Mandatory2FAChecker (T427251)]] * 09:06 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs4009.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 09:03 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92943 and previous config saved to /var/cache/conftool/dbconfig/20260526-090315-fceratto.json * 09:03 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2224.codfw.wmnet with reason: Maintenance * 09:03 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs4009.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92942 and previous config saved to /var/cache/conftool/dbconfig/20260526-090256-fceratto.json * 08:57 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs4008.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 08:56 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox.discovery.wmnet. on all recursors * 08:56 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors * 08:55 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1226.eqiad.wmnet with OS trixie * 08:53 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs4008.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 08:53 fabfur: start rebooting ulsfo liberica instances ([[phab:T426563|T426563]]) * 08:53 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293594{{!}}Allow to remove passkeys when there's only one standard 2FA method (T426872)]] (duration: 07m 23s) * 08:53 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs5005.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:53 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1226: Upgrading db1226.eqiad.wmnet * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P92941 and previous config saved to /var/cache/conftool/dbconfig/20260526-085248-fceratto.json * 08:51 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox.discovery.wmnet. on all recursors * 08:51 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors * 08:51 jmm@cumin2002: START - Cookbook sre.netbox.restart-reboot rolling reboot on A:netbox * 08:50 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1226: Upgrading db1226.eqiad.wmnet * 08:50 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs5005.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:50 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2195.codfw.wmnet with reason: host reimage * 08:49 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1222: Migration of db1222.eqiad.wmnet completed * 08:48 mszwarc@deploy1003: mszwarc: Continuing with deployment * 08:47 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1293594{{!}}Allow to remove passkeys when there's only one standard 2FA method (T426872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:46 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293594{{!}}Allow to remove passkeys when there's only one standard 2FA method (T426872)]] * 08:43 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs5004.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet * 08:43 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2195.codfw.wmnet with reason: host reimage * 08:43 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1292032{{!}}Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964{{!}}hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] (duration: 09m 56s) * 08:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P92939 and previous config saved to /var/cache/conftool/dbconfig/20260526-084240-fceratto.json * 08:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1222.eqiad.wmnet with OS trixie * 08:40 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs5004.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:40 fabfur: start rebooting eqsin liberica instances ([[phab:T426563|T426563]]) * 08:39 kartik@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 08:39 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet * 08:39 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 08:39 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs5006.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:35 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs5006.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1024.eqiad.wmnet * 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1024.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 08:35 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1292032{{!}}Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964{{!}}hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:33 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs6002.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:33 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1292032{{!}}Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964{{!}}hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] * 08:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92938 and previous config saved to /var/cache/conftool/dbconfig/20260526-083233-fceratto.json * 08:30 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs6002.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:25 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2217 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92937 and previous config saved to /var/cache/conftool/dbconfig/20260526-082531-fceratto.json * 08:25 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2217.codfw.wmnet with reason: Maintenance * 08:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92936 and previous config saved to /var/cache/conftool/dbconfig/20260526-082458-fceratto.json * 08:23 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2195.codfw.wmnet with OS trixie * 08:23 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage * 08:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2195: Upgrading db2195.codfw.wmnet * 08:20 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2195: Upgrading db2195.codfw.wmnet * 08:19 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:18 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage * 08:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P92934 and previous config saved to /var/cache/conftool/dbconfig/20260526-081451-fceratto.json * 08:13 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs6001.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 08:10 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs6001.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:09 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1024.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 08:04 jmm@cumin2002: START - Cookbook sre.dns.netbox * 08:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P92932 and previous config saved to /var/cache/conftool/dbconfig/20260526-080443-fceratto.json * 08:01 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1222.eqiad.wmnet with OS trixie * 08:00 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs6003.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet * 07:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet * 07:59 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1024.eqiad.wmnet * 07:59 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1023.eqiad.wmnet * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1023.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:59 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 07:59 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:58 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1023.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:56 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs6003.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 07:56 fabfur: start rebooting drmrs liberica instances ([[phab:T426563|T426563]]) * 07:56 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs7002.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:54 jmm@cumin2002: START - Cookbook sre.dns.netbox * 07:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92931 and previous config saved to /var/cache/conftool/dbconfig/20260526-075435-fceratto.json * 07:52 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs7002.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:51 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1047.eqiad.wmnet * 07:51 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:51 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1047.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:49 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1023.eqiad.wmnet * 07:47 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2193 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92930 and previous config saved to /var/cache/conftool/dbconfig/20260526-074739-fceratto.json * 07:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2193.codfw.wmnet with reason: Maintenance * 07:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92929 and previous config saved to /var/cache/conftool/dbconfig/20260526-074710-fceratto.json * 07:46 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet * 07:45 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet * 07:45 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:45 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs7001.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:44 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1025.eqiad.wmnet * 07:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:43 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:41 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs7001.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:40 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1046.eqiad.wmnet * 07:40 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1046.eqiad.wmnet * 07:38 arthurtaylor@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291951{{!}}Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] (duration: 12m 01s) * 07:38 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1047.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P92928 and previous config saved to /var/cache/conftool/dbconfig/20260526-073702-fceratto.json * 07:37 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet * 07:36 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet * 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance * 07:35 fabfur: start rebooting magru liberica instances ([[phab:T426563|T426563]]) * 07:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92926 and previous config saved to /var/cache/conftool/dbconfig/20260526-073459-fceratto.json * 07:32 arthurtaylor@deploy1003: arthurtaylor: Continuing with deployment * 07:31 arthurtaylor@deploy1003: arthurtaylor: Backport for [[gerrit:1291951{{!}}Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:30 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1046.eqiad.wmnet * 07:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20260526-072643-fceratto.json * 07:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1046.eqiad.wmnet * 07:26 arthurtaylor@deploy1003: Started scap sync-world: Backport for [[gerrit:1291951{{!}}Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] * 07:25 jiji@cumin1003: START - Cookbook sre.dns.netbox * 07:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P92924 and previous config saved to /var/cache/conftool/dbconfig/20260526-072452-fceratto.json * 07:24 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet * 07:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1047.eqiad.wmnet * 07:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1047.eqiad.wmnet * 07:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92923 and previous config saved to /var/cache/conftool/dbconfig/20260526-071635-fceratto.json * 07:15 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet * 07:15 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1026.eqiad.wmnet * 07:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P92922 and previous config saved to /var/cache/conftool/dbconfig/20260526-071444-fceratto.json * 07:13 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet * 07:11 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1025.eqiad.wmnet * 07:10 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1025.eqiad.wmnet * 07:09 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92921 and previous config saved to /var/cache/conftool/dbconfig/20260526-070946-fceratto.json * 07:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2180.codfw.wmnet with reason: Maintenance * 07:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92920 and previous config saved to /var/cache/conftool/dbconfig/20260526-070916-fceratto.json * 07:09 moritzm: failover Ganeti master in eqiad to ganeti1048 * 07:09 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1047.eqiad.wmnet * 07:07 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1046.eqiad.wmnet * 07:07 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:06 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1046.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org * 07:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92919 and previous config saved to /var/cache/conftool/dbconfig/20260526-070436-fceratto.json * 07:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet * 07:04 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1046.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet * 07:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org * 06:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P92918 and previous config saved to /var/cache/conftool/dbconfig/20260526-065909-fceratto.json * 06:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org * 06:58 jiji@cumin1003: START - Cookbook sre.dns.netbox * 06:58 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet * 06:55 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet * 06:53 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1046.eqiad.wmnet * 06:53 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1045.eqiad.wmnet * 06:53 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:53 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1045.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 06:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org * 06:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P92917 and previous config saved to /var/cache/conftool/dbconfig/20260526-064901-fceratto.json * 06:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92916 and previous config saved to /var/cache/conftool/dbconfig/20260526-064833-fceratto.json * 06:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1222.eqiad.wmnet with reason: Maintenance * 06:47 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1222: Switchover * 06:41 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92914 and previous config saved to /var/cache/conftool/dbconfig/20260526-063853-fceratto.json * 06:35 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 06:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92912 and previous config saved to /var/cache/conftool/dbconfig/20260526-063155-fceratto.json * 06:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance * 06:28 fceratto@cumin1003: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance * 06:23 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1222: Switchover * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1222 [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92910 and previous config saved to /var/cache/conftool/dbconfig/20260526-061656-fceratto.json * 06:15 fceratto@dns1005: END - running authdns-update * 06:14 fceratto@dns1005: START - running authdns-update * 06:11 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1162 to s2 primary and set section read-write [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92909 and previous config saved to /var/cache/conftool/dbconfig/20260526-061114-fceratto.json * 06:10 fceratto@cumin1003: dbctl commit (dc=all): 'Set s2 eqiad as read-only for maintenance - [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92908 and previous config saved to /var/cache/conftool/dbconfig/20260526-061021-fceratto.json * 06:10 federico3: Starting s2 eqiad failover from db1222 to db1162 - [[phab:T425622|T425622]] * 06:04 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1162 with weight 0 [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92907 and previous config saved to /var/cache/conftool/dbconfig/20260526-060443-fceratto.json * 06:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s2 [[phab:T425622|T425622]] * 06:02 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:02 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:01 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:00 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 05:15 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1014.eqiad.wmnet: Maintenance on pc4 * 05:15 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 05:15 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 05:15 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1014.eqiad.wmnet: Maintenance on pc4 * 05:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2024.codfw.wmnet,pc[1014,1024].eqiad.wmnet with reason: Maintenance on pc4 * 04:37 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 04:34 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.1 (duration: 02m 32s) * 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] (duration: 36m 24s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 20s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-25 == * 21:00 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1045.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:49 jiji@cumin1003: START - Cookbook sre.dns.netbox * 20:38 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1045.eqiad.wmnet * 20:37 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1044.eqiad.wmnet * 20:37 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:37 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1044.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:25 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1044.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:15 moritzm: truncate krb5kdc.log1 (which made log rotation fail) * 20:06 jiji@cumin1003: START - Cookbook sre.dns.netbox * 19:57 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1044.eqiad.wmnet * 19:25 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1043.eqiad.wmnet * 19:25 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:25 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1043.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 19:22 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1043.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 18:49 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_eqiad * 18:49 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1115.eqiad.wmnet * 18:34 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5023.eqsin.wmnet [reason: manually pooling after reboot as icinga was down] * 18:33 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5030.eqsin.wmnet [reason: manually pooling after reboot as icinga was down] * 18:22 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp5030*<nowiki>}</nowiki> and A:cp * 18:22 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5030.eqsin.wmnet * 18:15 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp5023*<nowiki>}</nowiki> and A:cp * 18:15 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5023.eqsin.wmnet * 18:10 jiji@cumin1003: START - Cookbook sre.dns.netbox * 18:10 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp5030*<nowiki>}</nowiki> and A:cp * 18:09 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp1113*<nowiki>}</nowiki> and A:cp * 18:09 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet * 18:09 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet * 18:03 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp1113*<nowiki>}</nowiki> and A:cp * 18:02 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp5023*<nowiki>}</nowiki> and A:cp * 18:01 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-text_eqiad * 18:01 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-upload_eqsin * 18:01 sukhe: sre.cdn.roll-reboot cookbooks stalled due to icinga reboot * 18:00 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-text_eqsin * 17:35 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1043.eqiad.wmnet * 17:31 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1110.eqiad.wmnet [reason: manually pooling after reboot as icinga was down] * 17:30 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1042.eqiad.wmnet * 17:30 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:30 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1042.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:29 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1111.eqiad.wmnet * 17:28 sukhe: sukhe@alert1002:~$ sudo systemctl restart icinga.service * 17:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92903 and previous config saved to /var/cache/conftool/dbconfig/20260525-171310-fceratto.json * 17:11 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1042.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:06 jiji@cumin1003: START - Cookbook sre.dns.netbox * 17:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P92902 and previous config saved to /var/cache/conftool/dbconfig/20260525-170302-fceratto.json * 16:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P92901 and previous config saved to /var/cache/conftool/dbconfig/20260525-165255-fceratto.json * 16:51 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1042.eqiad.wmnet * 16:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92900 and previous config saved to /var/cache/conftool/dbconfig/20260525-164247-fceratto.json * 16:42 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1041.eqiad.wmnet * 16:42 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:42 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1041.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:41 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1041.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:40 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5021.eqsin.wmnet * 16:39 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5029.eqsin.wmnet * 16:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92899 and previous config saved to /var/cache/conftool/dbconfig/20260525-163559-fceratto.json * 16:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2158.codfw.wmnet with reason: Maintenance * 16:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92898 and previous config saved to /var/cache/conftool/dbconfig/20260525-163512-fceratto.json * 16:34 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1108.eqiad.wmnet * 16:30 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1109.eqiad.wmnet * 16:26 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249', diff saved to https://phabricator.wikimedia.org/P92897 and previous config saved to /var/cache/conftool/dbconfig/20260525-162505-fceratto.json * 16:20 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1041.eqiad.wmnet * 16:20 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1040.eqiad.wmnet * 16:20 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:20 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1040.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:16 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1040.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249', diff saved to https://phabricator.wikimedia.org/P92896 and previous config saved to /var/cache/conftool/dbconfig/20260525-161457-fceratto.json * 16:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92895 and previous config saved to /var/cache/conftool/dbconfig/20260525-160450-fceratto.json * 16:02 jiji@cumin1003: START - Cookbook sre.dns.netbox * 15:59 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2249 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92894 and previous config saved to /var/cache/conftool/dbconfig/20260525-155930-fceratto.json * 15:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2249.codfw.wmnet with reason: Maintenance * 15:57 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5020.eqsin.wmnet * 15:57 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5028.eqsin.wmnet * 15:52 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1106.eqiad.wmnet * 15:51 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1107.eqiad.wmnet * 15:29 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1040.eqiad.wmnet * 15:29 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1039.eqiad.wmnet * 15:29 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:29 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1039.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:27 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1039.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:17 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1013 from dbctl [[phab:T427190|T427190]]', diff saved to https://phabricator.wikimedia.org/P92893 and previous config saved to /var/cache/conftool/dbconfig/20260525-151718-marostegui.json * 15:15 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5019.eqsin.wmnet * 15:15 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5027.eqsin.wmnet * 15:12 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1104.eqiad.wmnet * 15:11 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1105.eqiad.wmnet * 15:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92892 and previous config saved to /var/cache/conftool/dbconfig/20260525-150309-fceratto.json * 14:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P92891 and previous config saved to /var/cache/conftool/dbconfig/20260525-145301-fceratto.json * 14:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P92890 and previous config saved to /var/cache/conftool/dbconfig/20260525-144253-fceratto.json * 14:33 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1102.eqiad.wmnet * 14:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92889 and previous config saved to /var/cache/conftool/dbconfig/20260525-143246-fceratto.json * 14:32 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5026.eqsin.wmnet * 14:32 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5018.eqsin.wmnet * 14:31 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1103.eqiad.wmnet * 14:25 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2228 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92888 and previous config saved to /var/cache/conftool/dbconfig/20260525-142551-fceratto.json * 14:25 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance * 14:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92887 and previous config saved to /var/cache/conftool/dbconfig/20260525-142520-fceratto.json * 14:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P92885 and previous config saved to /var/cache/conftool/dbconfig/20260525-141513-fceratto.json * 14:12 jiji@cumin1003: START - Cookbook sre.dns.netbox * 14:06 sukhe: curl localhost:9090/pools/inference-staging-grpc_30051 shows ml-staging200[1-3].codfw.wmnet as enabled and pooled: [[phab:T424049|T424049]] * 14:05 sukhe: sukhe@lvs2013:~$ sudo systemctl restart pybal.service: [[phab:T424049|T424049]] * 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P92884 and previous config saved to /var/cache/conftool/dbconfig/20260525-140505-fceratto.json * 14:03 sukhe: sudo cumin 'A:lvs and A:lvs-low-traffic-codfw' 'run-puppet-agent --enable "adding new ml-serve (grpc) [[phab:T424049|T424049]]"' * 14:02 sukhe: sukhe@lvs2014:~$ sudo systemctl restart pybal.service": [[phab:T424049|T424049]] * 14:02 sukhe: sukhe@lvs2014:~$ sudo systemctl restart pybal.service * 14:00 sukhe: sudo cumin 'A:lvs and A:lvs-secondary-codfw' 'run-puppet-agent --enable "adding new ml-serve (grpc) [[phab:T424049|T424049]]"' * 13:59 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1039.eqiad.wmnet * 13:58 sukhe: sudo cumin 'A:lvs and A:eqiad' 'run-puppet-agent --enable "adding new ml-serve (grpc) [[phab:T424049|T424049]]": NOOP change, since service is codfw only * 13:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92882 and previous config saved to /var/cache/conftool/dbconfig/20260525-135458-fceratto.json * 13:52 Msz2001: Everything deployed, UTC afternoon config+backport window done * 13:52 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293119{{!}}Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] (duration: 09m 43s) * 13:51 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1101.eqiad.wmnet * 13:51 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1100.eqiad.wmnet * 13:50 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5025.eqsin.wmnet * 13:50 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5017.eqsin.wmnet * 13:49 kart_: Updated Recommendation API to 2026-05-21-044522-production * 13:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2223 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92881 and previous config saved to /var/cache/conftool/dbconfig/20260525-134807-fceratto.json * 13:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2223.codfw.wmnet with reason: Maintenance * 13:47 mszwarc@deploy1003: vadymts1, mszwarc: Continuing with deployment * 13:47 kartik@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 13:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92880 and previous config saved to /var/cache/conftool/dbconfig/20260525-134737-fceratto.json * 13:45 mszwarc@deploy1003: vadymts1, mszwarc: Backport for [[gerrit:1293119{{!}}Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:45 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1162: Reboot * 13:43 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293119{{!}}Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] * 13:40 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqiad * 13:39 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqiad * 13:38 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290813{{!}}Article Guidance: enable experiment on phase 2 wikis (T426871)]] (duration: 08m 14s) * 13:38 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqsin * 13:38 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqsin * 13:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P92878 and previous config saved to /var/cache/conftool/dbconfig/20260525-133729-fceratto.json * 13:34 sbisson@deploy1003: sbisson: Continuing with deployment * 13:33 kartik@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 13:32 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1038.eqiad.wmnet * 13:32 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:32 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1038.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 13:31 sbisson@deploy1003: sbisson: Backport for [[gerrit:1290813{{!}}Article Guidance: enable experiment on phase 2 wikis (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:30 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1290813{{!}}Article Guidance: enable experiment on phase 2 wikis (T426871)]] * 13:27 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293094{{!}}Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953{{!}}Update logo, wordmark and tagline for zghwiki (T426406)]] (duration: 07m 43s) * 13:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P92876 and previous config saved to /var/cache/conftool/dbconfig/20260525-132722-fceratto.json * 13:23 mszwarc@deploy1003: mszwarc, jhsoby: Continuing with deployment * 13:21 mszwarc@deploy1003: mszwarc, jhsoby: Backport for [[gerrit:1293094{{!}}Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953{{!}}Update logo, wordmark and tagline for zghwiki (T426406)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:20 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1038.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 13:20 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293094{{!}}Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953{{!}}Update logo, wordmark and tagline for zghwiki (T426406)]] * 13:19 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291966{{!}}Modify various configurations for English Wikibooks (T426992)]] (duration: 15m 53s) * 13:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92875 and previous config saved to /var/cache/conftool/dbconfig/20260525-131714-fceratto.json * 13:12 mszwarc@deploy1003: vadymts1, mszwarc: Continuing with deployment * 13:12 jiji@cumin1003: START - Cookbook sre.dns.netbox * 13:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92873 and previous config saved to /var/cache/conftool/dbconfig/20260525-131023-fceratto.json * 13:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2211.codfw.wmnet with reason: Maintenance * 13:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92872 and previous config saved to /var/cache/conftool/dbconfig/20260525-130950-fceratto.json * 13:07 mszwarc@deploy1003: vadymts1, mszwarc: Backport for [[gerrit:1291966{{!}}Modify various configurations for English Wikibooks (T426992)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:03 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1291966{{!}}Modify various configurations for English Wikibooks (T426992)]] * 12:59 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1162: Reboot * 12:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P92870 and previous config saved to /var/cache/conftool/dbconfig/20260525-125942-fceratto.json * 12:59 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1162: Reboot * 12:59 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1162: Reboot * 12:58 kart_: Updated cxserver to 2026-05-24-103047-production ([[phab:T426808|T426808]], [[phab:T373418|T373418]]) * 12:56 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply * 12:56 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply * 12:54 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db1162: Reboot * 12:54 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1162: Reboot * 12:54 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply * 12:53 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply * 12:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1162.eqiad.wmnet with reason: Reboot * 12:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P92868 and previous config saved to /var/cache/conftool/dbconfig/20260525-124934-fceratto.json * 12:40 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply * 12:39 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply * 12:39 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1038.eqiad.wmnet * 12:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92867 and previous config saved to /var/cache/conftool/dbconfig/20260525-123927-fceratto.json * 12:32 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2192 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92866 and previous config saved to /var/cache/conftool/dbconfig/20260525-123239-fceratto.json * 12:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2192.codfw.wmnet with reason: Maintenance * 12:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92865 and previous config saved to /var/cache/conftool/dbconfig/20260525-123208-fceratto.json * 12:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P92864 and previous config saved to /var/cache/conftool/dbconfig/20260525-122201-fceratto.json * 12:17 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1037.eqiad.wmnet * 12:17 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:17 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1037.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 12:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P92863 and previous config saved to /var/cache/conftool/dbconfig/20260525-121153-fceratto.json * 12:10 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1037.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 12:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92862 and previous config saved to /var/cache/conftool/dbconfig/20260525-120145-fceratto.json * 11:58 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2178 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92861 and previous config saved to /var/cache/conftool/dbconfig/20260525-115504-fceratto.json * 11:54 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2178.codfw.wmnet with reason: Maintenance * 11:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92860 and previous config saved to /var/cache/conftool/dbconfig/20260525-115434-fceratto.json * 11:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P92859 and previous config saved to /var/cache/conftool/dbconfig/20260525-114426-fceratto.json * 11:43 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1037.eqiad.wmnet * 11:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P92858 and previous config saved to /var/cache/conftool/dbconfig/20260525-113419-fceratto.json * 11:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2160.codfw.wmnet with OS trixie * 11:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92857 and previous config saved to /var/cache/conftool/dbconfig/20260525-112411-fceratto.json * 11:17 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2171 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92856 and previous config saved to /var/cache/conftool/dbconfig/20260525-111717-fceratto.json * 11:17 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: Maintenance * 11:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92855 and previous config saved to /var/cache/conftool/dbconfig/20260525-111648-fceratto.json * 11:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P92854 and previous config saved to /var/cache/conftool/dbconfig/20260525-110640-fceratto.json * 11:05 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2160.codfw.wmnet with reason: host reimage * 11:00 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2160.codfw.wmnet with reason: host reimage * 10:58 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:57 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:56 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P92853 and previous config saved to /var/cache/conftool/dbconfig/20260525-105633-fceratto.json * 10:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92852 and previous config saved to /var/cache/conftool/dbconfig/20260525-104625-fceratto.json * 10:43 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2160.codfw.wmnet with OS trixie * 10:41 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc3 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92851 and previous config saved to /var/cache/conftool/dbconfig/20260525-104141-marostegui.json * 10:40 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1023 to pc3 as master [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92850 and previous config saved to /var/cache/conftool/dbconfig/20260525-104055-marostegui.json * 10:40 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1023 to dbctl', diff saved to https://phabricator.wikimedia.org/P92849 and previous config saved to /var/cache/conftool/dbconfig/20260525-104027-marostegui.json * 10:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92848 and previous config saved to /var/cache/conftool/dbconfig/20260525-103944-fceratto.json * 10:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance * 10:31 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply * 10:30 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply * 10:27 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 10:18 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 10:16 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1011.eqiad.wmnet * 10:08 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1011.eqiad.wmnet * 10:08 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1007.eqiad.wmnet * 09:59 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1007.eqiad.wmnet * 09:59 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1006.eqiad.wmnet * 09:57 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:49 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1006.eqiad.wmnet * 09:48 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:46 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:45 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:40 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:40 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:28 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:17 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:13 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92847 and previous config saved to /var/cache/conftool/dbconfig/20260525-091302-fceratto.json * 09:12 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231', diff saved to https://phabricator.wikimedia.org/P92846 and previous config saved to /var/cache/conftool/dbconfig/20260525-090255-fceratto.json * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231', diff saved to https://phabricator.wikimedia.org/P92845 and previous config saved to /var/cache/conftool/dbconfig/20260525-085247-fceratto.json * 08:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92844 and previous config saved to /var/cache/conftool/dbconfig/20260525-084239-fceratto.json * 08:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92843 and previous config saved to /var/cache/conftool/dbconfig/20260525-083540-fceratto.json * 08:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2231.codfw.wmnet with reason: Maintenance * 08:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92842 and previous config saved to /var/cache/conftool/dbconfig/20260525-083511-fceratto.json * 08:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215', diff saved to https://phabricator.wikimedia.org/P92841 and previous config saved to /var/cache/conftool/dbconfig/20260525-082504-fceratto.json * 08:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215', diff saved to https://phabricator.wikimedia.org/P92840 and previous config saved to /var/cache/conftool/dbconfig/20260525-081456-fceratto.json * 08:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92839 and previous config saved to /var/cache/conftool/dbconfig/20260525-080448-fceratto.json * 07:57 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2215 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92838 and previous config saved to /var/cache/conftool/dbconfig/20260525-075739-fceratto.json * 07:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2215.codfw.wmnet with reason: Maintenance * 07:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92837 and previous config saved to /var/cache/conftool/dbconfig/20260525-075708-fceratto.json * 07:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196', diff saved to https://phabricator.wikimedia.org/P92836 and previous config saved to /var/cache/conftool/dbconfig/20260525-074700-fceratto.json * 07:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196', diff saved to https://phabricator.wikimedia.org/P92835 and previous config saved to /var/cache/conftool/dbconfig/20260525-073653-fceratto.json * 07:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92834 and previous config saved to /var/cache/conftool/dbconfig/20260525-072645-fceratto.json * 07:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92833 and previous config saved to /var/cache/conftool/dbconfig/20260525-071953-fceratto.json * 07:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2196.codfw.wmnet with reason: Maintenance * 07:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92832 and previous config saved to /var/cache/conftool/dbconfig/20260525-071924-fceratto.json * 07:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186', diff saved to https://phabricator.wikimedia.org/P92831 and previous config saved to /var/cache/conftool/dbconfig/20260525-070917-fceratto.json * 07:03 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2233.codfw.wmnet with OS trixie * 06:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186', diff saved to https://phabricator.wikimedia.org/P92830 and previous config saved to /var/cache/conftool/dbconfig/20260525-065909-fceratto.json * 06:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92829 and previous config saved to /var/cache/conftool/dbconfig/20260525-064902-fceratto.json * 06:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92828 and previous config saved to /var/cache/conftool/dbconfig/20260525-064305-fceratto.json * 06:42 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance * 06:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2233.codfw.wmnet with reason: host reimage * 06:35 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2233.codfw.wmnet with reason: host reimage * 06:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2233.codfw.wmnet with OS trixie * 06:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2233.codfw.wmnet with reason: Reimage to Trixie * 06:17 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 06:17 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:15 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2160.codfw.wmnet with reason: Reboot upgrade m2 * 06:15 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2233.codfw.wmnet with reason: Reboot upgrade m2 * 06:08 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy1027.eqiad.wmnet with reason: Reboot * 05:18 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2023.codfw.wmnet,pc[1013,1023].eqiad.wmnet with reason: Maintenance on pc3 * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1013.eqiad.wmnet: Maintenance on pc3 * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 05:17 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 05:17 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1013.eqiad.wmnet: Maintenance on pc3 * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 43s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-24 == * 19:08 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down * 02:06 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 23s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-23 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 35s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-22 == * 23:39 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 23:39 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 23:39 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 23:39 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 23:38 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 23:37 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 23:37 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 23:37 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 22:20 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 22:12 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 22:11 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 20:29 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 20:28 inflatador: bking@deploy1003 set eqiad prod cirrus `node_concurrent_recoveries` up to 7 from 4 [[phab:T426585|T426585]] * 20:27 inflatador: bking@deploy1003 set codfw prod cirrus `node_concurrent_recoveries` back down to 4 from 7 [[phab:T426585|T426585]] * 18:39 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 17:34 topranks: enable ttl protection on esams CRs IBGP session * 17:28 topranks: enable ttl protection on ulsfo CRs IBGP session * 16:50 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 16:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 16:16 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 16:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 16:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 15:58 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 15:14 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 15:02 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 15:02 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 14:34 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudnet2008-dev.codfw.wmnet * 14:34 andrew@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:34 andrew@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2008-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 14:33 andrew@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2008-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 14:33 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb[1020,1022-1025].eqiad.wmnet * 14:29 andrew@cumin2002: START - Cookbook sre.dns.netbox * 14:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 14:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 14:23 andrew@cumin2002: START - Cookbook sre.hosts.decommission for hosts cloudnet2008-dev.codfw.wmnet * 14:23 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudnet2007-dev.codfw.wmnet * 14:23 andrew@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:23 andrew@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2007-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 14:03 andrew@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2007-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 13:59 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb[1020,1022-1025].eqiad.wmnet * 13:58 andrew@cumin2002: START - Cookbook sre.dns.netbox * 13:53 andrew@cumin2002: START - Cookbook sre.hosts.decommission for hosts cloudnet2007-dev.codfw.wmnet * 13:52 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1018.eqiad.wmnet * 13:50 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 13:50 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 13:46 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1018.eqiad.wmnet * 13:25 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for clouddb1018.eqiad.wmnet * 13:25 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1018.eqiad.wmnet * 13:25 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for 6 hosts * 13:16 inflatador: bking@deploy1002 set search_codfw cluster recovery settings from 4 to 7 [[phab:T426560|T426560]] * 13:15 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for 6 hosts * 13:15 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 13:11 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp5017.eqsin.wmnet<nowiki>}</nowiki> and A:cp * 13:11 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5017.eqsin.wmnet * 13:10 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1017.eqiad.wmnet * 13:09 elukey: uploaded spicerack_12.6.0 to apt.wikimedia.org bookworm-wikimedia * 13:08 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for clouddb1017.eqiad.wmnet * 12:59 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp5017.eqsin.wmnet<nowiki>}</nowiki> and A:cp * 12:57 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp308[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 12:57 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3081.esams.wmnet * 12:54 isaranto@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 12:41 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 12:15 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3080.esams.wmnet * 12:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 12:03 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp308[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[2-3].esams.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3073.esams.wmnet * 11:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2154: Migration of db2154.codfw.wmnet completed * 11:19 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3072.esams.wmnet * 11:15 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 11:11 fnegri@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb1017.eqiad.wmnet with reason: Rebooting clouddb1017 * 11:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1172: Migration of db1172.eqiad.wmnet completed * 11:07 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[2-3].esams.wmnet<nowiki>}</nowiki> and A:cp * 11:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1058.eqiad.wmnet * 11:01 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 11:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3079.esams.wmnet * 10:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1058.eqiad.wmnet * 10:55 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 10:55 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 10:48 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 10:47 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 10:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1024.eqiad.wmnet * 10:43 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:43 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:43 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:42 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:42 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:42 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2154: Migration of db2154.codfw.wmnet completed * 10:42 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:41 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1024.eqiad.wmnet * 10:37 moritzm: remove ganeti1024 foom eqiad Ganeti cluster [[phab:T424680|T424680]] * 10:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2154.codfw.wmnet with OS trixie * 10:31 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2010.codfw.wmnet with OS trixie * 10:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1024.eqiad.wmnet * 10:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1172: Migration of db1172.eqiad.wmnet completed * 10:19 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3078.esams.wmnet * 10:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2154.codfw.wmnet with reason: host reimage * 10:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1172.eqiad.wmnet with OS trixie * 10:15 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1017.eqiad.wmnet * 10:13 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2154.codfw.wmnet with reason: host reimage * 10:07 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 10:06 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 10:06 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3071.esams.wmnet * 09:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1172.eqiad.wmnet with reason: host reimage * 09:56 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2154.codfw.wmnet with OS trixie * 09:55 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage * 09:53 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1172.eqiad.wmnet with reason: host reimage * 09:51 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage * 09:39 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2154: Upgrading db2154.codfw.wmnet * 09:39 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2154: Upgrading db2154.codfw.wmnet * 09:38 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:38 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1172.eqiad.wmnet with OS trixie * 09:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1172: Upgrading db1172.eqiad.wmnet * 09:34 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1172: Upgrading db1172.eqiad.wmnet * 09:34 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:34 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2009.codfw.wmnet with OS trixie * 09:33 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS trixie * 09:26 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 09:26 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 09:26 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3070.esams.wmnet * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 09:16 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie * 09:14 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 09:11 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[6-7].esams.wmnet<nowiki>}</nowiki> and A:cp * 09:11 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3077.esams.wmnet * 09:04 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 09:03 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie * 08:47 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 08:46 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie * 08:40 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 08:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 08:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 08:30 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3076.esams.wmnet * 08:18 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[6-7].esams.wmnet<nowiki>}</nowiki> and A:cp * 08:15 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti1058.eqiad.wmnet on all recursors * 08:15 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:15 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change records for ganeti1058 - cmooney@cumin1003" * 08:15 cmooney@cumin1003: START - Cookbook sre.dns.wipe-cache ganeti1058.eqiad.wmnet on all recursors * 08:15 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change records for ganeti1058 - cmooney@cumin1003" * 08:09 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 08:07 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp306[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 08:07 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3069.esams.wmnet * 08:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 08:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 07:31 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1024.eqiad.wmnet * 07:26 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3068.esams.wmnet * 07:14 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp306[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 07:11 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1057.eqiad.wmnet to cluster eqiad and group A * 07:10 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3075.esams.wmnet<nowiki>}</nowiki> and A:cp * 07:10 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3075.esams.wmnet * 07:06 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1057.eqiad.wmnet to cluster eqiad and group A * 07:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1057.eqiad.wmnet * 07:02 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1057 * 07:01 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1057 * 06:58 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3075.esams.wmnet<nowiki>}</nowiki> and A:cp * 06:58 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3067.esams.wmnet<nowiki>}</nowiki> and A:cp * 06:58 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3067.esams.wmnet * 06:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1057.eqiad.wmnet * 06:46 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3067.esams.wmnet<nowiki>}</nowiki> and A:cp * 06:13 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1024.eqiad.wmnet * 06:08 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1024.eqiad.wmnet * 06:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org * 06:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org * 05:25 marostegui@dns1004: END - running authdns-update * 05:24 marostegui@dns1004: START - running authdns-update * 05:23 marostegui: Failover m5-master [[phab:T426633|T426633]] * 05:19 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy1028.eqiad.wmnet with reason: Reboot * 05:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy2005.codfw.wmnet with reason: Reboot * 05:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1012.eqiad.wmnet * 05:11 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 05:11 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1012.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 05:06 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1012.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 05:03 marostegui@cumin1003: START - Cookbook sre.dns.netbox * 04:56 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1012.eqiad.wmnet == 2026-05-21 == * 23:43 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290954{{!}}Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957{{!}}Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958{{!}}Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960{{!}}Drop unused wpReportIncidentLocalLinks]] (duration: 06m 42s) * 23:38 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 23:38 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1290954{{!}}Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957{{!}}Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958{{!}}Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960{{!}}Drop unused wpReportIncidentLocalLinks]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified * 23:36 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1290954{{!}}Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957{{!}}Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958{{!}}Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960{{!}}Drop unused wpReportIncidentLocalLinks]] * 22:26 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul2002.codfw.wmnet with OS trixie * 22:08 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul2002.codfw.wmnet with reason: host reimage * 22:03 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul2002.codfw.wmnet with reason: host reimage * 22:02 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 21:49 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 21:49 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 21:44 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host zuul2002.codfw.wmnet with OS trixie * 21:25 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:25 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 20:26 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 20:16 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 19:22 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-reboot (exit_code=0) rolling reboot on A:restbase * 19:10 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:59 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:53 papaul: rebooting msw1-codfw * 18:50 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:39 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:52 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:52 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:50 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:49 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:49 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:48 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:46 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 17:46 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 17:43 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:43 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:43 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:42 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:42 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:41 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:41 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:41 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:41 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:41 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:41 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:41 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:40 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:40 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:40 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:39 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 * 17:39 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:38 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down * 17:37 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 17:36 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:36 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:30 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:25 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:25 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:24 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:23 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:22 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1016.eqiad.wmnet * 17:22 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:14 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2031.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:14 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2030.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:13 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1016.eqiad.wmnet * 17:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:08 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repool pc2 ([[phab:T421705|T421705]])', diff saved to https://phabricator.wikimedia.org/P92810 and previous config saved to /var/cache/conftool/dbconfig/20260521-170823-ladsgroup.json * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:07 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2031.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:07 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2030.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:06 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:03 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:03 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:00 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2029 * 16:58 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2031 * 16:58 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:58 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 * 16:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 * 16:55 papaul: rebooting msw-d3-codfw * 16:55 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 16:52 papaul: rebooting msw-c7-codfw * 16:51 papaul: rebooting msw-c6-codfw * 16:48 papaul: rebooting msw-b7-codfw * 16:48 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1014.eqiad.wmnet * 16:45 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1014.eqiad.wmnet * 16:43 papaul: rebooting msw-b6-codfw * 16:40 papaul: rebooting msw-a1-codfw * 16:37 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 16:37 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1014.eqiad.wmnet * 16:37 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:36 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:35 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 16:35 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2030 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2030 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 * 16:34 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 16:34 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:33 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2028 to codfw - jhancock@cumin2002" * 16:33 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2028 to codfw - jhancock@cumin2002" * 16:26 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 16:24 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on pc1022.eqiad.wmnet with reason: Move to nftables * 16:24 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on pc2022.codfw.wmnet with reason: Move to nftables * 16:18 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2048: Repooling * 16:18 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depool pc2 ([[phab:T421705|T421705]])', diff saved to https://phabricator.wikimedia.org/P92807 and previous config saved to /var/cache/conftool/dbconfig/20260521-161808-ladsgroup.json * 16:15 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:15 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:15 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:15 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:52 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 15:42 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es2048: Repooling * 15:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92804 and previous config saved to /var/cache/conftool/dbconfig/20260521-154108-fceratto.json * 15:39 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:38 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:34 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:34 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:34 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:34 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:34 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92803 and previous config saved to /var/cache/conftool/dbconfig/20260521-153400-fceratto.json * 15:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2048.codfw.wmnet with reason: Maintenance * 15:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92802 and previous config saved to /var/cache/conftool/dbconfig/20260521-153331-fceratto.json * 15:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:25 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:24 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:24 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:24 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:24 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040', diff saved to https://phabricator.wikimedia.org/P92801 and previous config saved to /var/cache/conftool/dbconfig/20260521-152323-fceratto.json * 15:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1045.eqiad.wmnet * 15:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1045.eqiad.wmnet * 15:19 claime: Enabling puppet on A:cp-text - [[phab:T426323|T426323]] * 15:15 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1045.eqiad.wmnet * 15:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040', diff saved to https://phabricator.wikimedia.org/P92800 and previous config saved to /var/cache/conftool/dbconfig/20260521-151316-fceratto.json * 15:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1014.eqiad.wmnet * 15:11 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1045.eqiad.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2034.codfw.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2034.codfw.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1037.eqiad.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1037.eqiad.wmnet * 15:07 elukey@cumin1003: END (PASS) - Cookbook sre.misc-clusters.restart-reboot-config-master (exit_code=0) rolling reboot on A:config-master * 15:06 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1014.eqiad.wmnet * 15:05 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors * 15:05 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors * 15:04 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290805{{!}}hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] (duration: 10m 11s) * 15:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92799 and previous config saved to /var/cache/conftool/dbconfig/20260521-150308-fceratto.json * 15:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1037.eqiad.wmnet * 15:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2034.codfw.wmnet * 15:00 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors * 15:00 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors * 15:00 elukey@cumin1003: START - Cookbook sre.misc-clusters.restart-reboot-config-master rolling reboot on A:config-master * 15:00 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 15:00 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-lab1002.eqiad.wmnet * 14:59 elukey@cumin1003: END (PASS) - Cookbook sre.pki.restart-reboot (exit_code=0) rolling reboot on A:pki * 14:57 claime: Disabling puppet on A:cp-text - [[phab:T426323|T426323]] * 14:56 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1290805{{!}}hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:55 klausman@cumin1003: START - Cookbook sre.hosts.reboot-single for host ml-lab1002.eqiad.wmnet * 14:54 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-build1001.eqiad.wmnet * 14:54 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1290805{{!}}hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] * 14:54 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet * 14:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1013.eqiad.wmnet * 14:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1037.eqiad.wmnet * 14:53 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1028.eqiad.wmnet * 14:53 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>ml-serve1001.eqiad.wmnet<nowiki>}</nowiki> and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) * 14:53 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1001.eqiad.wmnet * 14:53 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1001.eqiad.wmnet * 14:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1028.eqiad.wmnet * 14:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92798 and previous config saved to /var/cache/conftool/dbconfig/20260521-145132-fceratto.json * 14:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2040.codfw.wmnet with reason: Maintenance * 14:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92797 and previous config saved to /var/cache/conftool/dbconfig/20260521-145103-fceratto.json * 14:50 klausman@cumin1003: START - Cookbook sre.hosts.reboot-single for host ml-build1001.eqiad.wmnet * 14:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: Migration of db2241.codfw.wmnet completed * 14:48 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1001.eqiad.wmnet * 14:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1013.eqiad.wmnet * 14:46 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1028.eqiad.wmnet * 14:45 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:44 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:42 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1001.eqiad.wmnet * 14:42 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>ml-serve1001.eqiad.wmnet<nowiki>}</nowiki> and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) * 14:42 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1028.eqiad.wmnet * 14:42 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-worker-eqiad * 14:42 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1011.eqiad.wmnet * 14:42 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1011.eqiad.wmnet * 14:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039', diff saved to https://phabricator.wikimedia.org/P92795 and previous config saved to /var/cache/conftool/dbconfig/20260521-144055-fceratto.json * 14:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1012.eqiad.wmnet * 14:38 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet. on all recursors * 14:37 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet. on all recursors * 14:37 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1011.eqiad.wmnet * 14:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1027.eqiad.wmnet * 14:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1027.eqiad.wmnet * 14:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1011.eqiad.wmnet * 14:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1012.eqiad.wmnet * 14:32 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1010.eqiad.wmnet * 14:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1010.eqiad.wmnet * 14:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039', diff saved to https://phabricator.wikimedia.org/P92793 and previous config saved to /var/cache/conftool/dbconfig/20260521-143045-fceratto.json * 14:30 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet. on all recursors * 14:30 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet. on all recursors * 14:29 elukey@cumin1003: START - Cookbook sre.pki.restart-reboot rolling reboot on A:pki * 14:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1027.eqiad.wmnet * 14:27 slyngshede@cumin1003: END (FAIL) - Cookbook sre.cdn.roll-reboot (exit_code=1) rolling reboot on P<nowiki>{</nowiki>cp601[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 14:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1027.eqiad.wmnet * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1054.eqiad.wmnet * 14:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1054.eqiad.wmnet * 14:24 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1010.eqiad.wmnet * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1011.eqiad.wmnet * 14:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92792 and previous config saved to /var/cache/conftool/dbconfig/20260521-142037-fceratto.json * 14:19 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1054.eqiad.wmnet * 14:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1054.eqiad.wmnet * 14:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1053.eqiad.wmnet * 14:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1053.eqiad.wmnet * 14:14 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1010.eqiad.wmnet * 14:14 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1009.eqiad.wmnet * 14:14 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1009.eqiad.wmnet * 14:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:13 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1011.eqiad.wmnet * 14:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:12 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2218: repool after maintenance * 14:11 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1053.eqiad.wmnet * 14:09 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92789 and previous config saved to /var/cache/conftool/dbconfig/20260521-140906-fceratto.json * 14:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2039.codfw.wmnet with reason: Maintenance * 14:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92788 and previous config saved to /var/cache/conftool/dbconfig/20260521-140837-fceratto.json * 14:08 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1009.eqiad.wmnet * 14:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:07 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1053.eqiad.wmnet * 14:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1035.eqiad.wmnet * 14:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1035.eqiad.wmnet * 14:04 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2241: Migration of db2241.codfw.wmnet completed * 14:03 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1009.eqiad.wmnet * 14:03 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1008.eqiad.wmnet * 14:03 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1008.eqiad.wmnet * 14:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2241.codfw.wmnet with OS trixie * 13:59 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 13:59 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1035.eqiad.wmnet * 13:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92786 and previous config saved to /var/cache/conftool/dbconfig/20260521-135830-fceratto.json * 13:58 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1008.eqiad.wmnet * 13:53 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1008.eqiad.wmnet * 13:53 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1007.eqiad.wmnet * 13:53 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1007.eqiad.wmnet * 13:51 Lucas_WMDE: UTC afternoon backport+config window done * 13:51 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290743{{!}}composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347{{!}}Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342{{!}}fix: simplify to show only one icon type for password reveal (T419413)]] (duration: 07m 20s) * 13:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92784 and previous config saved to /var/cache/conftool/dbconfig/20260521-134822-fceratto.json * 13:48 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1007.eqiad.wmnet * 13:47 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 13:46 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Continuing with deployment * 13:45 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 13:45 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Backport for [[gerrit:1290743{{!}}composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347{{!}}Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342{{!}}fix: simplify to show only one icon type for password reveal (T419413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes * 13:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2241.codfw.wmnet with reason: host reimage * 13:44 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 13:43 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1290743{{!}}composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347{{!}}Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342{{!}}fix: simplify to show only one icon type for password reveal (T419413)]] * 13:43 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 13:43 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1007.eqiad.wmnet * 13:42 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1006.eqiad.wmnet * 13:42 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1006.eqiad.wmnet * 13:41 dbrant@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290035{{!}}docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] (duration: 06m 52s) * 13:41 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 13:40 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2241.codfw.wmnet with reason: host reimage * 13:39 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1035.eqiad.wmnet * 13:38 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in codfw/ml-serve-codfw: maintenance * 13:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92782 and previous config saved to /var/cache/conftool/dbconfig/20260521-133815-fceratto.json * 13:37 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1006.eqiad.wmnet * 13:37 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in codfw/ml-serve-codfw: maintenance * 13:37 dbrant@deploy1003: dbrant: Continuing with deployment * 13:36 dbrant@deploy1003: dbrant: Backport for [[gerrit:1290035{{!}}docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1032.eqiad.wmnet * 13:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1032.eqiad.wmnet * 13:35 dbrant@deploy1003: Started scap sync-world: Backport for [[gerrit:1290035{{!}}docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] * 13:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1006.eqiad.wmnet * 13:32 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1005.eqiad.wmnet * 13:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1005.eqiad.wmnet * 13:31 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290014{{!}}Enable AG on phase 2 wikis (T426871)]] (duration: 09m 11s) * 13:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92781 and previous config saved to /var/cache/conftool/dbconfig/20260521-133116-fceratto.json * 13:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1048.eqiad.wmnet with reason: Maintenance * 13:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92780 and previous config saved to /var/cache/conftool/dbconfig/20260521-133048-fceratto.json * 13:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1032.eqiad.wmnet * 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1032.eqiad.wmnet * 13:27 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1005.eqiad.wmnet * 13:27 sbisson@deploy1003: sbisson: Continuing with deployment * 13:27 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2218: repool after maintenance * 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1031.eqiad.wmnet * 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1031.eqiad.wmnet * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:25 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2241.codfw.wmnet with OS trixie * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:24 sbisson@deploy1003: sbisson: Backport for [[gerrit:1290014{{!}}Enable AG on phase 2 wikis (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:23 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: Upgrading db2241.codfw.wmnet * 13:23 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2241: Upgrading db2241.codfw.wmnet * 13:23 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:22 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1290014{{!}}Enable AG on phase 2 wikis (T426871)]] * 13:22 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1005.eqiad.wmnet * 13:22 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1004.eqiad.wmnet * 13:22 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1004.eqiad.wmnet * 13:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040', diff saved to https://phabricator.wikimedia.org/P92778 and previous config saved to /var/cache/conftool/dbconfig/20260521-132041-fceratto.json * 13:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1031.eqiad.wmnet * 13:20 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290088{{!}}Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032{{!}}Enable 'flood' user group at en.wikiversity (T426882)]] (duration: 11m 55s) * 13:18 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet * 13:17 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1018.eqiad.wmnet with OS trixie * 13:16 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1031.eqiad.wmnet * 13:16 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1039: Repooling * 13:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1030.eqiad.wmnet * 13:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1030.eqiad.wmnet * 13:15 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Continuing with deployment * 13:15 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1004.eqiad.wmnet * 13:14 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet * 13:11 eevans@cumin1003: START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:restbase * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . * 13:10 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1004.eqiad.wmnet * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . * 13:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040', diff saved to https://phabricator.wikimedia.org/P92776 and previous config saved to /var/cache/conftool/dbconfig/20260521-131033-fceratto.json * 13:10 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1003.eqiad.wmnet * 13:10 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1003.eqiad.wmnet * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 13:10 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2241 [[phab:T426936|T426936]]', diff saved to https://phabricator.wikimedia.org/P92775 and previous config saved to /var/cache/conftool/dbconfig/20260521-131025-cwilliams.json * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'readability' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'logo-detection' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . * 13:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1030.eqiad.wmnet * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . * 13:10 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Backport for [[gerrit:1290088{{!}}Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032{{!}}Enable 'flood' user group at en.wikiversity (T426882)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'llm' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . * 13:08 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1290088{{!}}Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032{{!}}Enable 'flood' user group at en.wikiversity (T426882)]] * 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet * 13:06 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp601[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 13:06 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3074.esams.wmnet<nowiki>}</nowiki> and A:cp * 13:06 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3074.esams.wmnet * 13:06 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2162 to x3 primary [[phab:T426936|T426936]]', diff saved to https://phabricator.wikimedia.org/P92774 and previous config saved to /var/cache/conftool/dbconfig/20260521-130609-cwilliams.json * 13:04 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 13:04 cezmunsta: Starting x3 codfw failover from db2241 to db2162 - [[phab:T426936|T426936]] * 13:04 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1003.eqiad.wmnet * 13:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1030.eqiad.wmnet * 13:03 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet * 13:00 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 13:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92772 and previous config saved to /var/cache/conftool/dbconfig/20260521-130018-fceratto.json * 12:59 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1003.eqiad.wmnet * 12:59 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1018.eqiad.wmnet with reason: host reimage * 12:59 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1002.eqiad.wmnet * 12:59 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1002.eqiad.wmnet * 12:58 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:57 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:56 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2162 with weight 0 [[phab:T426936|T426936]]', diff saved to https://phabricator.wikimedia.org/P92771 and previous config saved to /var/cache/conftool/dbconfig/20260521-125645-cwilliams.json * 12:56 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Primary switchover x3 [[phab:T426936|T426936]] * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:55 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1029.eqiad.wmnet * 12:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1029.eqiad.wmnet * 12:54 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3074.esams.wmnet<nowiki>}</nowiki> and A:cp * 12:54 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1002.eqiad.wmnet * 12:54 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp600[7-8].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 12:54 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6008.drmrs.wmnet * 12:53 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:52 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1018.eqiad.wmnet with reason: host reimage * 12:51 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:49 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1002.eqiad.wmnet * 12:49 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-worker-eqiad * 12:48 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1029.eqiad.wmnet * 12:48 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3066.esams.wmnet<nowiki>}</nowiki> and A:cp * 12:48 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3066.esams.wmnet * 12:47 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:47 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92770 and previous config saved to /var/cache/conftool/dbconfig/20260521-124707-fceratto.json * 12:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1040.eqiad.wmnet with reason: Maintenance * 12:46 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es1039: Repooling * 12:46 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:45 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1029.eqiad.wmnet * 12:45 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:44 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:43 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:43 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290727{{!}}hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] (duration: 07m 54s) * 12:42 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92768 and previous config saved to /var/cache/conftool/dbconfig/20260521-124014-fceratto.json * 12:39 kharlan@deploy1003: kharlan: Continuing with deployment * 12:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1052.eqiad.wmnet * 12:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1052.eqiad.wmnet * 12:37 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1018.eqiad.wmnet with OS trixie * 12:37 kharlan@deploy1003: kharlan: Backport for [[gerrit:1290727{{!}}hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:36 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:36 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3066.esams.wmnet<nowiki>}</nowiki> and A:cp * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:35 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1290727{{!}}hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:34 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1017.eqiad.wmnet with OS trixie * 12:34 kart_: Updated cxserver to 2026-05-20-034002-production ([[phab:T388690|T388690]], [[phab:T404295|T404295]], [[phab:T391703|T391703]], [[phab:T426605|T426605]]) * 12:34 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet * 12:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1052.eqiad.wmnet * 12:30 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply * 12:30 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply * 12:30 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet * 12:29 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:29 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92767 and previous config saved to /var/cache/conftool/dbconfig/20260521-122905-fceratto.json * 12:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1039.eqiad.wmnet with reason: Maintenance * 12:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92766 and previous config saved to /var/cache/conftool/dbconfig/20260521-122839-fceratto.json * 12:27 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply * 12:27 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply * 12:26 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:23 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-staging-worker * 12:23 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2003.codfw.wmnet * 12:23 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2003.codfw.wmnet * 12:22 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1052.eqiad.wmnet * 12:21 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply * 12:21 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply * 12:21 moritzm: installing nginx security updates * 12:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1051.eqiad.wmnet * 12:20 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in codfw/ml-serve-codfw: maintenance * 12:19 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1017.eqiad.wmnet with reason: host reimage * 12:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1051.eqiad.wmnet * 12:19 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in codfw/ml-serve-codfw: maintenance * 12:19 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in codfw/ml-staging-codfw: maintenance * 12:19 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in codfw/ml-staging-codfw: maintenance * 12:19 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in codfw/ml-staging-codfw: maintenance * 12:18 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in codfw/ml-staging-codfw: maintenance * 12:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047', diff saved to https://phabricator.wikimedia.org/P92765 and previous config saved to /var/cache/conftool/dbconfig/20260521-121832-fceratto.json * 12:17 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2003.codfw.wmnet * 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet * 12:15 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1017.eqiad.wmnet with reason: host reimage * 12:14 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1051.eqiad.wmnet * 12:13 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6007.drmrs.wmnet * 12:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet * 12:10 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1051.eqiad.wmnet * 12:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047', diff saved to https://phabricator.wikimedia.org/P92764 and previous config saved to /var/cache/conftool/dbconfig/20260521-120824-fceratto.json * 12:07 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2003.codfw.wmnet * 12:07 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2002.codfw.wmnet * 12:07 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2002.codfw.wmnet * 12:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1050.eqiad.wmnet * 12:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1050.eqiad.wmnet * 12:02 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp600[7-8].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp601[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6014.drmrs.wmnet * 12:00 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1017.eqiad.wmnet with OS trixie * 12:00 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2002.codfw.wmnet * 11:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1002.wikimedia.org * 11:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92763 and previous config saved to /var/cache/conftool/dbconfig/20260521-115817-fceratto.json * 11:57 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1050.eqiad.wmnet * 11:53 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt1002.wikimedia.org * 11:51 taavi: disabling puppet on C:bird to roll out {{Gerrit|1289919}} * 11:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92762 and previous config saved to /var/cache/conftool/dbconfig/20260521-115112-fceratto.json * 11:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2047.codfw.wmnet with reason: Maintenance * 11:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1050.eqiad.wmnet * 11:50 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2002.codfw.wmnet * 11:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92761 and previous config saved to /var/cache/conftool/dbconfig/20260521-115043-fceratto.json * 11:50 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2001.codfw.wmnet * 11:50 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2001.codfw.wmnet * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1049.eqiad.wmnet * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt2002.wikimedia.org * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1049.eqiad.wmnet * 11:45 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2001.codfw.wmnet * 11:45 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker-exp1001.eqiad.wmnet * 11:44 kartik@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 11:44 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1049.eqiad.wmnet * 11:43 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt2002.wikimedia.org * 11:42 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet * 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet * 11:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037', diff saved to https://phabricator.wikimedia.org/P92760 and previous config saved to /var/cache/conftool/dbconfig/20260521-114036-fceratto.json * 11:39 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker-exp1001.eqiad.wmnet * 11:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker-exp2001.codfw.wmnet * 11:38 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host testreduce1002.eqiad.wmnet * 11:37 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1049.eqiad.wmnet * 11:36 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet * 11:36 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet * 11:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1038.eqiad.wmnet * 11:35 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2001.codfw.wmnet * 11:35 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-staging-worker * 11:35 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet * 11:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1038.eqiad.wmnet * 11:34 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host testreduce1002.eqiad.wmnet * 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker-exp2001.codfw.wmnet * 11:32 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet * 11:31 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet * 11:30 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt-staging2001.codfw.wmnet * 11:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037', diff saved to https://phabricator.wikimedia.org/P92759 and previous config saved to /var/cache/conftool/dbconfig/20260521-113028-fceratto.json * 11:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2014.codfw.wmnet * 11:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1038.eqiad.wmnet * 11:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt-staging2001.codfw.wmnet * 11:26 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet * 11:24 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1038.eqiad.wmnet * 11:24 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1034.eqiad.wmnet * 11:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1034.eqiad.wmnet * 11:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2014.codfw.wmnet * 11:20 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6013.drmrs.wmnet * 11:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92758 and previous config saved to /var/cache/conftool/dbconfig/20260521-112021-fceratto.json * 11:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1034.eqiad.wmnet * 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling reboot on A:ldap-replicas-eqiad * 11:13 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2013.codfw.wmnet * 11:11 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1034.eqiad.wmnet * 11:09 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp601[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92757 and previous config saved to /var/cache/conftool/dbconfig/20260521-110851-fceratto.json * 11:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2037.codfw.wmnet with reason: Maintenance * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92756 and previous config saved to /var/cache/conftool/dbconfig/20260521-110822-fceratto.json * 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1033.eqiad.wmnet * 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1033.eqiad.wmnet * 11:05 jmm@cumin2002: START - Cookbook sre.ldap.roll-restart-reboot-replica rolling reboot on A:ldap-replicas-eqiad * 11:05 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2013.codfw.wmnet * 11:04 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp600[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 11:04 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6006.drmrs.wmnet * 11:02 jmm@cumin2002: END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling reboot on A:ldap-replicas-codfw * 11:00 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1033.eqiad.wmnet * 10:59 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1016.eqiad.wmnet with reason: host reimage * 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036', diff saved to https://phabricator.wikimedia.org/P92753 and previous config saved to /var/cache/conftool/dbconfig/20260521-105815-fceratto.json * 10:57 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1033.eqiad.wmnet * 10:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1044.eqiad.wmnet * 10:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1044.eqiad.wmnet * 10:55 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1016.eqiad.wmnet with reason: host reimage * 10:54 jmm@cumin2002: START - Cookbook sre.ldap.roll-restart-reboot-replica rolling reboot on A:ldap-replicas-codfw * 10:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2012.codfw.wmnet * 10:51 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:51 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:51 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1044.eqiad.wmnet * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036', diff saved to https://phabricator.wikimedia.org/P92752 and previous config saved to /var/cache/conftool/dbconfig/20260521-104807-fceratto.json * 10:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2012.codfw.wmnet * 10:46 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1044.eqiad.wmnet * 10:44 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290709{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] (duration: 08m 02s) * 10:43 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:41 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:40 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet * 10:40 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:39 jiji@deploy1003: jiji: Continuing with deployment * 10:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92751 and previous config saved to /var/cache/conftool/dbconfig/20260521-103759-fceratto.json * 10:37 jiji@deploy1003: jiji: Backport for [[gerrit:1290709{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:36 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1290709{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] * 10:35 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet * 10:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1043.eqiad.wmnet * 10:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1043.eqiad.wmnet * 10:34 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:29 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 10:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1043.eqiad.wmnet * 10:27 dcausse: [[phab:T423993|T423993]]: reindexing all archive indices * 10:27 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . * 10:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92749 and previous config saved to /var/cache/conftool/dbconfig/20260521-102630-fceratto.json * 10:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance * 10:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1043.eqiad.wmnet * 10:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92748 and previous config saved to /var/cache/conftool/dbconfig/20260521-102601-fceratto.json * 10:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2011.codfw.wmnet * 10:24 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6005.drmrs.wmnet * 10:22 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1042.eqiad.wmnet * 10:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1042.eqiad.wmnet * 10:17 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2011.codfw.wmnet * 10:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1042.eqiad.wmnet * 10:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047', diff saved to https://phabricator.wikimedia.org/P92747 and previous config saved to /var/cache/conftool/dbconfig/20260521-101552-fceratto.json * 10:15 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:14 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . * 10:13 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1042.eqiad.wmnet * 10:13 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1041.eqiad.wmnet * 10:12 moritzm: installing postgresql security updates * 10:12 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp600[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 10:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1041.eqiad.wmnet * 10:10 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet * 10:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org * 10:09 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 10:08 fnegri@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb1013.eqiad.wmnet * 10:08 fnegri@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb1013.eqiad.wmnet * 10:07 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1013.eqiad.wmnet * 10:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1041.eqiad.wmnet * 10:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047', diff saved to https://phabricator.wikimedia.org/P92746 and previous config saved to /var/cache/conftool/dbconfig/20260521-100545-fceratto.json * 10:05 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet * 10:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1041.eqiad.wmnet * 10:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1040.eqiad.wmnet * 10:04 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet * 10:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1040.eqiad.wmnet * 10:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org * 10:01 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1013.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 10:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1040.eqiad.wmnet * 10:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org * 09:59 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet * 09:58 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-codfw * 09:58 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2005.codfw.wmnet * 09:58 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2005.codfw.wmnet * 09:56 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1040.eqiad.wmnet * 09:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1039.eqiad.wmnet * 09:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1039.eqiad.wmnet * 09:56 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 09:56 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:55 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ml-serve1013.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 09:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92745 and previous config saved to /var/cache/conftool/dbconfig/20260521-095536-fceratto.json * 09:54 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1384.eqiad.wmnet * 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org * 09:54 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:54 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:52 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2005.codfw.wmnet * 09:52 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2005.codfw.wmnet * 09:52 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply * 09:52 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2004.codfw.wmnet * 09:52 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2004.codfw.wmnet * 09:51 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply * 09:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1039.eqiad.wmnet * 09:49 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1384.eqiad.wmnet * 09:49 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 09:49 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1383.eqiad.wmnet * 09:48 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1039.eqiad.wmnet * 09:48 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1036.eqiad.wmnet * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92744 and previous config saved to /var/cache/conftool/dbconfig/20260521-094829-fceratto.json * 09:48 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1036.eqiad.wmnet * 09:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1047.eqiad.wmnet with reason: Maintenance * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92743 and previous config saved to /var/cache/conftool/dbconfig/20260521-094801-fceratto.json * 09:47 fnegri@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1013.eqiad.wmnet * 09:47 fnegri@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1013.eqiad.wmnet with reason: Rebooting clouddb1013 [[phab:T426563|T426563]] * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2004.codfw.wmnet * 09:45 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2004.codfw.wmnet * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet * 09:45 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet * 09:45 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet * 09:44 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1383.eqiad.wmnet * 09:44 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 09:44 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1382.eqiad.wmnet * 09:42 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2002.codfw.wmnet * 09:40 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1036.eqiad.wmnet * 09:39 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet * 09:38 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1382.eqiad.wmnet * 09:38 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1381.eqiad.wmnet * 09:38 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1036.eqiad.wmnet * 09:38 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2003.codfw.wmnet * 09:38 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2003.codfw.wmnet * 09:38 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2002.codfw.wmnet * 09:38 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2002.codfw.wmnet * 09:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037', diff saved to https://phabricator.wikimedia.org/P92742 and previous config saved to /var/cache/conftool/dbconfig/20260521-093754-fceratto.json * 09:37 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 09:37 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet * 09:37 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet * 09:37 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet * 09:37 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet * 09:36 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host build2002.codfw.wmnet * 09:36 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 09:35 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp601[1-2].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 09:35 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6012.drmrs.wmnet * 09:34 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet * 09:33 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet * 09:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1381.eqiad.wmnet * 09:33 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1380.eqiad.wmnet * 09:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1023.eqiad.wmnet * 09:31 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet * 09:31 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2002.codfw.wmnet * 09:31 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2002.codfw.wmnet * 09:31 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2001.codfw.wmnet * 09:31 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2001.codfw.wmnet * 09:30 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet * 09:30 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet * 09:30 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet * 09:30 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet * 09:29 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet * 09:29 jayme@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad * 09:29 jayme@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw * 09:29 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet * 09:28 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet * 09:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037', diff saved to https://phabricator.wikimedia.org/P92741 and previous config saved to /var/cache/conftool/dbconfig/20260521-092746-fceratto.json * 09:27 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1380.eqiad.wmnet * 09:27 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1379.eqiad.wmnet * 09:27 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet * 09:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1023.eqiad.wmnet * 09:25 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet * 09:24 jayme@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw * 09:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1056.eqiad.wmnet to cluster eqiad and group A * 09:23 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet * 09:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad * 09:22 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1379.eqiad.wmnet * 09:22 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1378.eqiad.wmnet * 09:21 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2001.codfw.wmnet * 09:21 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2001.codfw.wmnet * 09:21 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-codfw * 09:21 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1056.eqiad.wmnet to cluster eqiad and group A * 09:20 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 09:18 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-jumbo1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:18 moritzm: remove ganeti1023 foom eqiad Ganeti cluster [[phab:T424680|T424680]] * 09:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92740 and previous config saved to /var/cache/conftool/dbconfig/20260521-091738-fceratto.json * 09:16 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1378.eqiad.wmnet * 09:16 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1377.eqiad.wmnet * 09:12 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1377.eqiad.wmnet * 09:12 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1376.eqiad.wmnet * 09:07 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1036: Repooling * 09:07 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1376.eqiad.wmnet * 09:07 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1375.eqiad.wmnet * 09:06 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92738 and previous config saved to /var/cache/conftool/dbconfig/20260521-090609-fceratto.json * 09:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1037.eqiad.wmnet with reason: Maintenance * 09:02 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1375.eqiad.wmnet * 09:01 btullis@cumin1003: START - Cookbook sre.hosts.provision for host kafka-jumbo1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:55 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6011.drmrs.wmnet * 08:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1023.eqiad.wmnet * 08:47 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 08:47 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1256: Migration of db1256.eqiad.wmnet completed * 08:44 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp601[1-2].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 08:42 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp600[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 08:42 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6004.drmrs.wmnet * 08:37 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es1036: Repooling * 08:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92733 and previous config saved to /var/cache/conftool/dbconfig/20260521-082951-fceratto.json * 08:29 hashar@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.3 refs [[phab:T423912|T423912]] * 08:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92731 and previous config saved to /var/cache/conftool/dbconfig/20260521-081642-fceratto.json * 08:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1036.eqiad.wmnet with reason: Maintenance * 08:02 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1256: Migration of db1256.eqiad.wmnet completed * 08:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6003.drmrs.wmnet * 08:00 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1256.eqiad.wmnet with OS trixie * 07:52 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp600[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:51 marostegui@dns1004: END - running authdns-update * 07:50 marostegui@dns1004: START - running authdns-update * 07:48 marostegui: Failover m3-master [[phab:T426633|T426633]] * 07:47 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1023.eqiad.wmnet * 07:46 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp6010.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:46 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6010.drmrs.wmnet * 07:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster1005.eqiad.wmnet to plain * 07:44 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster1005.eqiad.wmnet to plain * 07:43 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1256.eqiad.wmnet with reason: host reimage * 07:42 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster1005.eqiad.wmnet to drbd * 07:38 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1256.eqiad.wmnet with reason: host reimage * 07:35 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp6010.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:35 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp6002.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:35 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6002.drmrs.wmnet * 07:27 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster1005.eqiad.wmnet to drbd * 07:24 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp6002.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:24 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1256.eqiad.wmnet with OS trixie * 07:22 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1256: Upgrading db1256.eqiad.wmnet * 07:21 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1256: Upgrading db1256.eqiad.wmnet * 07:21 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to plain * 07:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to plain * 07:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbproxy1025.eqiad.wmnet with reason: Rebooting * 07:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to drbd * 06:54 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to drbd * 06:53 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to plain * 06:52 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to plain * 06:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to drbd * 06:42 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists1004.wikimedia.org * 06:40 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org * 06:39 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1003.eqiad.wmnet * 06:34 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org * 06:34 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host lists1004.wikimedia.org * 06:33 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host vrts1003.eqiad.wmnet * 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to drbd * 06:23 arnaudb@cumin1003: END (FAIL) - Cookbook sre.gerrit.reboot-gerrit (exit_code=99) Rebooting Gerrit on gerrit2003 * 06:22 arnaudb@cumin1003: START - Cookbook sre.gerrit.reboot-gerrit Rebooting Gerrit on gerrit2003 * 06:15 marostegui@dns1004: END - running authdns-update * 06:14 marostegui: Failover m2-master [[phab:T426633|T426633]] * 06:13 marostegui@dns1004: START - running authdns-update * 05:39 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1012 from dbctl [[phab:T426930|T426930]]', diff saved to https://phabricator.wikimedia.org/P92728 and previous config saved to /var/cache/conftool/dbconfig/20260521-053858-marostegui.json * 05:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc2 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92727 and previous config saved to /var/cache/conftool/dbconfig/20260521-053000-marostegui.json * 05:29 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1022 to pc2 master [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92726 and previous config saved to /var/cache/conftool/dbconfig/20260521-052905-marostegui.json * 05:21 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc1012.eqiad.wmnet with reason: Cloning * 02:41 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on planet1003.eqiad.wmnet with reason: debug wip * 02:11 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 29s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:29 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1027.eqiad.wmnet * 01:22 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1027.eqiad.wmnet * 00:55 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> flidxie56s7pzior4rheqwdodmlrq7c 2426651 2426650 2026-06-14T11:02:55Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2426651 wikitext text/x-wiki == 2026-06-14 == * 11:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 11:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:06 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-06-13 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 35s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-06-12 == * 19:54 dwisehaupt@dns1004: END - running authdns-update * 19:52 dwisehaupt@dns1004: START - running authdns-update * 18:33 dwisehaupt@dns1006: END - running authdns-update * 18:32 dwisehaupt@dns1006: START - running authdns-update * 16:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:10 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:10 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:59 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 15:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:43 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1301371{{!}}Hotfix for T428620 (T428620)]] (duration: 11m 17s) * 14:36 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 14:35 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1301371{{!}}Hotfix for T428620 (T428620)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:31 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1301371{{!}}Hotfix for T428620 (T428620)]] * 14:29 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:28 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 13:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 12:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 12:22 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:22 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:22 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:22 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 12:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 12:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 12:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 12:04 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:04 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:04 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 12:02 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of prometheus5003.eqsin.wmnet to drbd * 12:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus5003.eqsin.wmnet to drbd * 11:40 moritzm: installing Linux 5.10.257 on Bullseye hosts * 11:36 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 11:35 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:24 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 11:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:56 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 10:56 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 10:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:49 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 10:49 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 10:40 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply * 10:37 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply * 10:36 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply * 10:35 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply * 10:35 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply * 10:35 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply * 10:12 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 10:12 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 10:08 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 09:59 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 09:58 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 09:57 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 06:13 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) * 06:11 jmm@cumin2002: START - Cookbook sre.puppet.disable-merges * 03:07 ryankemper: [[phab:T427951|T427951]] sorry, `[eqiad,codfw].mediawiki.page_html_content_change.rc0` (accidentally a word) * 03:06 ryankemper: [[phab:T427951|T427951]] Deleted all 20 unused dev/test topics on kafka-jumbo (verified empty first); 2 (`[eqiad,codfw]page_html_content_change.rc0`) were immediately auto-recreated empty by a still-running `dse-k8s` enrichment consumer; awaiting owner confirmation before final re-delete * 02:01 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 01m 13s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:00 bblack@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () == 2026-06-11 == * 22:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 22:26 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 22:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 22:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 22:05 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300906{{!}}Restore MediaViewer toggle in Special:Preferences (T428742)]] (duration: 30m 51s) * 21:58 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host releases2003.codfw.wmnet with OS trixie * 21:52 egardner@deploy1003: egardner: Continuing with deployment * 21:51 egardner@deploy1003: egardner: Backport for [[gerrit:1300906{{!}}Restore MediaViewer toggle in Special:Preferences (T428742)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:34 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1300906{{!}}Restore MediaViewer toggle in Special:Preferences (T428742)]] * 21:34 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet with reason: host reimage * 21:29 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300913{{!}}Avoid the escaping from nowiki processing (T398967)]] (duration: 09m 09s) * 21:28 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on releases2003.codfw.wmnet with reason: host reimage * 21:25 arlolra@deploy1003: arlolra: Continuing with deployment * 21:22 arlolra@deploy1003: arlolra: Backport for [[gerrit:1300913{{!}}Avoid the escaping from nowiki processing (T398967)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:20 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1300913{{!}}Avoid the escaping from nowiki processing (T398967)]] * 21:07 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300911{{!}}hCaptcha: Enable for badlogin for all small wikis (T426875)]], [[gerrit:1300905{{!}}RadioRangeBallot: Fix strict mode issue (T428947)]] (duration: 10m 43s) * 21:06 bblack@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text and not P<nowiki>{</nowiki>cp7008*<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 21:01 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 21:00 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300911{{!}}hCaptcha: Enable for badlogin for all small wikis (T426875)]], [[gerrit:1300905{{!}}RadioRangeBallot: Fix strict mode issue (T428947)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:56 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300911{{!}}hCaptcha: Enable for badlogin for all small wikis (T426875)]], [[gerrit:1300905{{!}}RadioRangeBallot: Fix strict mode issue (T428947)]] * 20:51 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300842{{!}}Donor Delight Badge: Unify on "Remove badge" language across treatments (T427313)]], [[gerrit:1300843{{!}}[A11y] Donor Badge: Remove Badge button disappears too quickly (T428646)]], [[gerrit:1300896{{!}}Donor Delight Badge, styles: Amending to final design review feedback (T427313)]] (duration: 34m 10s) * 20:39 jdrewniak@deploy1003: annet, jdrewniak: Continuing with deployment * 20:35 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host releases2003.codfw.wmnet with OS trixie * 20:34 jdrewniak@deploy1003: annet, jdrewniak: Backport for [[gerrit:1300842{{!}}Donor Delight Badge: Unify on "Remove badge" language across treatments (T427313)]], [[gerrit:1300843{{!}}[A11y] Donor Badge: Remove Badge button disappears too quickly (T428646)]], [[gerrit:1300896{{!}}Donor Delight Badge, styles: Amending to final design review feedback (T427313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug * 20:17 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1300842{{!}}Donor Delight Badge: Unify on "Remove badge" language across treatments (T427313)]], [[gerrit:1300843{{!}}[A11y] Donor Badge: Remove Badge button disappears too quickly (T428646)]], [[gerrit:1300896{{!}}Donor Delight Badge, styles: Amending to final design review feedback (T427313)]] * 19:12 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 18:12 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 18:12 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 17:52 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300865{{!}}UploadWizard.config.php: Fix cc-by-4.0-heirs msg issue (T428935 T405146)]] (duration: 08m 15s) * 17:48 reedy@deploy1003: reedy: Continuing with deployment * 17:46 reedy@deploy1003: reedy: Backport for [[gerrit:1300865{{!}}UploadWizard.config.php: Fix cc-by-4.0-heirs msg issue (T428935 T405146)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:44 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1300865{{!}}UploadWizard.config.php: Fix cc-by-4.0-heirs msg issue (T428935 T405146)]] * 17:26 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 17:25 blake@deploy1003: Scap cancelled without rolling back. * 17:25 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 17:24 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 17:24 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 17:24 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 17:24 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 17:23 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 17:23 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 17:23 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 17:23 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 17:23 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 17:23 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 17:20 blake@deploy1003: blake: apache config update ([[phab:T428772|T428772]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:20 blake@deploy1003: Started scap sync-world: apache config update ([[phab:T428772|T428772]]) * 17:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 17:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2212: Migration of db2212.codfw.wmnet completed * 17:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 17:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1235: Migration of db1235.eqiad.wmnet completed * 17:08 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 16:45 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:43 dzahn@dns1005: END - running authdns-update * 16:42 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:41 dzahn@dns1005: START - running authdns-update * 16:41 mutante: releases.wikimedia.org - switching backend from codfw to eqiad - releases1003 is now the source of rsync for uploaded releases files (use releases.discovery.wmnet to not have to think about it) - [[phab:T418299|T418299]] * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts rdb2007.codfw.wmnet * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts rdb1011.eqiad.wmnet * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) * 16:34 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb2009.codfw.wmnet * 16:34 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:34 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2009.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:33 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Migration of db2212.codfw.wmnet completed * 16:27 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2009.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:27 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1235: Migration of db1235.eqiad.wmnet completed * 16:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2212.codfw.wmnet with OS trixie * 16:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1235.eqiad.wmnet with OS trixie * 16:13 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:07 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:06 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 16:05 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 16:05 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 16:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 16:04 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2212.codfw.wmnet with reason: host reimage * 16:01 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 16:01 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:01 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 16:01 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 16:00 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 16:00 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 16:00 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 16:00 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2212.codfw.wmnet with reason: host reimage * 15:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1235.eqiad.wmnet with reason: host reimage * 15:58 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 15:58 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 15:57 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 15:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 15:57 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 15:57 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 15:56 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2009.codfw.wmnet * 15:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:55 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb1011.eqiad.wmnet * 15:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 15:55 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2007.codfw.wmnet * 15:54 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 15:54 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1235.eqiad.wmnet with reason: host reimage * 15:54 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 15:53 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 15:53 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 15:40 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 15:40 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2212.codfw.wmnet with OS trixie * 15:39 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 15:39 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1235.eqiad.wmnet with OS trixie * 15:36 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 15:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1235: Upgrading db1235.eqiad.wmnet * 15:35 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 15:35 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1235: Upgrading db1235.eqiad.wmnet * 15:35 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:32 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 15:32 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:31 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 15:30 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300822{{!}}T428849: temporarily disable noisy warnings in HandleParsoidSectionLinks (T428849 T417530)]] (duration: 11m 29s) * 15:27 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2212: Upgrading db2212.codfw.wmnet * 15:26 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2212: Upgrading db2212.codfw.wmnet * 15:26 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:26 cscott@deploy1003: cscott: Continuing with deployment * 15:26 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1235: Upgrading db1235.eqiad.wmnet * 15:25 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1235: Upgrading db1235.eqiad.wmnet * 15:25 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:21 cscott@deploy1003: cscott: Backport for [[gerrit:1300822{{!}}T428849: temporarily disable noisy warnings in HandleParsoidSectionLinks (T428849 T417530)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:19 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1300822{{!}}T428849: temporarily disable noisy warnings in HandleParsoidSectionLinks (T428849 T417530)]] * 15:18 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 15:17 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 15:13 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 15:13 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 15:13 moritzm: installing libdbi-perl security updates * 14:53 moritzm: installing Bind security updates (just client-side tools/libraries) * 14:51 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry * 14:48 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry * 14:43 moritzm: installing Poppler security updates * 14:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:33 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 14:32 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 14:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1234: Migration of db1234.eqiad.wmnet completed * 14:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin02 and group 01 * 14:24 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin02 and group 01 * 14:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:18 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet * 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet * 14:00 Lucas_WMDE: UTC afternoon backport+config window done * 13:58 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300733{{!}}stream: webrequest.page_view_stats.dev0 (T428725)]] (duration: 08m 12s) * 13:57 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp5024.* * 13:55 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: name=cp5024.* * 13:55 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp5020.* * 13:54 javiermonton@deploy1003: javiermonton: Continuing with deployment * 13:52 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1300733{{!}}stream: webrequest.page_view_stats.dev0 (T428725)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:51 slyngshede@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P<nowiki>{</nowiki>lvs5004*<nowiki>}</nowiki> and A:liberica * 13:50 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1300733{{!}}stream: webrequest.page_view_stats.dev0 (T428725)]] * 13:50 slyngshede@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P<nowiki>{</nowiki>lvs5004*<nowiki>}</nowiki> and A:liberica * 13:50 slyngs: reloading liberica config on lvs5004 * 13:50 moritzm: installing openssl security updates * 13:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:46 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 13:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm * 13:46 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1234: Migration of db1234.eqiad.wmnet completed * 13:46 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 13:45 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 13:45 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 13:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2202.codfw.wmnet with OS trixie * 13:43 alexsanford@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298890{{!}}Add 2FA enforcement demotion config for phase 3 groups (T423120)]] (duration: 07m 19s) * 13:39 alexsanford@deploy1003: alexsanford: Continuing with deployment * 13:38 alexsanford@deploy1003: alexsanford: Backport for [[gerrit:1298890{{!}}Add 2FA enforcement demotion config for phase 3 groups (T423120)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:36 alexsanford@deploy1003: Started scap sync-world: Backport for [[gerrit:1298890{{!}}Add 2FA enforcement demotion config for phase 3 groups (T423120)]] * 13:36 slyngshede@dns1004: END - running authdns-update * 13:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1234.eqiad.wmnet with OS trixie * 13:34 moritzm: installing dovecot security updates * 13:34 slyngshede@dns1004: START - running authdns-update * 13:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300787{{!}}hCaptcha: Enable for MobileFrontend on all group1 wikis (T425940)]] (duration: 06m 59s) * 13:29 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 13:29 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 13:29 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 13:29 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 13:28 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 13:28 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 13:28 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 13:27 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300787{{!}}hCaptcha: Enable for MobileFrontend on all group1 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:26 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2202.codfw.wmnet with reason: host reimage * 13:25 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300787{{!}}hCaptcha: Enable for MobileFrontend on all group1 wikis (T425940)]] * 13:25 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/Android_FAQ 'Wikimedia Apps/FAQ/Android' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:24 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/Android_FAQ 'Wikimedia Apps/FAQ/Android' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:22 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300736{{!}}fix: correct intake-url and payload type for NCS experiment events (T422295)]] (duration: 06m 51s) * 13:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage * 13:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1234.eqiad.wmnet with reason: host reimage * 13:18 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Continuing with deployment * 13:18 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2202.codfw.wmnet with reason: host reimage * 13:18 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Backport for [[gerrit:1300736{{!}}fix: correct intake-url and payload type for NCS experiment events (T422295)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:18 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 13:17 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 13:16 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1300736{{!}}fix: correct intake-url and payload type for NCS experiment events (T422295)]] * 13:15 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage * 13:14 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/Android_FAQ 'Wikimedia Apps/FAQ/Android' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:13 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:13 gkyziridis@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300731{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] (duration: 08m 47s) * 13:13 andrewbogott: sudo -i reprepro --noskipold --component thirdparty/openstack-trixie-flamingo-backports update trixie-wikimedia * 13:12 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1234.eqiad.wmnet with reason: host reimage * 13:12 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 13:12 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/iOS_FAQ 'Wikimedia Apps/FAQ/iOS' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:12 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 13:12 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 13:11 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 13:11 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 13:11 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 13:11 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 13:11 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 13:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 13:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 13:09 gkyziridis@deploy1003: gkyziridis: Continuing with deployment * 13:06 gkyziridis@deploy1003: gkyziridis: Backport for [[gerrit:1300731{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:06 claime: echo 'https://api.wikimedia.org/service/lw/specs/openapi.yaml' {{!}} mwscript-k8s --attach -- purgeList.php * 13:04 gkyziridis@deploy1003: Started scap sync-world: Backport for [[gerrit:1300731{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] * 13:02 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2202.codfw.wmnet with OS trixie * 13:00 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:57 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1234.eqiad.wmnet with OS trixie * 12:55 moritzm: installing Exim security updates on Bullseye * 12:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ganeti5006 * 12:47 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti5006 * 12:46 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti5006 * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti5006.eqsin.wmnet 9.0.132.10.in-addr.arpa 9.0.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 12:46 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache ganeti5006.eqsin.wmnet 9.0.132.10.in-addr.arpa 9.0.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5006 - jmm@cumin2002" * 12:46 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5006 - jmm@cumin2002" * 12:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1234: Upgrading db1234.eqiad.wmnet * 12:44 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1234: Upgrading db1234.eqiad.wmnet * 12:44 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2188: Migration of db2188.codfw.wmnet completed * 12:29 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "UX improvements - oblivian@cumin1003" * 12:29 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: UX improvements - oblivian@cumin1003 * 12:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1232: Migration of db1232.eqiad.wmnet completed * 12:28 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: UX improvements - oblivian@cumin1003 * 12:28 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "UX improvements - oblivian@cumin1003" * 12:27 jmm@cumin2002: START - Cookbook sre.dns.netbox * 12:26 jmm@cumin2002: START - Cookbook sre.hosts.move-vlan for host ganeti5006 * 12:26 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm * 12:21 moritzm: remove ganeti5006 from eqsin cluster for reimage [[phab:T428229|T428229]] * 12:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet * 12:10 moritzm: installing openjdk-21 security updates on Bookworm * 12:03 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300764{{!}}Remove GrowthExperiments extension from closed wikis (T428884)]] (duration: 06m 53s) * 11:59 urbanecm@deploy1003: urbanecm: Continuing with deployment * 11:58 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1300764{{!}}Remove GrowthExperiments extension from closed wikis (T428884)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:56 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1300764{{!}}Remove GrowthExperiments extension from closed wikis (T428884)]] * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb1012.eqiad.wmnet * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb2010.codfw.wmnet * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:48 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2010.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 11:46 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:46 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb2008.codfw.wmnet * 11:46 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:46 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2188: Migration of db2188.codfw.wmnet completed * 11:44 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 11:43 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:43 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2010.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 11:43 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1232: Migration of db1232.eqiad.wmnet completed * 11:38 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:37 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 11:37 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 11:36 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 11:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2188.codfw.wmnet with OS trixie * 11:35 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb1012.eqiad.wmnet * 11:34 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2008.codfw.wmnet * 11:34 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2010.codfw.wmnet * 11:33 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 11:32 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 11:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1232.eqiad.wmnet with OS trixie * 11:27 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2002.codfw.wmnet * 11:25 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300749{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300751{{!}}hCaptcha: Enable for DiscussionTools on all wikis (T426039)]] (duration: 08m 38s) * 11:21 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 11:19 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300749{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300751{{!}}hCaptcha: Enable for DiscussionTools on all wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2188.codfw.wmnet with reason: host reimage * 11:17 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300749{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300751{{!}}hCaptcha: Enable for DiscussionTools on all wikis (T426039)]] * 11:15 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2188.codfw.wmnet with reason: host reimage * 11:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1232.eqiad.wmnet with reason: host reimage * 11:13 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc2002.codfw.wmnet * 11:13 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet * 11:11 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 11:09 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2001.codfw.wmnet * 11:09 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1232.eqiad.wmnet with reason: host reimage * 11:08 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet * 11:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:04 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc2001.codfw.wmnet * 11:04 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host testreduce1002.eqiad.wmnet * 11:04 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:02 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1262.eqiad.wmnet with reason: crash * 11:00 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 11:00 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host testreduce1002.eqiad.wmnet * 10:59 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 10:59 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 10:58 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 10:55 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2188.codfw.wmnet with OS trixie * 10:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2188: Upgrading db2188.codfw.wmnet * 10:52 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2188: Upgrading db2188.codfw.wmnet * 10:52 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:52 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1232.eqiad.wmnet with OS trixie * 10:48 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1232: Upgrading db1232.eqiad.wmnet * 10:48 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1232: Upgrading db1232.eqiad.wmnet * 10:48 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:33 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:32 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:31 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300734{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300727{{!}}hCaptcha: Enable for DiscussionTools on group 1 wikis (T426039)]] (duration: 11m 01s) * 10:26 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:23 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:23 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:22 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300734{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300727{{!}}hCaptcha: Enable for DiscussionTools on group 1 wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:20 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300734{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300727{{!}}hCaptcha: Enable for DiscussionTools on group 1 wikis (T426039)]] * 10:18 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:18 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:10 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 10:10 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 10:09 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2045.codfw.wmnet with OS trixie * 10:09 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repool es2046', diff saved to https://phabricator.wikimedia.org/P94069 and previous config saved to /var/cache/conftool/dbconfig/20260611-100221-marostegui.json * 10:01 marostegui@cumin1003: dbctl commit (dc=all): 'Depool es2046', diff saved to https://phabricator.wikimedia.org/P94068 and previous config saved to /var/cache/conftool/dbconfig/20260611-100145-marostegui.json * 10:01 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:59 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300580{{!}}ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] (duration: 15m 41s) * 09:54 jiji@deploy1003: jiji: Continuing with deployment * 09:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2045.codfw.wmnet with reason: host reimage * 09:45 jiji@deploy1003: jiji: Backport for [[gerrit:1300580{{!}}ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:43 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1300580{{!}}ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] * 09:42 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2045.codfw.wmnet with reason: host reimage * 09:37 elukey: uploaded spicerack_12.8.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia * 09:26 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS trixie * 09:26 marostegui@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host es2045.codfw.wmnet with OS bookworm * 09:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2176: Migration of db2176.codfw.wmnet completed * 09:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1219: Migration of db1219.eqiad.wmnet completed * 09:11 claime: cumin -x 'A:swift-fe' "disable-puppet 'Disabling puppet for ratelimit deploy - cgoubert'" * 08:57 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS bookworm * 08:39 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2176: Migration of db2176.codfw.wmnet completed * 08:34 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94055) * 08:34 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1219: Migration of db1219.eqiad.wmnet completed * 08:33 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94053) * 08:30 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T428823|T428823]] (duration: 01m 18s) * 08:29 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T428823|T428823]] * 08:27 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2176.codfw.wmnet with OS trixie * 08:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Migration to 10.11.17 * 08:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 08:25 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 08:25 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool pc1021: Migration to 10.11.17 * 08:25 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94052) * 08:24 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): Testing upgrade for [[phab:T428823|T428823]] (duration: 01m 17s) * 08:23 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): Testing upgrade for [[phab:T428823|T428823]] * 08:22 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94051) * 08:22 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1219.eqiad.wmnet with OS trixie * 08:17 moritzm: installing PHP 8.2 security updates * 08:15 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 08:14 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 08:11 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 08:11 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 08:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2176.codfw.wmnet with reason: host reimage * 08:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1013.eqiad.wmnet with OS trixie * 08:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5004.eqsin.wmnet to cluster eqsin02 and group 01 * 08:06 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 08:06 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 08:05 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2021.codfw.wmnet,pc1021.eqiad.wmnet with reason: upgrade * 08:05 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1219.eqiad.wmnet with reason: host reimage * 08:05 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5004.eqsin.wmnet to cluster eqsin02 and group 01 * 08:05 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 08:05 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 08:04 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2176.codfw.wmnet with reason: host reimage * 08:04 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 08:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 08:03 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 08:03 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5004.eqsin.wmnet * 07:58 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1219.eqiad.wmnet with reason: host reimage * 07:56 marostegui: install mariadb 10.11.17 on pc1 [[phab:T427345|T427345]] * 07:54 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1013.eqiad.wmnet with reason: host reimage * 07:50 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1013.eqiad.wmnet with reason: host reimage * 07:49 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 07:49 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 07:49 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5004.eqsin.wmnet * 07:47 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 07:47 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 07:46 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2176.codfw.wmnet with OS trixie * 07:43 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1219.eqiad.wmnet with OS trixie * 07:43 moritzm: imported Jenkins 2.541.3 for thirdparty/ci (Bullseye) and thirdparty/jenkins (Bookworm, Trixie) * 07:42 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 07:35 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1013.eqiad.wmnet with OS trixie * 07:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2176: Upgrading db2176.codfw.wmnet * 07:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1219: Upgrading db1219.eqiad.wmnet * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2176: Upgrading db2176.codfw.wmnet * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1219: Upgrading db1219.eqiad.wmnet * 07:31 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:30 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 07:29 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1163: Repooling * 07:19 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 06:51 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS trixie * 06:50 marostegui@cumin1003: dbctl commit (dc=all): 'Repool es2042', diff saved to https://phabricator.wikimedia.org/P94044 and previous config saved to /var/cache/conftool/dbconfig/20260611-065049-marostegui.json * 06:50 marostegui@cumin1003: dbctl commit (dc=all): 'Depool es2042', diff saved to https://phabricator.wikimedia.org/P94043 and previous config saved to /var/cache/conftool/dbconfig/20260611-065027-marostegui.json * 06:44 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1163: Repooling * 06:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1163 [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94041 and previous config saved to /var/cache/conftool/dbconfig/20260611-064319-fceratto.json * 06:42 fceratto@dns1005: END - running authdns-update * 06:40 fceratto@dns1005: START - running authdns-update * 06:33 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:33 fceratto@cumin1003: MariaDB change: Setting sections s1 as read-write for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:33 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:33 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1184 to s1 primary and set section read-write [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94040 and previous config saved to /var/cache/conftool/dbconfig/20260611-063323-fceratto.json * 06:32 fceratto@cumin1003: dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94039 and previous config saved to /var/cache/conftool/dbconfig/20260611-063251-fceratto.json * 06:32 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:32 fceratto@cumin1003: Dbctl change: Setting sections s1 as read-write for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:32 fceratto@cumin1003: MariaDB change: Setting sections s1 as read-write for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:31 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:31 fceratto@cumin1003: dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94037 and previous config saved to /var/cache/conftool/dbconfig/20260611-063100-fceratto.json * 06:30 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:30 fceratto@cumin1003: MariaDB change: Setting sections s1 as read-only for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:30 fceratto@cumin1003: Dbctl change: Setting sections s1 as read-only for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:29 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:29 federico3: Starting s1 eqiad failover from db1163 to db1184 - [[phab:T426083|T426083]] * 06:22 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1184 with weight 0 [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94035 and previous config saved to /var/cache/conftool/dbconfig/20260611-062224-fceratto.json * 06:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 30 hosts with reason: Primary switchover s1 [[phab:T426083|T426083]] * 05:37 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 05:28 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 05:27 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 05:18 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 05:17 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS trixie * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2045: Upgrading es2045.codfw.wmnet * 05:16 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2045: Upgrading es2045.codfw.wmnet * 05:16 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 44s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:23 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2046.* * 01:19 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync * 01:18 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync * 01:18 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1009.eqiad.wmnet with OS trixie * 01:12 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:12 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 01:12 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:12 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:11 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:11 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 01:11 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:10 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:10 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 01:09 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 01:09 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 01:08 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 01:08 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 01:08 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 01:07 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 01:07 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 01:06 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 01:06 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 01:06 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 01:05 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 01:05 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 01:05 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 01:02 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1009.eqiad.wmnet with reason: host reimage * 00:58 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1009.eqiad.wmnet with reason: host reimage * 00:54 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main1009 * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main1009 * 00:41 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main1009 * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main1009.eqiad.wmnet 37.48.64.10.in-addr.arpa 7.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:41 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main1009.eqiad.wmnet 37.48.64.10.in-addr.arpa 7.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1009 - jasmine@cumin2002" * 00:40 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1009 - jasmine@cumin2002" * 00:39 cdanis@cumin1003: dbctl commit (dc=all): 'depool db1262', diff saved to https://phabricator.wikimedia.org/P94032 and previous config saved to /var/cache/conftool/dbconfig/20260611-003950-cdanis.json * 00:36 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 00:34 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5020.* * 00:30 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main1009 * 00:30 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1009.eqiad.wmnet with OS trixie * 00:03 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5024.* == 2026-06-10 == * 23:53 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5024.* * 23:15 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300154{{!}}Disable ShortUrl on bdwikimedia, bhwiki, bnwiki, bnwikisource, eswikibooks, gomwiki (T107188)]] (duration: 11m 37s) * 23:11 krinkle@deploy1003: krinkle: Continuing with deployment * 23:06 krinkle@deploy1003: krinkle: Backport for [[gerrit:1300154{{!}}Disable ShortUrl on bdwikimedia, bhwiki, bnwiki, bnwikisource, eswikibooks, gomwiki (T107188)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:04 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1300154{{!}}Disable ShortUrl on bdwikimedia, bhwiki, bnwiki, bnwikisource, eswikibooks, gomwiki (T107188)]] * 22:57 ladsgroup@dns1004: END - running authdns-update * 22:55 ladsgroup@dns1004: START - running authdns-update * 22:13 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5024.eqsin.wmnet with OS trixie * 22:13 mutante: gerrit - restarting service for logging change * 22:11 dzahn@cumin2002: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 0:10:00 on gerrit.wikimedia.org with reason: service restart * 22:09 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on gerrit2003.wikimedia.org with reason: service restart * 22:06 mutante: gerrit-spare: restarting gerrit * 22:06 mutante: gerrit-replica: restarting gerrit * 21:44 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage * 21:37 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage * 21:22 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300250{{!}}ExecuteTestAndCacheJob: Fix stdClasses serialised wrongly by JobQueue (T428801)]], [[gerrit:1300248{{!}}tests: Fix StandaloneHooksTest ordering, now broken by DB upgrade]] (duration: 08m 23s) * 21:17 jforrester@deploy1003: jforrester: Continuing with deployment * 21:15 jforrester@deploy1003: jforrester: Backport for [[gerrit:1300250{{!}}ExecuteTestAndCacheJob: Fix stdClasses serialised wrongly by JobQueue (T428801)]], [[gerrit:1300248{{!}}tests: Fix StandaloneHooksTest ordering, now broken by DB upgrade]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:13 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1300250{{!}}ExecuteTestAndCacheJob: Fix stdClasses serialised wrongly by JobQueue (T428801)]], [[gerrit:1300248{{!}}tests: Fix StandaloneHooksTest ordering, now broken by DB upgrade]] * 21:03 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5024 * 21:02 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5024 * 21:02 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300247{{!}}Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] (duration: 06m 51s) * 21:00 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp5024 * 21:00 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5024.eqsin.wmnet 35.0.132.10.in-addr.arpa 5.3.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 21:00 brett@cumin2002: START - Cookbook sre.dns.wipe-cache cp5024.eqsin.wmnet 35.0.132.10.in-addr.arpa 5.3.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 21:00 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:00 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5024 - brett@cumin2002" * 20:59 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5024 - brett@cumin2002" * 20:57 catrope@deploy1003: catrope: Continuing with deployment * 20:57 catrope@deploy1003: catrope: Backport for [[gerrit:1300247{{!}}Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:55 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1300247{{!}}Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] * 20:54 brett@cumin2002: START - Cookbook sre.dns.netbox * 20:50 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5024 * 20:49 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5024.eqsin.wmnet with OS trixie * 20:48 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5020.* * 20:44 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300073{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] (duration: 11m 55s) * 20:40 catrope@deploy1003: catrope, gkyziridis: Continuing with deployment * 20:34 catrope@deploy1003: catrope, gkyziridis: Backport for [[gerrit:1300073{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:32 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1300073{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] * 20:30 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5020.eqsin.wmnet with OS trixie * 20:30 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300226{{!}}[arzwiki] Change the wordmark (T427720)]] (duration: 09m 49s) * 20:25 catrope@deploy1003: gergesshamon, catrope: Continuing with deployment * 20:22 catrope@deploy1003: gergesshamon, catrope: Backport for [[gerrit:1300226{{!}}[arzwiki] Change the wordmark (T427720)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1300226{{!}}[arzwiki] Change the wordmark (T427720)]] * 19:59 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5020.eqsin.wmnet with reason: host reimage * 19:53 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5020.eqsin.wmnet with reason: host reimage * 19:30 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 19:27 bblack@cumin1003: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 19:23 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2046.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:19 brett@cumin2002: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2046.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5020 * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5020 * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2044.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:18 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp5020 * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5020.eqsin.wmnet 24.0.132.10.in-addr.arpa 4.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:18 brett@cumin2002: START - Cookbook sre.dns.wipe-cache cp5020.eqsin.wmnet 24.0.132.10.in-addr.arpa 4.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:17 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5020 - brett@cumin2002" * 19:17 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5020 - brett@cumin2002" * 19:14 brett@cumin2002: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2044.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:11 brett@cumin2002: START - Cookbook sre.dns.netbox * 19:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 19:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2174: Migration of db2174.codfw.wmnet completed * 19:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 19:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1218: Migration of db1218.eqiad.wmnet completed * 18:24 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5020 * 18:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5020.eqsin.wmnet with OS trixie * 18:22 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2174: Migration of db2174.codfw.wmnet completed * 18:20 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 18:17 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1218: Migration of db1218.eqiad.wmnet completed * 18:16 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5018.* * 18:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2174.codfw.wmnet with OS trixie * 18:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1218.eqiad.wmnet with OS trixie * 17:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2174.codfw.wmnet with reason: host reimage * 17:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1218.eqiad.wmnet with reason: host reimage * 17:46 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2010.codfw.wmnet with OS trixie * 17:45 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync * 17:44 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync * 17:44 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2174.codfw.wmnet with reason: host reimage * 17:42 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1218.eqiad.wmnet with reason: host reimage * 17:33 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94021) * 17:29 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2010.codfw.wmnet with reason: host reimage * 17:26 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1218.eqiad.wmnet with OS trixie * 17:26 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2174.codfw.wmnet with OS trixie * 17:25 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:24 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 17:24 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1218: Upgrading db1218.eqiad.wmnet * 17:24 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:24 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:24 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1218: Upgrading db1218.eqiad.wmnet * 17:23 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 17:23 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2174: Upgrading db2174.codfw.wmnet * 17:23 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 17:23 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2010.codfw.wmnet with reason: host reimage * 17:23 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:22 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2174: Upgrading db2174.codfw.wmnet * 17:22 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:22 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 17:22 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 17:22 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 17:22 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text and not P<nowiki>{</nowiki>cp7008*<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 17:21 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 17:21 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 17:19 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 17:19 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 17:18 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 17:18 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 17:17 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 17:17 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 17:17 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 17:13 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 17:12 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox) * 17:03 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 17:03 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1206: Migration of db1206.eqiad.wmnet completed * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2010 * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2010 * 17:02 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2010 * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2010.codfw.wmnet 35.48.192.10.in-addr.arpa 5.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:02 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2010.codfw.wmnet 35.48.192.10.in-addr.arpa 5.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2010 - jasmine@cumin2002" * 17:01 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2010 - jasmine@cumin2002" * 16:57 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 16:50 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2010 * 16:50 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2010.codfw.wmnet with OS trixie * 16:41 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 16:39 bblack@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 16:39 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 16:34 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 16:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5018.eqsin.wmnet with OS trixie * 16:22 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 16:20 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 16:17 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1206: Migration of db1206.eqiad.wmnet completed * 16:15 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 16:15 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 16:14 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 16:12 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 16:12 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 16:11 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 16:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1206.eqiad.wmnet with OS trixie * 16:01 blblack: apt: uploaded libvmod-wmfuniq 0.3.0 for trixie * 15:59 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5018.eqsin.wmnet with reason: host reimage * 15:53 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:52 vriley@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:51 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5018.eqsin.wmnet with reason: host reimage * 15:50 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1206.eqiad.wmnet with reason: host reimage * 15:45 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1206.eqiad.wmnet with reason: host reimage * 15:43 sukhe@cumin1003: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool drmrs [reason: no reason specified, no task ID specified] * 15:42 sukhe@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: no reason specified, no task ID specified] * 15:38 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 15:38 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2173: Migration of db2173.codfw.wmnet completed * 15:34 topranks: drain traffic through cr2-drmrs to reset pic0 * 15:33 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94013) * 15:30 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1206.eqiad.wmnet with OS trixie * 15:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1206: Upgrading db1206.eqiad.wmnet * 15:28 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1206: Upgrading db1206.eqiad.wmnet * 15:27 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:25 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:24 vriley@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:24 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1009 * 15:24 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Harroyo-wmf out of all services on: 2436 hosts * 15:23 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1009 * 15:21 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:20 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist translate extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release * 15:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5018 * 15:19 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5018 * 15:18 vriley@cumin1003: START - Cookbook sre.dns.netbox * 15:18 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp5018 * 15:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5018.eqsin.wmnet 18.0.132.10.in-addr.arpa 8.1.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 15:18 brett@cumin2002: START - Cookbook sre.dns.wipe-cache cp5018.eqsin.wmnet 18.0.132.10.in-addr.arpa 8.1.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 15:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:15 brett@cumin2002: START - Cookbook sre.dns.netbox * 15:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 15:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1195: Migration of db1195.eqiad.wmnet completed * 15:12 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:11 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:11 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin1003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:11 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin1003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:08 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300169{{!}}Fix snak value display for rtl languages (T360854)]], [[gerrit:1300168{{!}}Fix snak value display for rtl languages (T360854)]] (duration: 08m 39s) * 15:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 15:01 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1300169{{!}}Fix snak value display for rtl languages (T360854)]], [[gerrit:1300168{{!}}Fix snak value display for rtl languages (T360854)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:59 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:59 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1300169{{!}}Fix snak value display for rtl languages (T360854)]], [[gerrit:1300168{{!}}Fix snak value display for rtl languages (T360854)]] * 14:58 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:55 Lucas_WMDE: lucaswerkmeister-wmde@deploy1003 $ printf 'https://www.mediawiki.org/keys/%s\n' '' 'keys.txt' 'keys.html' {{!}} mwscript-k8s --attach --comment=[[phab:T423267|T423267]] purgeList mediawikiwiki * 14:54 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist translate extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release, now with correct schema * 14:53 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2173: Migration of db2173.codfw.wmnet completed * 14:50 ayounsi@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin2003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:50 ayounsi@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:49 ayounsi@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:48 ayounsi@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:47 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299614{{!}}Add my public key to mediawiki.org/keys (T423267)]] (duration: 08m 33s) * 14:46 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:42 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, matmarex: Continuing with deployment * 14:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2173.codfw.wmnet with OS trixie * 14:40 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, matmarex: Backport for [[gerrit:1299614{{!}}Add my public key to mediawiki.org/keys (T423267)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:40 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:40 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:38 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1299614{{!}}Add my public key to mediawiki.org/keys (T423267)]] * 14:38 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox and (A:dnsbox) * 14:34 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:34 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:33 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:29 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1195: Migration of db1195.eqiad.wmnet completed * 14:28 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:27 vriley@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:26 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 14:26 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 14:24 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist translate extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release, now with dblist translate * 14:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2173.codfw.wmnet with reason: host reimage * 14:23 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 14:22 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 14:22 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply * 14:21 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply * 14:20 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox) * 14:20 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2173.codfw.wmnet with reason: host reimage * 14:20 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:19 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:19 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:18 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:18 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:18 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:18 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1195.eqiad.wmnet with OS trixie * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 14:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 14:15 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 14:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 14:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 14:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 14:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:13 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:13 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 14:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 14:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 14:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 14:09 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 14:08 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:08 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 14:07 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 14:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 14:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 14:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 14:02 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2173.codfw.wmnet with OS trixie * 14:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 14:00 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1195.eqiad.wmnet with reason: host reimage * 14:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 13:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2173: Upgrading db2173.codfw.wmnet * 13:59 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2173: Upgrading db2173.codfw.wmnet * 13:58 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:58 atsuko@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/ttmserver-export.php --wiki=default --ttmserver eqiad-test # [[phab:T425377|T425377]] populating production index on test cluster to estimate time required for the release * 13:56 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1195.eqiad.wmnet with reason: host reimage * 13:54 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Atieno out of all services on: 2436 hosts * 13:42 Lucas_WMDE: UTC afternoon backport+config window done * 13:42 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1195.eqiad.wmnet with OS trixie * 13:36 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297237{{!}}wmf-config: Update private subnets to include additions (T427393)]] (duration: 07m 20s) * 13:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1195: Upgrading db1195.eqiad.wmnet * 13:33 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling restart_daemons on A:hcaptcha-proxy and A:hcaptcha-proxy * 13:33 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum and A:durum * 13:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2170: Migration of db2170.codfw.wmnet completed * 13:33 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1195: Upgrading db1195.eqiad.wmnet * 13:32 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:32 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, brett: Continuing with deployment * 13:32 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough * 13:31 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply * 13:31 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, brett: Backport for [[gerrit:1297237{{!}}wmf-config: Update private subnets to include additions (T427393)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:31 eevans@deploy1003: helmfile [staging] START helmfile.d/services/data-gateway: apply * 13:29 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1297237{{!}}wmf-config: Update private subnets to include additions (T427393)]] * 13:28 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp5018.eqsin.wmnet with reason: host down * 13:28 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy (exit_code=0) rolling restart_daemons on A:tcpproxy and A:tcpproxy * 13:25 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5018.eqsin.wmnet,service=(cdn{{!}}ats-be) * 13:22 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox) * 13:20 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum and A:durum * 13:20 sukhe@cumin1003: START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling restart_daemons on A:hcaptcha-proxy and A:hcaptcha-proxy * 13:19 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299676{{!}}Enable ULS v2 on group0 wikis]] (duration: 17m 00s) * 13:19 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough * 13:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1186: Migration of db1186.eqiad.wmnet completed * 13:18 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-test: apply * 13:18 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-test: apply * 13:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply * 13:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply * 13:15 sbisson@deploy1003: sbisson, abi: Continuing with deployment * 13:10 sukhe@cumin1003: START - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy rolling restart_daemons on A:tcpproxy and A:tcpproxy * 13:05 sbisson@deploy1003: sbisson, abi: Backport for [[gerrit:1299676{{!}}Enable ULS v2 on group0 wikis]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:03 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1014.eqiad.wmnet with OS trixie * 13:02 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299676{{!}}Enable ULS v2 on group0 wikis]] * 12:47 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2170: Migration of db2170.codfw.wmnet completed * 12:46 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5004.eqsin.wmnet with OS bookworm * 12:46 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:46 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:46 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:45 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1014.eqiad.wmnet with reason: host reimage * 12:42 topranks: re-map DSCP AF41 from 'low' to 'normal' priority qos class on network [[phab:T424640|T424640]] * 12:41 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1014.eqiad.wmnet with reason: host reimage * 12:36 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2170.codfw.wmnet with OS trixie * 12:33 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1186: Migration of db1186.eqiad.wmnet completed * 12:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5004.eqsin.wmnet with reason: host reimage * 12:24 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1014 * 12:24 jiji@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host rdb1014 * 12:23 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1186.eqiad.wmnet with OS trixie * 12:21 jiji@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host rdb1014 * 12:21 jiji@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb1014.eqiad.wmnet 42.48.64.10.in-addr.arpa 2.4.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 12:21 jiji@cumin1003: START - Cookbook sre.dns.wipe-cache rdb1014.eqiad.wmnet 42.48.64.10.in-addr.arpa 2.4.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 12:21 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:21 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host rdb1014 - jiji@cumin1003" * 12:21 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host rdb1014 - jiji@cumin1003" * 12:20 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5004.eqsin.wmnet with reason: host reimage * 12:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2170.codfw.wmnet with reason: host reimage * 12:16 jiji@cumin1003: START - Cookbook sre.dns.netbox * 12:13 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1014 * 12:12 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1014.eqiad.wmnet with OS trixie * 12:12 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2170.codfw.wmnet with reason: host reimage * 12:08 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300104{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643{{!}}wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] (duration: 11m 06s) * 12:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1186.eqiad.wmnet with reason: host reimage * 12:03 reedy@deploy1003: reedy: Continuing with deployment * 12:02 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1186.eqiad.wmnet with reason: host reimage * 11:59 reedy@deploy1003: reedy: Backport for [[gerrit:1300104{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643{{!}}wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes c * 11:57 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1300104{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643{{!}}wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] * 11:53 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2170.codfw.wmnet with OS trixie * 11:51 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ganeti5004 * 11:51 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti5004 * 11:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2170: Upgrading db2170.codfw.wmnet * 11:49 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2170: Upgrading db2170.codfw.wmnet * 11:49 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti5004 * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti5004.eqsin.wmnet 40.0.132.10.in-addr.arpa 0.4.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 11:49 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache ganeti5004.eqsin.wmnet 40.0.132.10.in-addr.arpa 0.4.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5004 - jmm@cumin2002" * 11:49 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5004 - jmm@cumin2002" * 11:49 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:48 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1186.eqiad.wmnet with OS trixie * 11:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1186: Upgrading db1186.eqiad.wmnet * 11:45 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1186: Upgrading db1186.eqiad.wmnet * 11:45 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:38 jmm@cumin2002: START - Cookbook sre.dns.netbox * 11:35 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 11:34 jmm@cumin2002: START - Cookbook sre.hosts.move-vlan for host ganeti5004 * 11:34 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 11:34 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5004.eqsin.wmnet with OS bookworm * 11:34 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 11:33 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1151: Security updates * 11:33 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 11:33 root@cumin1003: START - Cookbook sre.mysql.parsercache * 11:33 root@cumin1003: START - Cookbook sre.mysql.pool pool db1151: Security updates * 11:31 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply * 11:30 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply * 11:30 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply * 11:30 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:23 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 11:23 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:23 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 11:23 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 11:16 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 11:15 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 11:15 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 11:15 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:09 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1151: Security updates * 11:09 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 11:09 root@cumin1003: START - Cookbook sre.mysql.parsercache * 11:09 root@cumin1003: START - Cookbook sre.mysql.depool depool db1151: Security updates * 11:08 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300092{{!}}ProductionServices: re-add poolcounter2006 (T426736)]] (duration: 06m 55s) * 11:04 blake@deploy1003: blake: Continuing with deployment * 11:04 blake@deploy1003: blake: Backport for [[gerrit:1300092{{!}}ProductionServices: re-add poolcounter2006 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:03 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:01 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300092{{!}}ProductionServices: re-add poolcounter2006 (T426736)]] * 10:59 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet * 10:57 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 10:57 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 10:57 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 10:56 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 10:56 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply * 10:56 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply * 10:56 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet * 10:56 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300087{{!}}ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] (duration: 06m 42s) * 10:51 blake@deploy1003: blake: Continuing with deployment * 10:51 moritzm: remove ganeti5004 from eqsin cluster for reimage [[phab:T428229|T428229]] * 10:51 blake@deploy1003: blake: Backport for [[gerrit:1300087{{!}}ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:49 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300087{{!}}ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] * 10:47 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet * 10:47 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 10:46 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 10:46 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 10:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 10:43 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet * 10:43 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300082{{!}}ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] (duration: 07m 38s) * 10:41 moritzm: installing nginx security updates * 10:38 blake@deploy1003: blake: Continuing with deployment * 10:38 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1152: Security updates * 10:38 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:38 root@cumin1003: START - Cookbook sre.mysql.parsercache * 10:38 root@cumin1003: START - Cookbook sre.mysql.pool pool db1152: Security updates * 10:38 blake@deploy1003: blake: Backport for [[gerrit:1300082{{!}}ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:37 moritzm: failover Ganeti master in eqsin to ganeti5007 [[phab:T428229|T428229]] * 10:35 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300082{{!}}ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] * 10:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 10:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 10:33 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet * 10:29 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet * 10:29 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300072{{!}}ProductionServices: reboot poolcounter1007 (T426736)]] (duration: 07m 45s) * 10:27 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 10:27 jmm@cumin2002: DONE (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for sretest2009.codfw.wmnet: Renew puppet certificate - jmm@cumin2002 * 10:24 blake@deploy1003: blake: Continuing with deployment * 10:23 blake@deploy1003: blake: Backport for [[gerrit:1300072{{!}}ProductionServices: reboot poolcounter1007 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:22 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 10:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 10:21 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:21 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300072{{!}}ProductionServices: reboot poolcounter1007 (T426736)]] * 10:21 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:21 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:21 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:21 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:20 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet * 10:14 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1152: Security updates * 10:14 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:14 root@cumin1003: START - Cookbook sre.mysql.parsercache * 10:14 root@cumin1003: START - Cookbook sre.mysql.depool depool db1152: Security updates * 10:13 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet * 10:12 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300064{{!}}ProductionServices: reboot poolcounter1006.eqiad (T426736)]] (duration: 07m 46s) * 10:07 blake@deploy1003: blake: Continuing with deployment * 10:06 blake@deploy1003: blake: Backport for [[gerrit:1300064{{!}}ProductionServices: reboot poolcounter1006.eqiad (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:04 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300064{{!}}ProductionServices: reboot poolcounter1006.eqiad (T426736)]] * 09:57 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300058{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] (duration: 09m 32s) * 09:52 kharlan@deploy1003: kharlan: Continuing with deployment * 09:49 kharlan@deploy1003: kharlan: Backport for [[gerrit:1300058{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:47 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1300058{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] * 09:35 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox * 09:34 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 09:32 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 09:32 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 09:26 moritzm: upgrade routinator in eqiad to 0.15.2 [[phab:T428456|T428456]] * 09:23 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 09:23 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 09:22 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 09:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus5003.eqsin.wmnet to plain * 09:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus5003.eqsin.wmnet to plain * 09:15 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:04 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:54 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:20 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:09 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:09 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:07 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 08:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:04 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:01 fceratto@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host db1215.eqiad.wmnet with OS trixie * 07:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:53 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:48 javiermonton@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply * 07:48 javiermonton@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply * 07:44 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1215.eqiad.wmnet with reason: host reimage * 07:41 javiermonton@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply * 07:40 javiermonton@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply * 07:40 moritzm: installing openssl security updates * 07:39 fceratto@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1215.eqiad.wmnet with reason: host reimage * 07:38 javiermonton@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply * 07:37 javiermonton@deploy1003: helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply * 07:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:29 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299556{{!}}ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561{{!}}ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529{{!}}translate: adding separate read/write endpoints (T425377)]] (duration: 14m 03s) * 07:25 atsuko@deploy1003: atsuko: Continuing with deployment * 07:23 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db1215.eqiad.wmnet with OS trixie * 07:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1215.eqiad.wmnet with reason: Reimage * 07:21 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:20 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:20 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:17 atsuko@deploy1003: atsuko: Backport for [[gerrit:1299556{{!}}ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561{{!}}ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529{{!}}translate: adding separate read/write endpoints (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be veri * 07:16 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:15 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1299556{{!}}ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561{{!}}ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529{{!}}translate: adding separate read/write endpoints (T425377)]] * 07:14 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:12 atsukoito: backporting extensions/Translate to wmf/1.47.0-wmf.5 and applying the config * 07:12 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 06:45 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 05:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 05:43 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 05:42 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 05:41 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 47s) * 02:07 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1008.eqiad.wmnet with OS trixie * 02:03 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync * 02:02 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:52 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:51 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 01:51 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:50 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:50 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:49 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1008.eqiad.wmnet with reason: host reimage * 01:49 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 01:49 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:49 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:49 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 01:48 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 01:48 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 01:47 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 01:47 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 01:46 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 01:46 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 01:44 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 01:44 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 01:43 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 01:43 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1008.eqiad.wmnet with reason: host reimage * 01:25 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main1008 * 01:24 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main1008 * 01:24 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main1008 * 01:24 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main1008.eqiad.wmnet 45.32.64.10.in-addr.arpa 5.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 01:23 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main1008.eqiad.wmnet 45.32.64.10.in-addr.arpa 5.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 01:23 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 01:23 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1008 - jasmine@cumin2002" * 01:23 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1008 - jasmine@cumin2002" * 01:19 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 01:12 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main1008 * 01:11 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1008.eqiad.wmnet with OS trixie * 01:00 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2009.codfw.wmnet with OS trixie * 00:54 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync * 00:53 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync * 00:43 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2009.codfw.wmnet with reason: host reimage * 00:40 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:38 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2009.codfw.wmnet with reason: host reimage * 00:38 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 00:38 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:37 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:37 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 00:36 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 00:36 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 00:34 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 00:34 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 00:33 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 00:33 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 00:32 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 00:32 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 00:32 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2009 * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2009 * 00:15 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2009 * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2009.codfw.wmnet 33.48.192.10.in-addr.arpa 3.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:15 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2009.codfw.wmnet 33.48.192.10.in-addr.arpa 3.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2009 - jasmine@cumin2002" * 00:15 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2009 - jasmine@cumin2002" * 00:10 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 00:03 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2009 * 00:03 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2009.codfw.wmnet with OS trixie == 2026-06-09 == * 22:50 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299640{{!}}HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] (duration: 08m 59s) * 22:45 cscott@deploy1003: cscott: Continuing with deployment * 22:43 cscott@deploy1003: cscott: Backport for [[gerrit:1299640{{!}}HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:41 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299640{{!}}HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] * 22:15 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299639{{!}}[Bug] Donor Badge: Remove client prefs for control group (T428501)]] (duration: 20m 57s) * 22:11 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 22:07 mutante: gerrit - apache httpd log file location moved to /srv/gerrit/site_path/review_site/logs/ [[phab:T425667|T425667]] * 22:06 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on gerrit2003.wikimedia.org with reason: debug * 21:56 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1299639{{!}}[Bug] Donor Badge: Remove client prefs for control group (T428501)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:54 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299639{{!}}[Bug] Donor Badge: Remove client prefs for control group (T428501)]] * 21:52 ryankemper: [[phab:T428241|T428241]] removed retired wdqs2009 full-graph journal dump (446G x2, ~892G) from clouddumps100[1-2]:/srv/dumps/xmldatadumps/public/other/wdqs * 21:49 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299602{{!}}Revert "Create VectorComponentPageToolbar component" (T428649)]] (duration: 08m 16s) * 21:48 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 21:45 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 21:43 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1299602{{!}}Revert "Create VectorComponentPageToolbar component" (T428649)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:41 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299602{{!}}Revert "Create VectorComponentPageToolbar component" (T428649)]] * 21:34 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gerrit1003.wikimedia.org with reason: debug * 21:27 maryum: Deployed security fix for [[phab:T428324|T428324]] * 21:24 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 21:15 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart * 21:06 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart * 20:50 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs2002.codfw.wmnet with OS trixie * 20:50 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299588{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] (duration: 11m 13s) * 20:46 cscott@deploy1003: cscott: Continuing with deployment * 20:43 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs2002.codfw.wmnet with OS trixie * 20:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:42 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:41 cscott@deploy1003: cscott: Backport for [[gerrit:1299588{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:39 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299588{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] * 20:38 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:38 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:33 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:33 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:32 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299454{{!}}wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] (duration: 22m 08s) * 20:28 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:28 cscott@deploy1003: cscott, gkyziridis: Continuing with deployment * 20:24 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2004 * 20:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2004 * 20:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2003 * 20:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2003 * 20:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2002 * 20:13 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2002 * 20:13 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2001 * 20:13 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2001 * 20:12 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:12 cscott@deploy1003: cscott, gkyziridis: Backport for [[gerrit:1299454{{!}}wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:10 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299454{{!}}wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] * 20:09 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:04 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:59 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:54 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:53 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:48 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:47 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:47 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:46 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:46 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:45 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:45 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:28 ryankemper@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts wdqs1015.eqiad.wmnet * 19:28 ryankemper@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:28 ryankemper@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002" * 19:27 ryankemper@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002" * 19:20 ryankemper@cumin2002: START - Cookbook sre.dns.netbox * 19:15 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2008.codfw.wmnet with OS trixie * 19:15 ryankemper@cumin2002: START - Cookbook sre.hosts.decommission for hosts wdqs1015.eqiad.wmnet * 19:12 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync * 19:12 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync * 19:00 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:58 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 18:58 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2008.codfw.wmnet with reason: host reimage * 18:58 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 18:58 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 18:57 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 18:57 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 18:56 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 18:56 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 18:55 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 18:55 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 18:55 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 18:54 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 18:54 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:54 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 18:53 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 18:53 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 18:53 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 18:52 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:52 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2003 to codfw - jhancock@cumin2002" * 18:52 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2003 to codfw - jhancock@cumin2002" * 18:52 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 18:52 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 18:51 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2008.codfw.wmnet with reason: host reimage * 18:51 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 18:51 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 18:51 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 18:50 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 18:50 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 18:47 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:47 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:47 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:46 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:46 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:42 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:42 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:31 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 18:29 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2008.codfw.wmnet with OS trixie * 18:26 jasmine@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2008.codfw.wmnet with OS trixie * 17:48 mutante: https://releases.wikimedia.org {{!}} https://releases-jenkins.wikimedia.org - down for maintenance [[phab:T418299|T418299]] * 17:48 cmooney@dns2005: END - running authdns-update * 17:47 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet with reason: reimage * 17:47 cmooney@dns2005: START - running authdns-update * 17:46 sukhe: sudo cumin 'A:hcaptcha-proxy' 'run-puppet-agent': rolling out CR {{Gerrit|1299427}} [[phab:T428539|T428539]] * 17:43 jayme: kafka-main2008 is down due to hardware failure [[phab:T428654|T428654]] * 17:32 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf1002.eqiad.wmnet with OS trixie * 17:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf1002.eqiad.wmnet with reason: host reimage * 17:06 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf1002.eqiad.wmnet with reason: host reimage * 17:05 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2008 * 17:05 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2008 * 17:04 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 17:04 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2008 * 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2008.codfw.wmnet 4.32.192.10.in-addr.arpa 4.0.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:04 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 17:04 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2008.codfw.wmnet 4.32.192.10.in-addr.arpa 4.0.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2008 - jasmine@cumin2002" * 17:04 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5018 * 17:04 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2008 - jasmine@cumin2002" * 17:03 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5018.eqsin.wmnet with OS trixie * 16:58 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 16:58 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 16:57 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 16:57 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 16:57 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 16:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 16:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 16:50 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf1002.eqiad.wmnet with OS trixie * 16:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 16:47 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf1001.eqiad.wmnet with OS trixie * 16:47 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply * 16:47 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply * 16:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 16:41 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 16:41 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 16:35 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2008 * 16:34 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2008.codfw.wmnet with OS trixie * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:31 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply * 16:30 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply * 16:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf1001.eqiad.wmnet with reason: host reimage * 16:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:26 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf1001.eqiad.wmnet with reason: host reimage * 16:23 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply * 16:22 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: apply * 16:20 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:16 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:15 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:13 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:13 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:12 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf1001.eqiad.wmnet with OS trixie * 16:10 jiji@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. * 16:09 jiji@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. * 16:07 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf2002.codfw.wmnet with OS trixie * 16:02 jiji@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 16:02 jiji@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 16:00 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. * 15:59 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 15:59 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. * 15:59 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 15:59 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. * 15:59 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 15:58 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] DONE helmfile.d/services/termbox: apply * 15:58 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] START helmfile.d/services/termbox: apply * 15:57 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 15:57 jiji@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 15:57 lucaswerkmeister-wmde@deploy1003: helmfile [staging] DONE helmfile.d/services/termbox: apply * 15:56 lucaswerkmeister-wmde@deploy1003: helmfile [staging] START helmfile.d/services/termbox: apply * 15:54 jiji@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 15:53 jiji@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 15:51 jiji@deploy1003: Finished scap sync-world: redeploy {{Gerrit|1299468}} (duration: 07m 23s) * 15:49 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf2002.codfw.wmnet with reason: host reimage * 15:47 jiji@deploy1003: jiji: Continuing with deployment * 15:46 jiji@deploy1003: jiji: redeploy {{Gerrit|1299468}} synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:46 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf2002.codfw.wmnet with reason: host reimage * 15:45 jiji@deploy1003: Started scap sync-world: redeploy {{Gerrit|1299468}} * 15:43 brouberol@cumin1003: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-eqiad * 15:34 brennen@deploy1003: Finished deploy [phabricator/deployment@73e57ce]: deploy phab1004 for [[phab:T410849|T410849]] (followup for robots.txt) (duration: 00m 40s) * 15:33 brennen@deploy1003: Started deploy [phabricator/deployment@73e57ce]: deploy phab1004 for [[phab:T410849|T410849]] (followup for robots.txt) * 15:33 brennen@deploy1003: Finished deploy [phabricator/deployment@73e57ce]: deploy phab2002 for [[phab:T410849|T410849]] (followup for robots.txt) (duration: 00m 45s) * 15:32 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299468{{!}}ProductionServices.php: switch filebackend.php to rdb2015:6381 #2 (T418918 T291916)]] (duration: 07m 21s) * 15:32 brennen@deploy1003: Started deploy [phabricator/deployment@73e57ce]: deploy phab2002 for [[phab:T410849|T410849]] (followup for robots.txt) * 15:28 jiji@deploy1003: Rolling back deployment * 15:27 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf2002.codfw.wmnet with OS trixie * 15:27 jiji@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 15:26 jiji@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 15:25 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1299468{{!}}ProductionServices.php: switch filebackend.php to rdb2015:6381 #2 (T418918 T291916)]] * 15:22 urbanecm: Remove `migrateMentorStatusAwayToCommunityConfiguration` from updatelog on all wikis ([[phab:T409170|T409170]]; the script was only ever run as a dry-run) * 15:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 15:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 15:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf2001.codfw.wmnet with OS trixie * 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@d244a3e]: deploy phab1004 for [[phab:T410849|T410849]] (duration: 00m 42s) * 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@d244a3e]: deploy phab1004 for [[phab:T410849|T410849]] * 15:02 brennen@deploy1003: Finished deploy [phabricator/deployment@d244a3e]: deploy phab2002 for [[phab:T410849|T410849]] (duration: 00m 45s) * 15:01 brennen@deploy1003: Started deploy [phabricator/deployment@d244a3e]: deploy phab2002 for [[phab:T410849|T410849]] * 14:58 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf2001.codfw.wmnet with reason: host reimage * 14:52 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf2001.codfw.wmnet with reason: host reimage * 14:52 arnaudb@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phab[2002-2003].codfw.wmnet,phab[1004-1006].eqiad.wmnet with reason: [[phab:T410849|T410849]] * 14:47 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 14:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 14:40 moritzm: upgrade routinator in codfw to 0.15.2 [[phab:T428456|T428456]] * 14:35 brouberol@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad * 14:33 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf2001.codfw.wmnet with OS trixie * 14:26 brouberol@cumin1003: END (ERROR) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=97) rolling reboot on A:cephosd-eqiad * 14:26 brouberol@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad * 14:20 btullis@cumin1003: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-codfw * 14:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host parsoidtest1001.eqiad.wmnet * 14:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2153: Migration of db2153.codfw.wmnet completed * 14:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of rpki2003.codfw.wmnet to drbd * 14:14 moritzm: imported routinator 0.15.2-1bookworm to thirdparty/routinator for bookworm-wikimedia [[phab:T428456|T428456]] * 14:12 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:12 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1184: Migration of db1184.eqiad.wmnet completed * 14:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host parsoidtest1001.eqiad.wmnet * 14:07 Dreamy_Jazz: Afternoon UTC backport window done * 14:07 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 14:06 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299495{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502{{!}}SecurePollLogPager: Cast user IDs to ints before use (T428599)]] (duration: 06m 53s) * 14:06 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 14:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: rack depool * 14:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of rpki2003.codfw.wmnet to drbd * 14:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow2004.codfw.wmnet to drbd * 14:02 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:02 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1299495{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502{{!}}SecurePollLogPager: Cast user IDs to ints before use (T428599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:59 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1299495{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502{{!}}SecurePollLogPager: Cast user IDs to ints before use (T428599)]] * 13:58 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:58 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:56 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 13:56 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 13:55 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:55 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * {{safesubst:SAL entry|1=13:55 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298929{{!}}Simplify fragment processing (T423700)]], [[gerrit:1298926{{!}}Move ::getFragmentsToTransform() to Content<nowiki>{</nowiki>Text,DOM<nowiki>}</nowiki>TransformStage]], [[gerrit:1298927{{!}}OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925{{!}}Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497}} * 13:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:51 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow2004.codfw.wmnet to drbd * 13:50 cscott@deploy1003: cscott: Continuing with deployment * 13:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2045.codfw.wmnet to cluster codfw and group A * 13:48 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2045.codfw.wmnet to cluster codfw and group A * 13:48 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2027.codfw.wmnet to cluster codfw and group A * 13:47 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2027.codfw.wmnet to cluster codfw and group A * 13:46 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 13:45 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 13:44 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * {{safesubst:SAL entry|1=13:42 cscott@deploy1003: cscott: Backport for [[gerrit:1298929{{!}}Simplify fragment processing (T423700)]], [[gerrit:1298926{{!}}Move ::getFragmentsToTransform() to Content<nowiki>{</nowiki>Text,DOM<nowiki>}</nowiki>TransformStage]], [[gerrit:1298927{{!}}OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925{{!}}Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497{{!}}Store indicators}} * 13:41 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * {{safesubst:SAL entry|1=13:40 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1298929{{!}}Simplify fragment processing (T423700)]], [[gerrit:1298926{{!}}Move ::getFragmentsToTransform() to Content<nowiki>{</nowiki>Text,DOM<nowiki>}</nowiki>TransformStage]], [[gerrit:1298927{{!}}OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925{{!}}Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497{{!}}}} * 13:40 btullis@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-codfw * 13:39 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 13:37 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 13:35 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 13:33 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 13:32 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 13:32 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298834{{!}}config: Disable EmailConfirmationBanner on all wikis (T428291)]] (duration: 07m 01s) * 13:30 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2153: Migration of db2153.codfw.wmnet completed * 13:28 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 lucaswerkmeister-wmde@deploy1003: mmartorana, lucaswerkmeister-wmde: Continuing with deployment * 13:27 lucaswerkmeister-wmde@deploy1003: mmartorana, lucaswerkmeister-wmde: Backport for [[gerrit:1298834{{!}}config: Disable EmailConfirmationBanner on all wikis (T428291)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:26 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1184: Migration of db1184.eqiad.wmnet completed * 13:25 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298834{{!}}config: Disable EmailConfirmationBanner on all wikis (T428291)]] * 13:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 13:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 13:23 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:21 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 13:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2153.codfw.wmnet with OS trixie * 13:20 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2241: rack depool * 13:20 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1237: repool after maintenance db1237 * 13:19 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298654{{!}}Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] (duration: 09m 40s) * 13:17 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2006.codfw.wmnet * 13:17 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2006.codfw.wmnet * 13:16 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2251-2253].codfw.wmnet * 13:16 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2251-2253].codfw.wmnet * 13:16 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2005.codfw.wmnet * 13:16 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2005.codfw.wmnet * 13:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1184.eqiad.wmnet with OS trixie * 13:14 lucaswerkmeister-wmde@deploy1003: neriah, lucaswerkmeister-wmde: Continuing with deployment * 13:11 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 * 13:11 lucaswerkmeister-wmde@deploy1003: neriah, lucaswerkmeister-wmde: Backport for [[gerrit:1298654{{!}}Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:09 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298654{{!}}Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] * 13:04 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:04 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:04 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2153.codfw.wmnet with reason: host reimage * 13:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:03 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1015.eqiad.wmnet with OS trixie * 12:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage * 12:58 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2153.codfw.wmnet with reason: host reimage * 12:57 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1016.eqiad.wmnet with OS trixie * 12:57 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 * 12:56 XioNoX: lsw1-a4-codfw> request system reboot - [[phab:T427357|T427357]] * 12:55 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:53 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage * 12:50 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299477{{!}}hCaptcha: Roll out to all wikis for api account creation. (T426050)]] (duration: 07m 21s) * 12:46 kharlan@deploy1003: kharlan, dbrant: Continuing with deployment * 12:46 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 * 12:45 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1015.eqiad.wmnet with reason: host reimage * 12:45 kharlan@deploy1003: kharlan, dbrant: Backport for [[gerrit:1299477{{!}}hCaptcha: Roll out to all wikis for api account creation. (T426050)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:45 topranks: shut sub-interfaces for row A/B legacy vlans on cr1-codfw [[phab:T427357|T427357]] * 12:45 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 * 12:43 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1299477{{!}}hCaptcha: Roll out to all wikis for api account creation. (T426050)]] * 12:42 topranks: increase OSPF cost on ssw1-a1-codfw link to lsw1-a4-codfw to force traffic via alternate spine [[phab:T427357|T427357]] * 12:41 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299478{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]] (duration: 07m 02s) * 12:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1016.eqiad.wmnet with reason: host reimage * 12:40 moritzm: installing wireshark security updates * 12:40 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2153.codfw.wmnet with OS trixie * 12:38 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1184.eqiad.wmnet with OS trixie * 12:37 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 12:36 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1299478{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:34 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2153: Upgrading db2153.codfw.wmnet * 12:34 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1237: repool after maintenance db1237 * 12:34 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1299478{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]] * 12:34 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2153: Upgrading db2153.codfw.wmnet * 12:34 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1184: Upgrading db1184.eqiad.wmnet * 12:33 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1184: Upgrading db1184.eqiad.wmnet * 12:33 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:32 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1237.eqiad.wmnet with OS trixie * 12:32 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1015.eqiad.wmnet with reason: host reimage * 12:32 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1016.eqiad.wmnet with reason: host reimage * 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 12:27 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2005.codfw.wmnet * 12:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2046: repool after maintenance * 12:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2006.codfw.wmnet * 12:23 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298829{{!}}wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] (duration: 16m 04s) * 12:23 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2006.codfw.wmnet * 12:22 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2251-2253].codfw.wmnet * 12:22 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2005.codfw.wmnet * 12:20 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2251-2253].codfw.wmnet * 12:20 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 12:20 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: rack depool * 12:20 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 12:20 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2241: rack depool * 12:19 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1016 * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1016 * 12:19 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1015 * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1015 * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1016.eqiad.wmnet with OS trixie * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1015.eqiad.wmnet with OS trixie * 12:17 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 * 12:17 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 24 hosts with reason: Rack A4 depool * 12:16 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Continuing with deployment * 12:15 topranks: drain traffic on ssw1-a1-codfw - add gshut community in evpn underlay - [[phab:T427357|T427357]] * 12:14 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 * 12:13 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Backport for [[gerrit:1298829{{!}}wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:10 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1237.eqiad.wmnet with reason: host reimage * 12:07 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1298829{{!}}wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] * 12:05 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1237.eqiad.wmnet with reason: host reimage * 12:00 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Dmaza out of all services on: 2435 hosts * 11:51 atsuko@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 11:51 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie * 11:49 atsuko@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 11:48 atsuko@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 11:47 atsuko@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 11:45 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 11:44 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 11:43 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:43 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2046: repool after maintenance * 11:38 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 11:36 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:36 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2046.codfw.wmnet with OS trixie * 11:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2185.codfw.wmnet with reason: Reimage * 11:31 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging HMonroy out of all services on: 2435 hosts * 11:28 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging KSiebert out of all services on: 2435 hosts * 11:26 slyngs: CAS-SSO upgrade to version 7.3.7.2 * 11:26 slyngshede@dns1004: END - running authdns-update * 11:24 slyngshede@dns1004: START - running authdns-update * 11:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2046.codfw.wmnet with reason: host reimage * 11:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1043: repool after upgrade * 11:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2046.codfw.wmnet with reason: host reimage * 10:55 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2046.codfw.wmnet with OS trixie * 10:53 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2046: Upgrading es2046.codfw.wmnet * 10:53 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2046: Upgrading es2046.codfw.wmnet * 10:52 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply * 10:52 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:52 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply * 10:52 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply * 10:52 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:52 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply * 10:51 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:32 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1043: repool after upgrade * 10:31 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:28 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1160: Repooling * 10:26 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1043.eqiad.wmnet with OS trixie * 10:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:17 elukey: complete rollout of apache2 upgrades * 10:16 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:15 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:13 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:12 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:12 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:08 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1043.eqiad.wmnet with reason: host reimage * 10:04 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:04 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1043.eqiad.wmnet with reason: host reimage * 10:04 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:04 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:04 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:04 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:04 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:57 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1160: Repooling * 09:51 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 09:51 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 09:50 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 09:50 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 09:49 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1043.eqiad.wmnet with OS trixie * 09:48 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool es1043: Upgrading es1043.eqiad.wmnet * 09:48 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 09:47 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:45 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 09:41 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 09:36 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=5 --verbose --last-checked="20260603"` (after stopping previous scan run) * 09:34 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=5 --verbose` (after stopping previous scan run) * 09:27 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 09:26 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 09:17 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 09:17 fceratto@cumin1003: MariaDB change: Setting sections s5 as read-write * 09:17 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 09:14 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1043: Upgrading es1043.eqiad.wmnet * 09:14 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:12 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1042 to es4 eqiad primary [[phab:T428386|T428386]]', diff saved to https://phabricator.wikimedia.org/P93943 and previous config saved to /var/cache/conftool/dbconfig/20260609-091215-marostegui.json * 09:11 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1043 to es4 eqiad primary [[phab:T428386|T428386]]', diff saved to https://phabricator.wikimedia.org/P93942 and previous config saved to /var/cache/conftool/dbconfig/20260609-091147-marostegui.json * 09:03 jiji@cumin1003: conftool action : set/pooled=yes; selector: service=docker-registry,name=registry2005.codfw.wmnet * 08:59 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:59 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:57 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1237.eqiad.wmnet with OS trixie * 08:55 jiji@cumin1003: conftool action : set/pooled=no; selector: service=docker-registry,name=registry2005.codfw.wmnet * 08:55 jiji@cumin1003: conftool action : set/pooled=yes; selector: service=docker-registry,name=registry2004.codfw.wmnet * 08:50 jiji@cumin1003: conftool action : set/pooled=no; selector: service=docker-registry,name=registry2004.codfw.wmnet * 08:22 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=docker-registry,name=codfw * 08:22 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=docker-registry,name=eqiad * 08:08 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=docker-registry,name=eqiad * 08:08 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=docker-registry,name=codfw * 07:59 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:59 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix typoes - ayounsi@cumin1003" * 07:59 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix typoes - ayounsi@cumin1003" * 07:52 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 07:47 brouberol@dns1004: END - running authdns-update * 07:46 brouberol@dns1004: START - running authdns-update * 07:44 brouberol@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/kafka-ui: apply * 07:43 brouberol@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/kafka-ui: apply * 07:43 brouberol@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/kafka-ui: apply * 07:42 brouberol@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/kafka-ui: apply * 07:41 brouberol@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/kafka-ui: apply * 07:39 brouberol@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/kafka-ui: apply * 07:38 brouberol@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:37 brouberol@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 07:37 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie * 07:36 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.major-upgrade (exit_code=97) * 07:36 brouberol@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:36 brouberol@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:26 fceratto@dns1004: END - running authdns-update * 07:24 fceratto@dns1004: START - running authdns-update * 07:22 marostegui@dns1004: END - running authdns-update * 07:21 marostegui@dns1004: START - running authdns-update * 07:19 elukey@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:19 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Fix dse-k8s-wdqs2002 duplicate ipv6 address - elukey@cumin1003" * 07:19 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Fix dse-k8s-wdqs2002 duplicate ipv6 address - elukey@cumin1003" * 07:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1160.eqiad.wmnet with reason: Maintenance * 07:12 elukey@cumin1003: START - Cookbook sre.dns.netbox * 07:11 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1160: Repooling * 07:11 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1160: Repooling * 07:11 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1160: Repooling * 07:11 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1160: Repooling * 07:00 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:00 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1237.eqiad.wmnet with OS trixie * 06:24 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1160 [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93940 and previous config saved to /var/cache/conftool/dbconfig/20260609-062412-fceratto.json * 06:17 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 06:16 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 06:16 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 06:16 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 06:15 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 06:15 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 06:15 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 06:14 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 06:12 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1244 to s4 primary and set section read-write [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93939 and previous config saved to /var/cache/conftool/dbconfig/20260609-061222-fceratto.json * 06:11 fceratto@cumin1003: dbctl commit (dc=all): 'Set s4 eqiad as read-only for maintenance - [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93938 and previous config saved to /var/cache/conftool/dbconfig/20260609-061131-fceratto.json * 06:10 federico3: Starting s4 eqiad failover from db1160 to db1244 - [[phab:T426086|T426086]] * 06:01 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1244 with weight 0 [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93937 and previous config saved to /var/cache/conftool/dbconfig/20260609-060121-fceratto.json * 06:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 40 hosts with reason: Primary switchover s4 [[phab:T426086|T426086]] * 05:40 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie * 05:37 marostegui@dns1004: START - running authdns-update * 05:27 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1237: Upgrading db1237.eqiad.wmnet * 05:27 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1237: Upgrading db1237.eqiad.wmnet * 05:27 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:24 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1237 [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93935 and previous config saved to /var/cache/conftool/dbconfig/20260609-052420-marostegui.json * 05:23 marostegui@dns1004: START - running authdns-update * 05:23 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93934 and previous config saved to /var/cache/conftool/dbconfig/20260609-052311-marostegui.json * 05:22 marostegui@cumin1003: dbctl commit (dc=all): 'Set x1 eqiad as read-only for maintenance - [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93933 and previous config saved to /var/cache/conftool/dbconfig/20260609-052253-marostegui.json * 05:22 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T428158|T428158]] * 05:19 marostegui@cumin1003: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93932 and previous config saved to /var/cache/conftool/dbconfig/20260609-051859-marostegui.json * 05:18 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: Primary switchover x1 [[phab:T428158|T428158]] * 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.3 (duration: 02m 43s) * 03:40 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] (duration: 37m 16s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 38s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-06-08 == * 22:00 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298915{{!}}CommonSettings: Set $wgScoreSafeMode = false (T428484)]] (duration: 07m 42s) * 21:56 reedy@deploy1003: reedy: Continuing with deployment * 21:54 reedy@deploy1003: reedy: Backport for [[gerrit:1298915{{!}}CommonSettings: Set $wgScoreSafeMode = false (T428484)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:53 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1298915{{!}}CommonSettings: Set $wgScoreSafeMode = false (T428484)]] * 21:12 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298891{{!}}OOUIHTMLForm: Avoid treating form header as a clickable label (T428359)]] (duration: 08m 10s) * 21:07 mlitn@deploy1003: mlitn, neriah: Continuing with deployment * 21:05 mlitn@deploy1003: mlitn, neriah: Backport for [[gerrit:1298891{{!}}OOUIHTMLForm: Avoid treating form header as a clickable label (T428359)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:03 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1298891{{!}}OOUIHTMLForm: Avoid treating form header as a clickable label (T428359)]] * 20:43 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297162{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias]], [[gerrit:1298841{{!}}Squashed diff to master]] (duration: 07m 05s) * 20:39 mlitn@deploy1003: mlitn: Continuing with deployment * 20:38 mlitn@deploy1003: mlitn: Backport for [[gerrit:1297162{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias]], [[gerrit:1298841{{!}}Squashed diff to master]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:36 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1297162{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias]], [[gerrit:1298841{{!}}Squashed diff to master]] * 20:29 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298390{{!}}English Wikibooks: update FlaggedRevs configuration (T428329)]], [[gerrit:1298328{{!}}English Wikiversity: Add new user group "autopatrolled" (T428269)]] (duration: 08m 58s) * 20:25 mlitn@deploy1003: mlitn, vadymts1: Continuing with deployment * 20:22 mlitn@deploy1003: mlitn, vadymts1: Backport for [[gerrit:1298390{{!}}English Wikibooks: update FlaggedRevs configuration (T428329)]], [[gerrit:1298328{{!}}English Wikiversity: Add new user group "autopatrolled" (T428269)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1298390{{!}}English Wikibooks: update FlaggedRevs configuration (T428329)]], [[gerrit:1298328{{!}}English Wikiversity: Add new user group "autopatrolled" (T428269)]] * 20:03 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298879{{!}}SimpleCaptcha: Re-render captcha when edit form is redisplayed (T428437)]] (duration: 37m 43s) * 19:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:31 kharlan@deploy1003: kharlan: Continuing with deployment * 19:30 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:30 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:29 kharlan@deploy1003: kharlan: Backport for [[gerrit:1298879{{!}}SimpleCaptcha: Re-render captcha when edit form is redisplayed (T428437)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 19:28 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:27 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:25 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1298879{{!}}SimpleCaptcha: Re-render captcha when edit form is redisplayed (T428437)]] * 19:24 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab (duration: 01m 32s) * 19:23 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:22 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab * 19:20 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab (duration: 01m 40s) * 19:19 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab * 19:16 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:14 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:06 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:59 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2004 * 18:52 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2004 * 18:52 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2003 * 18:52 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2003 * 18:51 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:51 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2004 to codfw - jhancock@cumin2002" * 18:51 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2004 to codfw - jhancock@cumin2002" * 18:44 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:42 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:42 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2030 to codfw - jhancock@cumin2002" * 18:42 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2030 to codfw - jhancock@cumin2002" * 18:37 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:33 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2002 * 18:32 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2002 * 18:31 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:31 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2002 to codfw - jhancock@cumin2002" * 18:31 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2002 to codfw - jhancock@cumin2002" * 18:25 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:22 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2001 * 18:22 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2001 * 18:21 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:21 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: updating dse-k8s-wdqs2001 to codfw - jhancock@cumin2002" * 18:21 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: updating dse-k8s-wdqs2001 to codfw - jhancock@cumin2002" * 18:17 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:02 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T427286|T427286]] (duration: 00m 12s) * 18:02 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T427286|T427286]] * 17:37 jnuche@deploy1003: Installation of scap version "4.268.0" completed for 2 hosts * 17:35 jnuche@deploy1003: Installing scap version "4.268.0" for 2 host(s) * 17:21 claime: restarting varnish-frontend service on cp6012 * 17:21 claime: restarting varnish-frontend service on cp6011 * 17:21 claime: restarted varnish-frontend service on cp6009 * 17:13 taavi: bounce sirenbot to get it to re-join a channel * 17:05 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 17:05 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:58 urbanecm@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 16:57 urbanecm@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 16:55 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 16:53 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 16:53 urbanecm@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 16:52 urbanecm@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 16:30 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 16:29 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 16:29 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 16:28 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 16:28 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 16:28 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 16:28 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 16:27 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 16:27 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 16:26 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 16:26 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 16:25 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 16:18 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 16:17 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 16:17 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 16:16 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 16:16 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 16:16 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 16:16 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 16:15 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 16:14 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 16:14 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 16:14 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 16:14 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 16:13 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 16:13 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 16:13 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 16:12 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 16:12 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 16:10 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 16:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 16:10 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 16:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 16:10 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 16:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 16:09 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 16:08 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 16:08 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 16:07 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 16:06 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 15:57 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2042: repool after upgrade * 15:45 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db[2183-2184].codfw.wmnet * 15:45 jynus@cumin2002: START - Cookbook sre.hosts.remove-downtime for db[2183-2184].codfw.wmnet * 15:18 jynus: dbmaint on backup1-codfw@codfw ([[phab:T428467|T428467]]) * 15:12 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2042: repool after upgrade * 15:12 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 15:09 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 15:09 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 15:09 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 15:07 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2042.codfw.wmnet with OS trixie * 15:04 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 15:04 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 15:03 jynus@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db[2183-2184].codfw.wmnet with reason: Switchover db * 15:03 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 15:03 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 15:02 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 15:01 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply * 15:00 eevans@deploy1003: helmfile [staging] START helmfile.d/services/data-gateway: apply * 14:59 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:55 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:55 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:54 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:50 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:50 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:50 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:49 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:49 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2042.codfw.wmnet with reason: host reimage * 14:42 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2042.codfw.wmnet with reason: host reimage * 14:32 Lucas_WMDE: UTC afternoon backport+config window done * 14:32 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298709{{!}}Add translatable messages for WikiProject names (T427804)]], [[gerrit:1298710{{!}}Use translatable messages for WikiProject links (T427804)]], [[gerrit:1297644{{!}}WikiProject links - remove 'text' config (T427804)]] (duration: 31m 57s) * 14:27 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 14:26 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2042.codfw.wmnet with OS trixie * 14:26 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 14:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2042: Upgrading es2042.codfw.wmnet * 14:25 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2042: Upgrading es2042.codfw.wmnet * 14:25 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:24 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2043 to es4 codfw primary [[phab:T428386|T428386]]', diff saved to https://phabricator.wikimedia.org/P93926 and previous config saved to /var/cache/conftool/dbconfig/20260608-142423-marostegui.json * 14:23 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 14:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1041: repool after maintenance * 14:19 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Continuing with deployment * 14:18 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Backport for [[gerrit:1298709{{!}}Add translatable messages for WikiProject names (T427804)]], [[gerrit:1298710{{!}}Use translatable messages for WikiProject links (T427804)]], [[gerrit:1297644{{!}}WikiProject links - remove 'text' config (T427804)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:11 cgoubert@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=liftwing-openapi-server.* * 14:10 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp6013.* * 14:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:05 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 14:05 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 13:54 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 13:52 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 13:50 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 13:50 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 13:50 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296550{{!}}hCaptcha: Don't show AbuseFilter CAPTCHA for wbsetclaim API (T427608)]] (duration: 08m 31s) * 13:48 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 13:46 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 13:43 cgoubert@dns1004: END - running authdns-update * 13:43 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296550{{!}}hCaptcha: Don't show AbuseFilter CAPTCHA for wbsetclaim API (T427608)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:41 cgoubert@dns1004: START - running authdns-update * 13:41 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296550{{!}}hCaptcha: Don't show AbuseFilter CAPTCHA for wbsetclaim API (T427608)]] * 13:39 urbanecm@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * {{safesubst:SAL entry|1=13:38 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298758{{!}}feat(V2): toggle experiment features based on custom url override (T424646)]], [[gerrit:1298762{{!}}specialCreateAccount: use GECreateAccountExperimentV2 instead of hook (T424646)]], [[gerrit:1298764{{!}}fix: correctly read experiments param on Special:UserLogin]], [[gerrit:1298765{{!}}signup.js: use JS var instead of TestKitchen to show exp}} * 13:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1041: repool after maintenance * 13:38 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 13:38 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:37 urbanecm@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 13:36 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 13:35 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1041.eqiad.wmnet with OS trixie * 13:34 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 13:34 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2041: repool after upgrade * 13:34 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Continuing with deployment * 13:34 urbanecm@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 13:32 urbanecm@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * {{safesubst:SAL entry|1=13:30 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Backport for [[gerrit:1298758{{!}}feat(V2): toggle experiment features based on custom url override (T424646)]], [[gerrit:1298762{{!}}specialCreateAccount: use GECreateAccountExperimentV2 instead of hook (T424646)]], [[gerrit:1298764{{!}}fix: correctly read experiments param on Special:UserLogin]], [[gerrit:1298765{{!}}signup.js: use JS var instead of TestKitchen to show}} * {{safesubst:SAL entry|1=13:29 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298758{{!}}feat(V2): toggle experiment features based on custom url override (T424646)]], [[gerrit:1298762{{!}}specialCreateAccount: use GECreateAccountExperimentV2 instead of hook (T424646)]], [[gerrit:1298764{{!}}fix: correctly read experiments param on Special:UserLogin]], [[gerrit:1298765{{!}}signup.js: use JS var instead of TestKitchen to show expe}} * 13:21 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298418{{!}}NewUserMessage: Add $wgNewUserMessageOnAutoCreateFirstEdit (T426206)]], [[gerrit:1298717{{!}}Replace NewUserMessageOnAutoCreateFirstEdit with wgNewUserMessageOnFirstEdit (T426206)]], [[gerrit:1298734{{!}}Enable wgNewUserMessageOnFirstEdit on incubatorwiki (T426206)]] (duration: 11m 06s) * 13:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1041.eqiad.wmnet with reason: host reimage * 13:17 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Continuing with deployment * 13:12 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 13:12 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Backport for [[gerrit:1298418{{!}}NewUserMessage: Add $wgNewUserMessageOnAutoCreateFirstEdit (T426206)]], [[gerrit:1298717{{!}}Replace NewUserMessageOnAutoCreateFirstEdit with wgNewUserMessageOnFirstEdit (T426206)]], [[gerrit:1298734{{!}}Enable wgNewUserMessageOnFirstEdit on incubatorwiki (T426206)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki * 13:12 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 13:12 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1041.eqiad.wmnet with reason: host reimage * 13:11 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 13:11 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 13:10 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298418{{!}}NewUserMessage: Add $wgNewUserMessageOnAutoCreateFirstEdit (T426206)]], [[gerrit:1298717{{!}}Replace NewUserMessageOnAutoCreateFirstEdit with wgNewUserMessageOnFirstEdit (T426206)]], [[gerrit:1298734{{!}}Enable wgNewUserMessageOnFirstEdit on incubatorwiki (T426206)]] * 12:57 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298767{{!}}Follow-up: Allow CaptchaConsequence to be skipped via hook (T427608)]] (duration: 06m 20s) * 12:57 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1041.eqiad.wmnet with OS trixie * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:56 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1041: Upgrading es1041.eqiad.wmnet * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:55 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1041: Upgrading es1041.eqiad.wmnet * 12:55 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:54 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:53 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 12:53 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1298767{{!}}Follow-up: Allow CaptchaConsequence to be skipped via hook (T427608)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:51 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:51 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1298767{{!}}Follow-up: Allow CaptchaConsequence to be skipped via hook (T427608)]] * 12:49 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:49 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2041: repool after upgrade * 12:49 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:47 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:46 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 12:44 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 12:43 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 12:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 12:41 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 12:40 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2063.codfw.wmnet with OS bullseye * 12:32 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2062.codfw.wmnet with OS bullseye * 12:27 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2041.codfw.wmnet with OS trixie * 12:21 joal@deploy1003: Finished deploy [analytics/refinery@d67c584] (thin): Regular analytics weekly train THIN [analytics/refinery@d67c584f] (duration: 02m 00s) * 12:19 joal@deploy1003: Started deploy [analytics/refinery@d67c584] (thin): Regular analytics weekly train THIN [analytics/refinery@d67c584f] * 12:19 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2063.codfw.wmnet with reason: host reimage * 12:18 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 12:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 12:16 joal@deploy1003: Finished deploy [analytics/refinery@d67c584]: Regular analytics weekly train [analytics/refinery@d67c584f] (duration: 07m 52s) * 12:15 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2063.codfw.wmnet with reason: host reimage * 12:13 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2062.codfw.wmnet with reason: host reimage * 12:09 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2041.codfw.wmnet with reason: host reimage * 12:08 joal@deploy1003: Started deploy [analytics/refinery@d67c584]: Regular analytics weekly train [analytics/refinery@d67c584f] * 12:08 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2062.codfw.wmnet with reason: host reimage * 12:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add eqiad e8 public vlans - ayounsi@cumin1003" * 12:06 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add eqiad e8 public vlans - ayounsi@cumin1003" * 12:03 joal@deploy1003: Finished deploy [analytics/refinery@d67c584] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@d67c584f] (duration: 02m 00s) * 12:03 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2041.codfw.wmnet with reason: host reimage * 12:01 joal@deploy1003: Started deploy [analytics/refinery@d67c584] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@d67c584f] * 12:01 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 12:00 ayounsi@cumin1003: END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) * 12:00 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 12:00 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 12:00 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:57 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2063 * 11:57 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2063 * 11:57 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2063 * 11:57 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2063.codfw.wmnet 52.16.192.10.in-addr.arpa 2.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:56 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2063.codfw.wmnet 52.16.192.10.in-addr.arpa 2.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:56 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:56 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2063 - mvernon@cumin2002" * 11:56 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2063 - mvernon@cumin2002" * 11:51 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:51 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2063 * 11:50 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2063.codfw.wmnet with OS bullseye * 11:50 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2062 * 11:50 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2062 * 11:49 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2062 * 11:49 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2062.codfw.wmnet 123.0.192.10.in-addr.arpa 3.2.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:49 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2062.codfw.wmnet 123.0.192.10.in-addr.arpa 3.2.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:49 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:49 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2062 - mvernon@cumin2002" * 11:49 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2062 - mvernon@cumin2002" * 11:47 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2041.codfw.wmnet with OS trixie * 11:45 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2041: Upgrading es2041.codfw.wmnet * 11:45 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2041: Upgrading es2041.codfw.wmnet * 11:44 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:44 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.major-upgrade (exit_code=97) * 11:44 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:44 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1042: repool after maintenance * 11:43 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:43 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2062 * 11:42 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2062.codfw.wmnet with OS bullseye * 11:30 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298728{{!}}SpecialMediaSearch: Prefer thumb steps over thumb limits (T424032)]] (duration: 17m 39s) * 11:25 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 11:18 Raine: progressively switching shellbox to bookworm (start) * 11:15 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 11:14 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 11:14 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1298728{{!}}SpecialMediaSearch: Prefer thumb steps over thumb limits (T424032)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:13 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 11:12 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 11:12 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1298728{{!}}SpecialMediaSearch: Prefer thumb steps over thumb limits (T424032)]] * 11:02 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be2062 * 11:02 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be2063 * 10:58 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1042: repool after maintenance * 10:58 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:56 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1042.eqiad.wmnet with OS trixie * 10:47 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298721{{!}}GuessedThumbnailInfo: Also allow showing webp originals (T428202)]] (duration: 16m 41s) * 10:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1042.eqiad.wmnet with reason: host reimage * 10:39 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 10:39 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 10:38 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 10:36 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2160.codfw.wmnet * 10:36 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2160.codfw.wmnet * 10:35 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2043: repool after upgrade * 10:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2160.codfw.wmnet with reason: Reboot * 10:34 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1298721{{!}}GuessedThumbnailInfo: Also allow showing webp originals (T428202)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:34 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1042.eqiad.wmnet with reason: host reimage * 10:30 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1298721{{!}}GuessedThumbnailInfo: Also allow showing webp originals (T428202)]] * 10:18 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1042.eqiad.wmnet with OS trixie * 10:18 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:18 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:18 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:18 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:15 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1042: Upgrading es1042.eqiad.wmnet * 10:14 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:14 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:14 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:14 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:13 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1042: Upgrading es1042.eqiad.wmnet * 10:13 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:12 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be2063 * 10:09 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be2062 * 10:07 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:07 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:07 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:06 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 09:52 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply * 09:52 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply * 09:50 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply * 09:49 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply * 09:49 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2043: repool after upgrade * 09:49 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2043.codfw.wmnet with OS trixie * 09:44 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 09:44 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 09:42 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: sync * 09:42 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: sync * 09:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2043.codfw.wmnet with reason: host reimage * 09:27 jelto@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org * 09:23 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2043.codfw.wmnet with reason: host reimage * 09:17 jelto@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org * 09:15 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: sync * 09:15 ozge@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: sync * 09:07 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2043.codfw.wmnet with OS trixie * 09:06 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2043: Upgrading es2043.codfw.wmnet * 09:06 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2043: Upgrading es2043.codfw.wmnet * 09:05 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:41 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1217.eqiad.wmnet with OS trixie * 08:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1217.eqiad.wmnet with reason: host reimage * 08:15 taavi@cumin1003: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database urwikisource ([[phab:T415977|T415977]]) * 08:14 taavi@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database urwikisource ([[phab:T415977|T415977]]) * 08:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1217.eqiad.wmnet with reason: host reimage * 08:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2052: repool after upgrade * 08:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1051: repool after maintenance * 08:03 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Managing sanitization for wikis urwikisource in section s5 * 07:55 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1217.eqiad.wmnet with OS trixie * 07:53 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1217.eqiad.wmnet with reason: reimage * 07:53 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis urwikisource in section s5 * 07:52 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Checking sanitization for wikis urwikisource in section s5 * 07:50 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis urwikisource in section s5 * 07:50 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.sanitize-wiki (exit_code=97) Managing sanitization for wikis urwikisource in section s5 * 07:50 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis urwikisource in section s5 * 07:44 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297681{{!}}Global rollout - Sub-ref deployments to Group 0, Group 1 and frwiki (T425662)]] (duration: 32m 51s) * 07:32 wmde-fisch@deploy1003: wmde-fisch, lilients: Continuing with deployment * 07:29 wmde-fisch@deploy1003: wmde-fisch, lilients: Backport for [[gerrit:1297681{{!}}Global rollout - Sub-ref deployments to Group 0, Group 1 and frwiki (T425662)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:21 elukey: upgrade sudo package on an-* hosts for [[phab:T428384|T428384]] * 07:18 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2052: repool after upgrade * 07:18 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1051: repool after maintenance * 07:17 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:17 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:12 taavi@cumin1003: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database urwikisource ([[phab:T415977|T415977]]) * 07:12 elukey: upgrade exim4 packages on seaborgium for security upgrades * 07:11 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1297681{{!}}Global rollout - Sub-ref deployments to Group 0, Group 1 and frwiki (T425662)]] * 06:36 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1051.eqiad.wmnet with OS trixie * 06:20 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1051.eqiad.wmnet with reason: host reimage * 06:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1051.eqiad.wmnet with reason: host reimage * 06:15 taavi@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database urwikisource ([[phab:T415977|T415977]]) * 05:58 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1051.eqiad.wmnet with OS trixie * 05:54 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2052.codfw.wmnet with OS trixie * 05:44 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool es1051: Upgrading es1051.eqiad.wmnet * 05:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2052.codfw.wmnet with reason: host reimage * 05:35 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2052.codfw.wmnet with reason: host reimage * 05:35 marostegui@dns1004: END - running authdns-update * 05:34 marostegui@dns1004: START - running authdns-update * 05:33 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1051: Upgrading es1051.eqiad.wmnet * 05:33 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:31 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1054 to es3 eqiad primary [[phab:T428050|T428050]]', diff saved to https://phabricator.wikimedia.org/P93895 and previous config saved to /var/cache/conftool/dbconfig/20260608-053156-marostegui.json * 05:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2052.codfw.wmnet with OS trixie * 05:18 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2052: Upgrading es2052.codfw.wmnet * 05:18 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2052: Upgrading es2052.codfw.wmnet * 05:18 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade == 2026-06-07 == * 16:32 elukey: `elukey@cumin1003:~$ sudo cumin 'cp6* and not cp6014* and not cp6010*' "varnish-frontend-restart" -b 1` * 16:29 elukey: restart varnish-frontend on cp6014 == 2026-06-06 == * 09:07 ammarpad@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=hewiki --logwiki=metawiki W.Mechelke Tungsten_Mechelke # [[phab:T428182|T428182]] == 2026-06-05 == * 22:16 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 21:01 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=10 --verbose` (after stopping the other commons scan) * 20:56 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=30 --verbose` (after stopping the other commons scan) * 20:20 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290093{{!}}Enable wmgUseUrlShortenerLegacy on test2wiki (T107188)]] (duration: 10m 02s) * 20:16 krinkle@deploy1003: krinkle: Continuing with deployment * 20:12 krinkle@deploy1003: krinkle: Backport for [[gerrit:1290093{{!}}Enable wmgUseUrlShortenerLegacy on test2wiki (T107188)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:10 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1290093{{!}}Enable wmgUseUrlShortenerLegacy on test2wiki (T107188)]] * 16:45 jgreen@dns1004: END - running authdns-update * 16:44 jgreen@dns1004: START - running authdns-update * 16:17 dzahn@dns1005: END - running authdns-update * 16:17 mutante: DNS - adding new project language "mag" - Magahi - a language spoken in India and Nepal by about 12 million native speakers ([[phab:T428266|T428266]]) * 16:16 dzahn@dns1005: START - running authdns-update * 14:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:38 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:37 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 12:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:30 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:30 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 12:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2202.codfw.wmnet with reason: Reboot * 12:28 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:28 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 12:08 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:07 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 12:07 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:06 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 11:29 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 11:28 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:55 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:54 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:31 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1054: repool after upgrade * 08:08 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 08:07 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 08:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 08:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/kafka-ui: apply * 07:39 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1054: repool after upgrade * 07:38 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 07:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 07:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/kafka-ui: apply * 07:07 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 06:01 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1054.eqiad.wmnet with OS trixie * 05:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1054.eqiad.wmnet with reason: host reimage * 05:37 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1054.eqiad.wmnet with reason: host reimage * 05:22 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1054.eqiad.wmnet with OS trixie * 05:21 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1054: Upgrading es1054.eqiad.wmnet * 05:21 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1054: Upgrading es1054.eqiad.wmnet * 05:20 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 01:55 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1010.eqiad.wmnet with OS trixie * 01:39 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: host reimage * 01:32 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: host reimage * 01:16 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1010.eqiad.wmnet with OS trixie * 00:56 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1007.eqiad.wmnet with OS trixie * 00:40 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1007.eqiad.wmnet with reason: host reimage * 00:33 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1007.eqiad.wmnet with reason: host reimage * 00:17 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1007.eqiad.wmnet with OS trixie * 00:02 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297268{{!}}Redirect unknown wikinews languages to portal (T427126)]] (duration: 07m 02s) == 2026-06-04 == * 23:57 ladsgroup@deploy1003: ladsgroup, pppery: Continuing with deployment * 23:57 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1006.eqiad.wmnet with OS trixie * 23:57 ladsgroup@deploy1003: ladsgroup, pppery: Backport for [[gerrit:1297268{{!}}Redirect unknown wikinews languages to portal (T427126)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:55 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1297268{{!}}Redirect unknown wikinews languages to portal (T427126)]] * 23:40 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1006.eqiad.wmnet with reason: host reimage * 23:36 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1006.eqiad.wmnet with reason: host reimage * 23:20 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1006.eqiad.wmnet with OS trixie * 21:28 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host releases1003.eqiad.wmnet with OS trixie * 21:04 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases1003.eqiad.wmnet with reason: host reimage * 20:58 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on releases1003.eqiad.wmnet with reason: host reimage * 20:50 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5030.* * 20:42 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host releases1003.eqiad.wmnet with OS trixie * 20:27 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1100.eqiad.wmnet,service=(cdn{{!}}ats-be) * 20:26 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6013.drmrs.wmnet,service=(cdn{{!}}ats-be) * 20:20 brett@dns1006: END - running authdns-update * 20:19 brett@dns1006: START - running authdns-update * 20:18 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5030.eqsin.wmnet with OS trixie * 20:10 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296015{{!}}Deploy PRV to 6 wikis (T427851)]] (duration: 07m 39s) * 20:08 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist group2.dblist extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` * 20:06 arlolra@deploy1003: arlolra: Continuing with deployment * 20:04 arlolra@deploy1003: arlolra: Backport for [[gerrit:1296015{{!}}Deploy PRV to 6 wikis (T427851)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:02 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1296015{{!}}Deploy PRV to 6 wikis (T427851)]] * 19:49 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5030.eqsin.wmnet with reason: host reimage * 19:43 cmooney@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5030.eqsin.wmnet with reason: host reimage * 19:15 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5030 * 19:15 cmooney@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5030 * 19:14 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cp5030 * 19:14 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5030.eqsin.wmnet 27.0.132.10.in-addr.arpa 7.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:14 cmooney@cumin1003: START - Cookbook sre.dns.wipe-cache cp5030.eqsin.wmnet 27.0.132.10.in-addr.arpa 7.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:14 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:14 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5030 - cmooney@cumin1003" * 19:13 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5030 - cmooney@cumin1003" * 19:09 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 19:08 cmooney@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5030 * 19:08 cmooney@cumin1003: START - Cookbook sre.hosts.reimage for host cp5030.eqsin.wmnet with OS trixie * 18:51 cmooney@dns2005: END - running authdns-update * 18:50 cmooney@dns2005: START - running authdns-update * 18:43 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:42 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove IPs that had been used for eqsin cr links - cmooney@cumin1003" * 18:40 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove IPs that had been used for eqsin cr links - cmooney@cumin1003" * 18:37 sukhe: sukhe@cp6013:~$ sudo traffic_server -C clear_cache * 18:36 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 18:08 dancy@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.5 refs [[phab:T423914|T423914]] * 17:17 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297751{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297752{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] (duration: 06m 40s) * 17:13 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 17:13 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297751{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297752{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:11 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297751{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297752{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] * 16:55 topranks: shift traffic off cr1-esams et-1/0/1 link to asw1-by27-esams [[phab:T427056|T427056]] * 16:45 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297741{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297742{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] (duration: 13m 58s) * 16:41 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 16:33 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297741{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297742{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:31 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297741{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297742{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] * 16:17 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 16:03 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297740{{!}}hCaptcha: Move ConfirmEditCaptchaClass hook inside hCaptcha block (T428183)]] (duration: 10m 21s) * 16:03 elukey: uploaded spicerack_12.7.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia * 15:59 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 15:55 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297740{{!}}hCaptcha: Move ConfirmEditCaptchaClass hook inside hCaptcha block (T428183)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:53 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297740{{!}}hCaptcha: Move ConfirmEditCaptchaClass hook inside hCaptcha block (T428183)]] * 15:44 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5030.* * 15:41 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2007.codfw.wmnet with OS trixie * 15:39 ladsgroup@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0) * 15:28 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views * 15:24 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297730{{!}}ptwiki: Disable Article Guidance experiment (T426871)]] (duration: 07m 26s) * 15:24 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2007.codfw.wmnet with reason: host reimage * 15:20 sbisson@deploy1003: sbisson: Continuing with deployment * 15:19 sbisson@deploy1003: sbisson: Backport for [[gerrit:1297730{{!}}ptwiki: Disable Article Guidance experiment (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:19 jayme@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2007.codfw.wmnet with reason: host reimage * 15:17 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1297730{{!}}ptwiki: Disable Article Guidance experiment (T426871)]] * 15:13 ladsgroup@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0) * 15:06 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297724{{!}}Revert "Start reading from new file tables on commons"]] (duration: 07m 00s) * 15:05 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views * 15:02 zabe@deploy1003: zabe: Continuing with deployment * 15:01 zabe@deploy1003: zabe: Backport for [[gerrit:1297724{{!}}Revert "Start reading from new file tables on commons"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:59 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1297724{{!}}Revert "Start reading from new file tables on commons"]] * 14:57 zabe@deploy1003: Finished scap sync-world: [[phab:T416548|T416548]] (duration: 05m 10s) * 14:56 jayme@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-main2007.codfw.wmnet with OS trixie * 14:52 zabe@deploy1003: Started scap sync-world: [[phab:T416548|T416548]] * 14:50 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:49 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:43 zabe@deploy1003: sync-world aborted: Backport for [[gerrit:1270513{{!}}Start reading from new file tables on commons (T416548)]] (duration: 03m 58s) * 14:43 zabe@deploy1003: zabe: Continuing with deployment * 14:41 zabe@deploy1003: zabe: Backport for [[gerrit:1270513{{!}}Start reading from new file tables on commons (T416548)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:40 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f1-codfw * 14:40 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device lsw1-f1-codfw * 14:39 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1270513{{!}}Start reading from new file tables on commons (T416548)]] * 14:36 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297711{{!}}hCaptcha: Enable for MobileFrontend in some Group 2 wikis (T425940)]] (duration: 08m 20s) * 14:32 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:30 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297711{{!}}hCaptcha: Enable for MobileFrontend in some Group 2 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:29 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1057: repool after upgrade * 14:28 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297711{{!}}hCaptcha: Enable for MobileFrontend in some Group 2 wikis (T425940)]] * 14:20 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 14:16 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:15 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply * 14:15 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:15 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply * 14:13 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297704{{!}}Use the globalblock-local-status right over globalblock-whitelist (T277942)]], [[gerrit:1296620{{!}}core-Permissions: Stop assigning unused globalblock-whitelist right (T277942)]] (duration: 06m 46s) * 14:10 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 14:08 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:08 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297704{{!}}Use the globalblock-local-status right over globalblock-whitelist (T277942)]], [[gerrit:1296620{{!}}core-Permissions: Stop assigning unused globalblock-whitelist right (T277942)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:07 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:06 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply * 14:06 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297704{{!}}Use the globalblock-local-status right over globalblock-whitelist (T277942)]], [[gerrit:1296620{{!}}core-Permissions: Stop assigning unused globalblock-whitelist right (T277942)]] * 14:06 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:06 tappof: bump space for prometheus k8s-aux in eqiad * 14:05 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply * 14:05 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:04 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply * 13:56 _joe_: transferred requestctl api tokens for all ops to the db ([[phab:T428119|T428119]]) * 13:56 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2050 to es3 codfw primary [[phab:T428050|T428050]]', diff saved to https://phabricator.wikimedia.org/P93878 and previous config saved to /var/cache/conftool/dbconfig/20260604-135631-marostegui.json * 13:56 Dreamy_Jazz: Afternoon UTC backport window done * 13:54 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297700{{!}}Revert "hCaptcha: Provide always challenge sitekey for account creation"]] (duration: 13m 38s) * 13:51 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 13:50 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 13:47 sukhe: sukhe@cp6011:~$ sudo -i varnish-frontend-restart * 13:44 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1057: repool after upgrade * 13:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:43 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297700{{!}}Revert "hCaptcha: Provide always challenge sitekey for account creation"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:41 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1057.eqiad.wmnet with OS trixie * 13:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297700{{!}}Revert "hCaptcha: Provide always challenge sitekey for account creation"]] * 13:38 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297692{{!}}hCaptcha: Provide always challenge sitekey for account creation (T421041)]] (duration: 05m 27s) * 13:38 dreamyjazz@deploy1003: dreamyjazz: Rolling back deployment * 13:36 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: down * 13:35 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297692{{!}}hCaptcha: Provide always challenge sitekey for account creation (T421041)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:33 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297692{{!}}hCaptcha: Provide always challenge sitekey for account creation (T421041)]] * 13:31 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295978{{!}}Update config for WikiProjects linking prototype (T427804)]] (duration: 17m 13s) * 13:26 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Continuing with deployment * 13:25 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1057.eqiad.wmnet with reason: host reimage * 13:17 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1057.eqiad.wmnet with reason: host reimage * 13:16 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Backport for [[gerrit:1295978{{!}}Update config for WikiProjects linking prototype (T427804)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:14 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1295978{{!}}Update config for WikiProjects linking prototype (T427804)]] * 13:13 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:13 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1220: Migration of db1220.eqiad.wmnet completed * 13:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: down * 13:12 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1224', diff saved to https://phabricator.wikimedia.org/P93875 and previous config saved to /var/cache/conftool/dbconfig/20260604-131219-marostegui.json * 13:00 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1057.eqiad.wmnet with OS trixie * 13:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1057: Upgrading es1057.eqiad.wmnet * 12:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1057: Upgrading es1057.eqiad.wmnet * 12:59 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:56 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296557{{!}}wmf-config: Skip CAPTCHA for action=mcrundo (T427612)]] (duration: 08m 30s) * 12:52 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Continuing with deployment * 12:50 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Backport for [[gerrit:1296557{{!}}wmf-config: Skip CAPTCHA for action=mcrundo (T427612)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2050: repool after upgrade * 12:48 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296557{{!}}wmf-config: Skip CAPTCHA for action=mcrundo (T427612)]] * 12:37 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 12:37 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 12:28 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1220: Migration of db1220.eqiad.wmnet completed * 12:20 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1220.eqiad.wmnet with OS trixie * 12:04 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2050: repool after upgrade * 12:04 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 12:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1220.eqiad.wmnet with reason: host reimage * 11:59 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1220.eqiad.wmnet with reason: host reimage * 11:42 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1220.eqiad.wmnet with OS trixie * 11:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2050.codfw.wmnet with OS trixie * 11:40 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1220: Upgrading db1220.eqiad.wmnet * 11:37 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1220: Upgrading db1220.eqiad.wmnet * 11:36 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1179: Migration of db1179.eqiad.wmnet completed * 11:23 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2050.codfw.wmnet with reason: host reimage * 11:16 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2050.codfw.wmnet with reason: host reimage * 11:00 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2050.codfw.wmnet with OS trixie * 11:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2050: Upgrading es2050.codfw.wmnet * 10:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2050: Upgrading es2050.codfw.wmnet * 10:59 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:59 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2057: repool after upgrade * 10:58 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:55 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 10:46 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1179: Migration of db1179.eqiad.wmnet completed * 10:38 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1179.eqiad.wmnet with OS trixie * 10:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1179.eqiad.wmnet with reason: host reimage * 10:16 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply * 10:15 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply * 10:15 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply * 10:15 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/kartotherian: apply * 10:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1179.eqiad.wmnet with reason: host reimage * 10:13 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2057: repool after upgrade * 10:13 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2057.codfw.wmnet with OS trixie * 09:59 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1179.eqiad.wmnet with OS trixie * 09:58 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1179: Upgrading db1179.eqiad.wmnet * 09:58 jynus: redoing m2 backups after grant change [[phab:T411111|T411111]] * 09:57 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1179: Upgrading db1179.eqiad.wmnet * 09:56 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:54 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2057.codfw.wmnet with reason: host reimage * 09:53 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 09:49 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2057.codfw.wmnet with reason: host reimage * 09:39 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:39 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Migration of db1224.eqiad.wmnet completed * 09:38 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 09:37 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 09:36 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 09:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/kafka-ui: apply * 09:33 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2057.codfw.wmnet with OS trixie * 09:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2057: Upgrading es2057.codfw.wmnet * 09:32 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2057: Upgrading es2057.codfw.wmnet * 09:31 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:26 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=30 --sleep=60 --verbose` * 09:25 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist "group0.dblist + group1.dblist - mediamoderation-continuous-scan.dblist" extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` * 08:54 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Introduce pluggable authentication - oblivian@cumin1003" * 08:54 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Introduce pluggable authentication - oblivian@cumin1003 * 08:53 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Migration of db1224.eqiad.wmnet completed * 08:53 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Introduce pluggable authentication - oblivian@cumin1003 * 08:53 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Introduce pluggable authentication - oblivian@cumin1003" * 08:29 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 08:29 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 08:24 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 08:24 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 08:21 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 08:21 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1224.eqiad.wmnet with OS trixie * 08:21 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 08:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1224.eqiad.wmnet with reason: host reimage * 08:02 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2249.codfw.wmnet with reason: upgrade * 08:00 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1224.eqiad.wmnet with reason: host reimage * 07:53 marostegui: Install mariadb 10.11.17 on db2249 [[phab:T427345|T427345]] * 07:43 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1224.eqiad.wmnet with OS trixie * 07:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1224: Upgrading db1224.eqiad.wmnet * 07:41 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1224: Upgrading db1224.eqiad.wmnet * 07:41 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:39 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:39 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1255: Migration of db1255.eqiad.wmnet completed * 07:34 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297536{{!}}hCaptcha risk scores: VE plugin to collect risk scores for block notices (T426943)]], [[gerrit:1297200{{!}}hCaptcha: Render a fresh mobile widget for each captcha attempt (T425929)]], [[gerrit:1297173{{!}}hCaptcha: Enable risk-score collection for users blocked by IP blocks (T424629)]] (duration: 08m 56s) * 07:29 kharlan@deploy1003: kharlan, harroyo-wmf: Continuing with deployment * 07:27 kharlan@deploy1003: kharlan, harroyo-wmf: Backport for [[gerrit:1297536{{!}}hCaptcha risk scores: VE plugin to collect risk scores for block notices (T426943)]], [[gerrit:1297200{{!}}hCaptcha: Render a fresh mobile widget for each captcha attempt (T425929)]], [[gerrit:1297173{{!}}hCaptcha: Enable risk-score collection for users blocked by IP blocks (T424629)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwd * 07:25 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297536{{!}}hCaptcha risk scores: VE plugin to collect risk scores for block notices (T426943)]], [[gerrit:1297200{{!}}hCaptcha: Render a fresh mobile widget for each captcha attempt (T425929)]], [[gerrit:1297173{{!}}hCaptcha: Enable risk-score collection for users blocked by IP blocks (T424629)]] * 07:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2191: Migration of db2191.codfw.wmnet completed * 07:12 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297550{{!}}Revert "EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion"]] (duration: 06m 45s) * 07:08 kharlan@deploy1003: kharlan: Continuing with deployment * 07:08 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297550{{!}}Revert "EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:06 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297550{{!}}Revert "EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion"]] * 07:04 otto@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297260{{!}}EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] (duration: 399m 30s) * 07:03 otto@deploy1003: otto: Rolling back deployment * 06:53 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1255: Migration of db1255.eqiad.wmnet completed * 06:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1255.eqiad.wmnet with OS trixie * 06:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2191: Migration of db2191.codfw.wmnet completed * 06:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1255.eqiad.wmnet with reason: host reimage * 06:32 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2191.codfw.wmnet with OS trixie * 06:31 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1255.eqiad.wmnet with reason: host reimage * 06:16 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1255.eqiad.wmnet with OS trixie * 06:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2191.codfw.wmnet with reason: host reimage * 06:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1255: Upgrading db1255.eqiad.wmnet * 06:12 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1255: Upgrading db1255.eqiad.wmnet * 06:12 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2191.codfw.wmnet with reason: host reimage * 06:04 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db1255 [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93836 and previous config saved to /var/cache/conftool/dbconfig/20260604-060428-cwilliams.json * 06:03 cwilliams@dns1004: END - running authdns-update * 06:02 cwilliams@dns1004: START - running authdns-update * 05:54 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db1258 to x3 primary and set section read-write [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93835 and previous config saved to /var/cache/conftool/dbconfig/20260604-055429-cwilliams.json * 05:53 cwilliams@cumin1003: dbctl commit (dc=all): 'Set x3 eqiad as read-only for maintenance - [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93834 and previous config saved to /var/cache/conftool/dbconfig/20260604-055346-cwilliams.json * 05:53 cezmunsta: Starting x3 eqiad failover from db1255 to db1258 - [[phab:T427895|T427895]] * 05:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2191.codfw.wmnet with OS trixie * 05:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2191: Upgrading db2191.codfw.wmnet * 05:50 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2191: Upgrading db2191.codfw.wmnet * 05:50 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db1258 with weight 0 [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93833 and previous config saved to /var/cache/conftool/dbconfig/20260604-055021-cwilliams.json * 05:50 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:50 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Primary switchover x3 [[phab:T427895|T427895]] * 05:48 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 05:46 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2191 [[phab:T428120|T428120]]', diff saved to https://phabricator.wikimedia.org/P93832 and previous config saved to /var/cache/conftool/dbconfig/20260604-054614-marostegui.json * 05:45 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db2215 to x1 primary [[phab:T428120|T428120]]', diff saved to https://phabricator.wikimedia.org/P93831 and previous config saved to /var/cache/conftool/dbconfig/20260604-054528-marostegui.json * 05:44 marostegui: Starting x1 codfw failover from db2191 to db2215 - [[phab:T428120|T428120]] * 05:27 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: Primary switchover x1 [[phab:T428120|T428120]] * 05:27 marostegui@cumin1003: dbctl commit (dc=all): 'Set db2215 with weight 0 [[phab:T428120|T428120]]', diff saved to https://phabricator.wikimedia.org/P93830 and previous config saved to /var/cache/conftool/dbconfig/20260604-052722-marostegui.json * 05:19 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 03:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93829 and previous config saved to /var/cache/conftool/dbconfig/20260604-034546-fceratto.json * 03:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P93828 and previous config saved to /var/cache/conftool/dbconfig/20260604-033538-fceratto.json * 03:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P93827 and previous config saved to /var/cache/conftool/dbconfig/20260604-032531-fceratto.json * 03:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93826 and previous config saved to /var/cache/conftool/dbconfig/20260604-031523-fceratto.json * 03:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1263 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93825 and previous config saved to /var/cache/conftool/dbconfig/20260604-030710-fceratto.json * 03:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1263.eqiad.wmnet with reason: Maintenance * 03:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93824 and previous config saved to /var/cache/conftool/dbconfig/20260604-030642-fceratto.json * 02:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P93823 and previous config saved to /var/cache/conftool/dbconfig/20260604-025634-fceratto.json * 02:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P93822 and previous config saved to /var/cache/conftool/dbconfig/20260604-024627-fceratto.json * 02:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93821 and previous config saved to /var/cache/conftool/dbconfig/20260604-023619-fceratto.json * 02:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1262 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93820 and previous config saved to /var/cache/conftool/dbconfig/20260604-022809-fceratto.json * 02:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Maintenance * 02:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93819 and previous config saved to /var/cache/conftool/dbconfig/20260604-022742-fceratto.json * 02:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P93818 and previous config saved to /var/cache/conftool/dbconfig/20260604-021734-fceratto.json * 02:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P93817 and previous config saved to /var/cache/conftool/dbconfig/20260604-020726-fceratto.json * 01:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93816 and previous config saved to /var/cache/conftool/dbconfig/20260604-015718-fceratto.json * 01:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1261 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93815 and previous config saved to /var/cache/conftool/dbconfig/20260604-014909-fceratto.json * 01:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1261.eqiad.wmnet with reason: Maintenance * 01:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93814 and previous config saved to /var/cache/conftool/dbconfig/20260604-014841-fceratto.json * 01:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P93813 and previous config saved to /var/cache/conftool/dbconfig/20260604-013833-fceratto.json * 01:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P93812 and previous config saved to /var/cache/conftool/dbconfig/20260604-012826-fceratto.json * 01:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93811 and previous config saved to /var/cache/conftool/dbconfig/20260604-011818-fceratto.json * 01:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1260 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93810 and previous config saved to /var/cache/conftool/dbconfig/20260604-011005-fceratto.json * 01:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1260.eqiad.wmnet with reason: Maintenance * 01:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93809 and previous config saved to /var/cache/conftool/dbconfig/20260604-010937-fceratto.json * 00:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P93808 and previous config saved to /var/cache/conftool/dbconfig/20260604-005929-fceratto.json * 00:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P93807 and previous config saved to /var/cache/conftool/dbconfig/20260604-004922-fceratto.json * 00:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93806 and previous config saved to /var/cache/conftool/dbconfig/20260604-003914-fceratto.json * 00:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1252 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93805 and previous config saved to /var/cache/conftool/dbconfig/20260604-002851-fceratto.json * 00:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1252.eqiad.wmnet with reason: Maintenance * 00:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93804 and previous config saved to /var/cache/conftool/dbconfig/20260604-002821-fceratto.json * 00:26 otto@deploy1003: otto: Backport for [[gerrit:1297260{{!}}EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 00:24 otto@deploy1003: Started scap sync-world: Backport for [[gerrit:1297260{{!}}EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] * 00:18 Amir1: mwscript-k8s --follow --dblist=all -- extensions/timeline/maintenance/DeleteOldTimelineFiles.php --date {{Gerrit|20210101000000}} * 00:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P93803 and previous config saved to /var/cache/conftool/dbconfig/20260604-001813-fceratto.json * 00:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P93802 and previous config saved to /var/cache/conftool/dbconfig/20260604-000805-fceratto.json == 2026-06-03 == * 23:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93801 and previous config saved to /var/cache/conftool/dbconfig/20260603-235758-fceratto.json * 23:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93800 and previous config saved to /var/cache/conftool/dbconfig/20260603-234935-fceratto.json * 23:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1248.eqiad.wmnet with reason: Maintenance * 23:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93799 and previous config saved to /var/cache/conftool/dbconfig/20260603-234907-fceratto.json * 23:42 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296561{{!}}Add a maintenance script to delete old files]], [[gerrit:1296560{{!}}Add a maintenance script to delete old files]] (duration: 07m 09s) * 23:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P93798 and previous config saved to /var/cache/conftool/dbconfig/20260603-233859-fceratto.json * 23:37 ladsgroup@deploy1003: ladsgroup, reedy: Continuing with deployment * 23:36 ladsgroup@deploy1003: ladsgroup, reedy: Backport for [[gerrit:1296561{{!}}Add a maintenance script to delete old files]], [[gerrit:1296560{{!}}Add a maintenance script to delete old files]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:34 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1296561{{!}}Add a maintenance script to delete old files]], [[gerrit:1296560{{!}}Add a maintenance script to delete old files]] * 23:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P93797 and previous config saved to /var/cache/conftool/dbconfig/20260603-232852-fceratto.json * 23:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 23:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 23:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93796 and previous config saved to /var/cache/conftool/dbconfig/20260603-231844-fceratto.json * 23:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93795 and previous config saved to /var/cache/conftool/dbconfig/20260603-231031-fceratto.json * 23:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1247.eqiad.wmnet with reason: Maintenance * 23:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93794 and previous config saved to /var/cache/conftool/dbconfig/20260603-231001-fceratto.json * 22:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P93793 and previous config saved to /var/cache/conftool/dbconfig/20260603-225953-fceratto.json * 22:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P93792 and previous config saved to /var/cache/conftool/dbconfig/20260603-224945-fceratto.json * 22:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93791 and previous config saved to /var/cache/conftool/dbconfig/20260603-223937-fceratto.json * 22:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1244 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93790 and previous config saved to /var/cache/conftool/dbconfig/20260603-223116-fceratto.json * 22:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1244.eqiad.wmnet with reason: Maintenance * 22:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93789 and previous config saved to /var/cache/conftool/dbconfig/20260603-223048-fceratto.json * 22:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P93788 and previous config saved to /var/cache/conftool/dbconfig/20260603-222041-fceratto.json * 22:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P93787 and previous config saved to /var/cache/conftool/dbconfig/20260603-221034-fceratto.json * 22:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93786 and previous config saved to /var/cache/conftool/dbconfig/20260603-220026-fceratto.json * 21:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1243 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93785 and previous config saved to /var/cache/conftool/dbconfig/20260603-215110-fceratto.json * 21:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1243.eqiad.wmnet with reason: Maintenance * 21:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93784 and previous config saved to /var/cache/conftool/dbconfig/20260603-215053-fceratto.json * 21:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P93783 and previous config saved to /var/cache/conftool/dbconfig/20260603-214046-fceratto.json * 21:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P93782 and previous config saved to /var/cache/conftool/dbconfig/20260603-213038-fceratto.json * 21:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93781 and previous config saved to /var/cache/conftool/dbconfig/20260603-212030-fceratto.json * 21:12 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1242 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93779 and previous config saved to /var/cache/conftool/dbconfig/20260603-211206-fceratto.json * 21:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1242.eqiad.wmnet with reason: Maintenance * 21:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93778 and previous config saved to /var/cache/conftool/dbconfig/20260603-211138-fceratto.json * 21:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P93774 and previous config saved to /var/cache/conftool/dbconfig/20260603-210130-fceratto.json * 20:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P93773 and previous config saved to /var/cache/conftool/dbconfig/20260603-205122-fceratto.json * 20:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93772 and previous config saved to /var/cache/conftool/dbconfig/20260603-204115-fceratto.json * 20:33 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297228{{!}}Attribution research don't use testKitchen compatibility layer (T417050)]] (duration: 06m 41s) * 20:32 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1241 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93771 and previous config saved to /var/cache/conftool/dbconfig/20260603-203254-fceratto.json * 20:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1241.eqiad.wmnet with reason: Maintenance * 20:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93770 and previous config saved to /var/cache/conftool/dbconfig/20260603-203227-fceratto.json * 20:29 cjming@deploy1003: cjming: Continuing with deployment * 20:29 cjming@deploy1003: cjming: Backport for [[gerrit:1297228{{!}}Attribution research don't use testKitchen compatibility layer (T417050)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:26 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1297228{{!}}Attribution research don't use testKitchen compatibility layer (T417050)]] * 20:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P93769 and previous config saved to /var/cache/conftool/dbconfig/20260603-202219-fceratto.json * 20:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P93766 and previous config saved to /var/cache/conftool/dbconfig/20260603-201211-fceratto.json * 20:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93765 and previous config saved to /var/cache/conftool/dbconfig/20260603-200203-fceratto.json * 19:59 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 19:59 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 19:59 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 19:59 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 19:53 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93764 and previous config saved to /var/cache/conftool/dbconfig/20260603-195341-fceratto.json * 19:53 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1238.eqiad.wmnet with reason: Maintenance * 19:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93763 and previous config saved to /var/cache/conftool/dbconfig/20260603-195313-fceratto.json * 19:47 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5032.* * 19:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P93762 and previous config saved to /var/cache/conftool/dbconfig/20260603-194306-fceratto.json * 19:39 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5032.* * 19:37 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5032.* * 19:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P93761 and previous config saved to /var/cache/conftool/dbconfig/20260603-193258-fceratto.json * 19:26 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 19:25 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 19:25 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 19:25 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 19:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93760 and previous config saved to /var/cache/conftool/dbconfig/20260603-192250-fceratto.json * 19:22 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 19:22 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 19:14 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93759 and previous config saved to /var/cache/conftool/dbconfig/20260603-191437-fceratto.json * 19:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1024-1025].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 19:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1221.eqiad.wmnet with reason: Maintenance * 19:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93758 and previous config saved to /var/cache/conftool/dbconfig/20260603-191348-fceratto.json * 19:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P93757 and previous config saved to /var/cache/conftool/dbconfig/20260603-190340-fceratto.json * 18:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P93756 and previous config saved to /var/cache/conftool/dbconfig/20260603-185331-fceratto.json * 18:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93755 and previous config saved to /var/cache/conftool/dbconfig/20260603-184324-fceratto.json * 18:34 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1199 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93754 and previous config saved to /var/cache/conftool/dbconfig/20260603-183455-fceratto.json * 18:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance * 18:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93753 and previous config saved to /var/cache/conftool/dbconfig/20260603-183427-fceratto.json * 18:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P93752 and previous config saved to /var/cache/conftool/dbconfig/20260603-182420-fceratto.json * 18:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P93751 and previous config saved to /var/cache/conftool/dbconfig/20260603-181412-fceratto.json * 18:10 dancy@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.5 refs [[phab:T423914|T423914]] * 18:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93750 and previous config saved to /var/cache/conftool/dbconfig/20260603-180404-fceratto.json * 17:57 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5032.* * 17:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93749 and previous config saved to /var/cache/conftool/dbconfig/20260603-175544-fceratto.json * 17:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance * 17:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93748 and previous config saved to /var/cache/conftool/dbconfig/20260603-175342-fceratto.json * 17:52 hashar: contint1003: sudo puppet agent --disable "Prevent Jenkins from coming back" * 17:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P93747 and previous config saved to /var/cache/conftool/dbconfig/20260603-174334-fceratto.json * 17:38 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:37 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 17:37 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:36 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:36 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:35 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:35 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:35 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:34 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:34 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:33 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P93746 and previous config saved to /var/cache/conftool/dbconfig/20260603-173327-fceratto.json * 17:33 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:32 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:29 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5032.* * 17:26 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host sretest2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 17:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93745 and previous config saved to /var/cache/conftool/dbconfig/20260603-172319-fceratto.json * 17:18 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: Stopping before sync operations * 17:17 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: Started scap sync-world: No-deploy scap run to verify scap config change * 17:17 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:15 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:15 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1253 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93744 and previous config saved to /var/cache/conftool/dbconfig/20260603-171521-fceratto.json * 17:15 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:15 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance * 17:14 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93743 and previous config saved to /var/cache/conftool/dbconfig/20260603-171452-fceratto.json * 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:13 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:12 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:10 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:09 ayounsi@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2012.wikimedia.org with OS trixie * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P93742 and previous config saved to /var/cache/conftool/dbconfig/20260603-170444-fceratto.json * 17:04 swfrench@deploy1003: Stopping before sync operations * 17:03 swfrench@deploy1003: Started scap sync-world: No-deploy scap run to verify clean state before config change * 16:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P93741 and previous config saved to /var/cache/conftool/dbconfig/20260603-165436-fceratto.json * 16:53 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:53 hashar: Restarting CI Jenkins one last time # [[phab:T418521|T418521]] * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:44 btullis@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295922{{!}}Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] (duration: 07m 16s) * 16:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93740 and previous config saved to /var/cache/conftool/dbconfig/20260603-164428-fceratto.json * 16:43 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:43 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:42 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:41 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:40 btullis@deploy1003: btullis: Continuing with deployment * 16:39 btullis@deploy1003: btullis: Backport for [[gerrit:1295922{{!}}Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93739 and previous config saved to /var/cache/conftool/dbconfig/20260603-163726-fceratto.json * 16:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance * 16:37 btullis@deploy1003: Started scap sync-world: Backport for [[gerrit:1295922{{!}}Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] * 16:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93738 and previous config saved to /var/cache/conftool/dbconfig/20260603-163658-fceratto.json * 16:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P93737 and previous config saved to /var/cache/conftool/dbconfig/20260603-162650-fceratto.json * 16:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P93736 and previous config saved to /var/cache/conftool/dbconfig/20260603-161643-fceratto.json * 16:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93735 and previous config saved to /var/cache/conftool/dbconfig/20260603-160635-fceratto.json * 16:04 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:59 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93734 and previous config saved to /var/cache/conftool/dbconfig/20260603-155928-fceratto.json * 15:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance * 15:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93733 and previous config saved to /var/cache/conftool/dbconfig/20260603-155859-fceratto.json * 15:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 15:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P93732 and previous config saved to /var/cache/conftool/dbconfig/20260603-154852-fceratto.json * 15:46 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:46 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2012.wikimedia.org with OS trixie * 15:40 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:40 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 15:40 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 15:40 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 15:39 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 15:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P93731 and previous config saved to /var/cache/conftool/dbconfig/20260603-153844-fceratto.json * 15:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93729 and previous config saved to /var/cache/conftool/dbconfig/20260603-152836-fceratto.json * 15:25 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host sretest2012 * 15:25 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2012 * 15:25 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host sretest2012 * 15:25 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2012 * 15:24 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:23 mutante: disabling jenkins on CI servers for maintenance * 15:23 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host sretest2012 * 15:23 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2012 * 15:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 15:21 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1202 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93728 and previous config saved to /var/cache/conftool/dbconfig/20260603-152129-fceratto.json * 15:21 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance * 15:21 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:21 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding sretest2012 to codfw - jhancock@cumin2002" * 15:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 15:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93727 and previous config saved to /var/cache/conftool/dbconfig/20260603-152102-fceratto.json * 15:20 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding sretest2012 to codfw - jhancock@cumin2002" * 15:18 brouberol@dns1004: END - running authdns-update * 15:18 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:16 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 15:16 brouberol@dns1004: START - running authdns-update * 15:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P93726 and previous config saved to /var/cache/conftool/dbconfig/20260603-151055-fceratto.json * 15:01 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P93725 and previous config saved to /var/cache/conftool/dbconfig/20260603-150047-fceratto.json * 14:57 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 14:52 cmooney@cumin1003: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox * 14:51 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93723 and previous config saved to /var/cache/conftool/dbconfig/20260603-145039-fceratto.json * 14:48 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297137{{!}}Revert "MultimediaViewer: enable image carousel as a beta feature on Wikipedias"]] (duration: 06m 46s) * 14:47 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 14:46 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:46 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:43 mlitn@deploy1003: mlitn: Continuing with deployment * 14:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93722 and previous config saved to /var/cache/conftool/dbconfig/20260603-144334-fceratto.json * 14:43 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:43 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance * 14:43 mlitn@deploy1003: mlitn: Backport for [[gerrit:1297137{{!}}Revert "MultimediaViewer: enable image carousel as a beta feature on Wikipedias"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93721 and previous config saved to /var/cache/conftool/dbconfig/20260603-144306-fceratto.json * 14:41 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:41 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:41 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1297137{{!}}Revert "MultimediaViewer: enable image carousel as a beta feature on Wikipedias"]] * 14:39 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:39 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:39 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:39 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:38 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:35 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 14:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 14:34 sgimeno@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297130{{!}}editor: make redesigned anon warning the default experience (T424595)]] (duration: 10m 45s) * 14:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P93719 and previous config saved to /var/cache/conftool/dbconfig/20260603-143259-fceratto.json * 14:30 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:28 sgimeno@deploy1003: sgimeno: Continuing with deployment * 14:25 sgimeno@deploy1003: sgimeno: Backport for [[gerrit:1297130{{!}}editor: make redesigned anon warning the default experience (T424595)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:24 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:24 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:23 sgimeno@deploy1003: Started scap sync-world: Backport for [[gerrit:1297130{{!}}editor: make redesigned anon warning the default experience (T424595)]] * 14:23 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P93717 and previous config saved to /var/cache/conftool/dbconfig/20260603-142251-fceratto.json * 14:22 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:22 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:21 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:21 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:21 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:20 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:20 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:20 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:20 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:19 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:19 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:16 vriley@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:16 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:16 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:13 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:12 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93716 and previous config saved to /var/cache/conftool/dbconfig/20260603-141242-fceratto.json * 14:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:11 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:11 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:10 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mc2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:10 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host mc2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:10 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:09 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:08 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:07 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:05 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296631{{!}}translate: adding separate read/write endpoints (T425377)]] (duration: 13m 06s) * 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1191 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93715 and previous config saved to /var/cache/conftool/dbconfig/20260603-140537-fceratto.json * 14:05 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance * 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93714 and previous config saved to /var/cache/conftool/dbconfig/20260603-140507-fceratto.json * 14:01 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:58 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:58 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:58 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 13:56 dcausse@deploy1003: atsuko, dcausse: Rolling back deployment * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T426633|T426633]])', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20260603-133440-fceratto.json * 13:29 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:29 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2186: Migration of db2186.codfw.wmnet completed * 13:28 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295910{{!}}hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]] (duration: 07m 36s) * 13:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1174 ([[phab:T426633|T426633]])', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20260603-132638-fceratto.json * 13:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance * 13:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93710 and previous config saved to /var/cache/conftool/dbconfig/20260603-132605-fceratto.json * 13:25 sukhe: sudo cumin 'A:lvs or A:liberica' 'disable-puppet "merging CR 1282764"' * 13:23 kharlan@deploy1003: kharlan: Continuing with deployment * 13:22 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295910{{!}}hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:20 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295910{{!}}hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]] * 13:18 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296649{{!}}hCaptcha: Roll out to all except enwiki for mobile apps. (T426048)]] (duration: 07m 46s) * 13:16 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20260603-131556-fceratto.json * 13:15 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:13 kharlan@deploy1003: dbrant, kharlan: Continuing with deployment * 13:12 kharlan@deploy1003: dbrant, kharlan: Backport for [[gerrit:1296649{{!}}hCaptcha: Roll out to all except enwiki for mobile apps. (T426048)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296649{{!}}hCaptcha: Roll out to all except enwiki for mobile apps. (T426048)]] * 13:09 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:09 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add codfw d3 and e5 public vlans - ayounsi@cumin1003" * 13:09 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add codfw d3 and e5 public vlans - ayounsi@cumin1003" * 13:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P93708 and previous config saved to /var/cache/conftool/dbconfig/20260603-130548-fceratto.json * 13:05 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 12:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93706 and previous config saved to /var/cache/conftool/dbconfig/20260603-125540-fceratto.json * 12:51 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297110{{!}}ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] (duration: 07m 44s) * 12:49 jgreen@dns1004: END - running authdns-update * 12:47 jgreen@dns1004: START - running authdns-update * 12:46 jiji@deploy1003: jiji: Continuing with deployment * 12:46 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93705 and previous config saved to /var/cache/conftool/dbconfig/20260603-124624-fceratto.json * 12:46 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance * 12:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93704 and previous config saved to /var/cache/conftool/dbconfig/20260603-124556-fceratto.json * 12:45 jiji@deploy1003: jiji: Backport for [[gerrit:1297110{{!}}ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:43 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2186: Migration of db2186.codfw.wmnet completed * 12:43 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1297110{{!}}ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] * 12:41 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1067.eqiad.wmnet with OS bullseye * 12:38 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1292364{{!}}Update hCaptcha checks to retrieve API parameters from $_REQUEST (T427105)]] (duration: 11m 15s) * 12:36 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2186.codfw.wmnet with OS trixie * 12:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P93702 and previous config saved to /var/cache/conftool/dbconfig/20260603-123548-fceratto.json * 12:34 dreamyjazz@deploy1003: somerandomdeveloper, dreamyjazz: Continuing with deployment * 12:31 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1066.eqiad.wmnet with OS bullseye * 12:29 dreamyjazz@deploy1003: somerandomdeveloper, dreamyjazz: Backport for [[gerrit:1292364{{!}}Update hCaptcha checks to retrieve API parameters from $_REQUEST (T427105)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:27 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1292364{{!}}Update hCaptcha checks to retrieve API parameters from $_REQUEST (T427105)]] * 12:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P93701 and previous config saved to /var/cache/conftool/dbconfig/20260603-122541-fceratto.json * 12:22 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1067.eqiad.wmnet with reason: host reimage * 12:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2186.codfw.wmnet with reason: host reimage * 12:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93700 and previous config saved to /var/cache/conftool/dbconfig/20260603-121533-fceratto.json * 12:13 mvernon@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ms-be1066.eqiad.wmnet with reason: host reimage * 12:13 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2186.codfw.wmnet with reason: host reimage * 12:11 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1067.eqiad.wmnet with reason: host reimage * 12:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93699 and previous config saved to /var/cache/conftool/dbconfig/20260603-120732-fceratto.json * 12:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 12:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance * 12:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93698 and previous config saved to /var/cache/conftool/dbconfig/20260603-120634-fceratto.json * 12:03 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1066.eqiad.wmnet with reason: host reimage * 11:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P93697 and previous config saved to /var/cache/conftool/dbconfig/20260603-115626-fceratto.json * 11:54 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2186.codfw.wmnet with OS trixie * 11:54 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be1067 * 11:54 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1067 * 11:52 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1067 * 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be1067.eqiad.wmnet 96.48.64.10.in-addr.arpa 6.9.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:52 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be1067.eqiad.wmnet 96.48.64.10.in-addr.arpa 6.9.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1067 - mvernon@cumin2002" * 11:52 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1067 - mvernon@cumin2002" * 11:48 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2186: Upgrading db2186.codfw.wmnet * 11:48 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2186: Upgrading db2186.codfw.wmnet * 11:48 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:47 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:46 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be1067 * 11:46 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be1067.eqiad.wmnet with OS bullseye * 11:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P93695 and previous config saved to /var/cache/conftool/dbconfig/20260603-114618-fceratto.json * 11:46 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be1066 * 11:46 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1066 * 11:45 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1066 * 11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be1066.eqiad.wmnet 117.32.64.10.in-addr.arpa 7.1.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:45 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be1066.eqiad.wmnet 117.32.64.10.in-addr.arpa 7.1.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1066 - mvernon@cumin2002" * 11:45 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1066 - mvernon@cumin2002" * 11:43 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply * 11:41 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:40 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be1066 * 11:40 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be1066.eqiad.wmnet with OS bullseye * 11:39 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be1067 * 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93693 and previous config saved to /var/cache/conftool/dbconfig/20260603-113611-fceratto.json * 11:33 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:33 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2196: Migration of db2196.codfw.wmnet completed * 11:29 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93691 and previous config saved to /var/cache/conftool/dbconfig/20260603-112909-fceratto.json * 11:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance * 11:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance * 11:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93690 and previous config saved to /var/cache/conftool/dbconfig/20260603-112838-fceratto.json * 11:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:20 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:20 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:20 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P93689 and previous config saved to /var/cache/conftool/dbconfig/20260603-111831-fceratto.json * 11:14 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:09 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply * 11:09 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply * 11:08 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P93687 and previous config saved to /var/cache/conftool/dbconfig/20260603-110823-fceratto.json * 11:07 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be1066 * 11:07 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply * 11:06 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply * 11:05 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply * 11:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:01 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:01 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:00 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289895{{!}}Update UserInfoCard to be enabled by default for certain user groups (T426021)]] (duration: 07m 37s) * 11:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:59 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply * 10:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:59 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply * 10:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:58 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply * 10:58 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93685 and previous config saved to /var/cache/conftool/dbconfig/20260603-105815-fceratto.json * 10:58 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:56 mszwarc@deploy1003: mszwarc: Continuing with deployment * 10:55 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1289895{{!}}Update UserInfoCard to be enabled by default for certain user groups (T426021)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:54 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply * 10:54 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply * 10:53 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply * 10:53 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1289895{{!}}Update UserInfoCard to be enabled by default for certain user groups (T426021)]] * 10:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:50 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1198 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93684 and previous config saved to /var/cache/conftool/dbconfig/20260603-105006-fceratto.json * 10:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance * 10:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93683 and previous config saved to /var/cache/conftool/dbconfig/20260603-104939-fceratto.json * 10:45 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:45 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:44 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2196: Migration of db2196.codfw.wmnet completed * 10:44 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:41 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 10:40 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 10:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P93681 and previous config saved to /var/cache/conftool/dbconfig/20260603-103931-fceratto.json * 10:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1053: repool after upgrade * 10:37 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2196.codfw.wmnet with OS trixie * 10:36 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297090{{!}}hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] (duration: 12m 03s) * 10:32 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P93679 and previous config saved to /var/cache/conftool/dbconfig/20260603-102924-fceratto.json * 10:26 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297090{{!}}hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:24 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297090{{!}}hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] * 10:22 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be1067 * 10:21 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be1066 * 10:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2196.codfw.wmnet with reason: host reimage * 10:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93677 and previous config saved to /var/cache/conftool/dbconfig/20260603-101916-fceratto.json * 10:15 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2013.codfw.wmnet * 10:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2196.codfw.wmnet with reason: host reimage * 10:11 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93676 and previous config saved to /var/cache/conftool/dbconfig/20260603-101105-fceratto.json * 10:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1189.eqiad.wmnet with reason: Maintenance * 10:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93675 and previous config saved to /var/cache/conftool/dbconfig/20260603-101037-fceratto.json * 10:10 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2013.codfw.wmnet * 10:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P93673 and previous config saved to /var/cache/conftool/dbconfig/20260603-100029-fceratto.json * 09:59 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2196.codfw.wmnet with OS trixie * 09:57 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2196: Upgrading db2196.codfw.wmnet * 09:57 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2196: Upgrading db2196.codfw.wmnet * 09:57 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:52 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1053: repool after upgrade * 09:52 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:52 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:52 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:52 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:51 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:51 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:51 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P93670 and previous config saved to /var/cache/conftool/dbconfig/20260603-095022-fceratto.json * 09:49 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:49 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:48 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1053.eqiad.wmnet with OS trixie * 09:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:43 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2013.codfw.wmnet * 09:41 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on es1053.eqiad.wmnet with reason: host reimage * 09:41 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1053.eqiad.wmnet with reason: host reimage * 09:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93669 and previous config saved to /var/cache/conftool/dbconfig/20260603-094014-fceratto.json * 09:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2215: Migration of db2215.codfw.wmnet completed * 09:38 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2013.codfw.wmnet * 09:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93667 and previous config saved to /var/cache/conftool/dbconfig/20260603-093146-fceratto.json * 09:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance * 09:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93666 and previous config saved to /var/cache/conftool/dbconfig/20260603-093119-fceratto.json * 09:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1211: Migration of db1211.eqiad.wmnet completed * 09:27 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297069{{!}}hCaptcha: Collect risk score for blocked account creations (T427784)]] (duration: 07m 26s) * 09:25 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1053.eqiad.wmnet with OS trixie * 09:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add public1-b3-codfw gateway IPs - ayounsi@cumin1003" * 09:24 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add public1-b3-codfw gateway IPs - ayounsi@cumin1003" * 09:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1053: Upgrading es1053.eqiad.wmnet * 09:23 kharlan@deploy1003: kharlan: Continuing with deployment * 09:22 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1053: Upgrading es1053.eqiad.wmnet * 09:22 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:21 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297069{{!}}hCaptcha: Collect risk score for blocked account creations (T427784)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:21 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply * 09:21 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2054: repool after upgrade * 09:21 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply * 09:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P93661 and previous config saved to /var/cache/conftool/dbconfig/20260603-092111-fceratto.json * 09:20 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 09:20 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297069{{!}}hCaptcha: Collect risk score for blocked account creations (T427784)]] * 09:14 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297065{{!}}Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] (duration: 07m 06s) * 09:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P93659 and previous config saved to /var/cache/conftool/dbconfig/20260603-091104-fceratto.json * 09:10 kharlan@deploy1003: kharlan: Continuing with deployment * 09:09 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297065{{!}}Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:07 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297065{{!}}Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] * 09:06 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 09:06 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297064{{!}}Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 10m 54s) * 09:05 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 09:04 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 09:01 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003 - [[phab:T422043|T422043]]" * 09:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93656 and previous config saved to /var/cache/conftool/dbconfig/20260603-090056-fceratto.json * 09:00 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003 - [[phab:T422043|T422043]]" * 09:00 ayounsi@cumin1003: END (ERROR) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=97) generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003" * 09:00 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003" * 08:59 kharlan@deploy1003: kharlan: Continuing with deployment * 08:59 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297064{{!}}Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:55 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297064{{!}}Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] * 08:53 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296635{{!}}Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 11m 43s) * 08:52 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2215: Migration of db2215.codfw.wmnet completed * 08:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet * 08:52 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet * 08:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb[1022-1023].eqiad.wmnet * 08:51 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb[1022-1023].eqiad.wmnet * 08:50 kharlan@deploy1003: kharlan: Rolling back deployment * 08:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93652 and previous config saved to /var/cache/conftool/dbconfig/20260603-084846-fceratto.json * 08:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance * 08:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93651 and previous config saved to /var/cache/conftool/dbconfig/20260603-084819-fceratto.json * 08:47 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296635{{!}}Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2215.codfw.wmnet with OS trixie * 08:45 jiji@cumin1003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check docker-registry: maintenance * 08:45 jiji@cumin1003: START - Cookbook sre.discovery.service-route check docker-registry: maintenance * 08:43 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1211: Migration of db1211.eqiad.wmnet completed * 08:41 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296635{{!}}Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] * 08:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1211.eqiad.wmnet with OS trixie * 08:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93649 and previous config saved to /var/cache/conftool/dbconfig/20260603-083811-fceratto.json * 08:37 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296632{{!}}Image Browsing: add accessible labels to carousel elements (T407793)]] (duration: 32m 11s) * 08:36 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2054: repool after upgrade * 08:35 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool es2054.codfw.wmnet: After reimage * 08:35 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2054.codfw.wmnet: After reimage * 08:35 jiji@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:34 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 08:34 jiji@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:33 jiji@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:33 jiji@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:31 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:31 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:31 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2054.codfw.wmnet with OS trixie * 08:30 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:29 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 08:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2215.codfw.wmnet with reason: host reimage * 08:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93647 and previous config saved to /var/cache/conftool/dbconfig/20260603-082804-fceratto.json * 08:25 mszwarc@deploy1003: mlitn, mszwarc: Continuing with deployment * 08:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1211.eqiad.wmnet with reason: host reimage * 08:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1049: repool after upgrade * 08:22 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2215.codfw.wmnet with reason: host reimage * 08:22 mszwarc@deploy1003: mlitn, mszwarc: Backport for [[gerrit:1296632{{!}}Image Browsing: add accessible labels to carousel elements (T407793)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:18 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1211.eqiad.wmnet with reason: host reimage * 08:18 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 08:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93645 and previous config saved to /var/cache/conftool/dbconfig/20260603-081756-fceratto.json * 08:17 jiji@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 08:17 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 08:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 08:14 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2054.codfw.wmnet with reason: host reimage * 08:08 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2054.codfw.wmnet with reason: host reimage * 08:05 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296632{{!}}Image Browsing: add accessible labels to carousel elements (T407793)]] * {{safesubst:SAL entry|1=08:04 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296580{{!}}Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703{{!}}jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713{{!}}conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627{{!}}Add missing lazy img to carousel (T427821)]], [[gerrit:1295968{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T426799)]}} * 08:03 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93643 and previous config saved to /var/cache/conftool/dbconfig/20260603-080346-fceratto.json * 08:03 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1211.eqiad.wmnet with OS trixie * 08:03 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance * 08:03 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2215.codfw.wmnet with OS trixie * 08:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1211: Upgrading db1211.eqiad.wmnet * 08:02 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2215: Upgrading db2215.codfw.wmnet * 08:01 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:01 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1211: Upgrading db1211.eqiad.wmnet * 08:01 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2215: Upgrading db2215.codfw.wmnet * 08:01 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:01 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:01 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1157: Repooling * 08:01 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1157: Repooling * 08:00 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 07:57 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1022-1023].eqiad.wmnet with reason: Reimaging upstream server * 07:57 mszwarc@deploy1003: anzx, mlitn, mfossati, mszwarc: Continuing with deployment * 07:56 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Reimaging upstream server * {{safesubst:SAL entry|1=07:54 mszwarc@deploy1003: anzx, mlitn, mfossati, mszwarc: Backport for [[gerrit:1296580{{!}}Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703{{!}}jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713{{!}}conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627{{!}}Add missing lazy img to carousel (T427821)]], [[gerrit:1295968{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T42}} * 07:52 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2231: repool after maintenance * 07:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2054.codfw.wmnet with OS trixie * 07:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2054: Upgrading es2054.codfw.wmnet * 07:50 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2054: Upgrading es2054.codfw.wmnet * 07:50 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:50 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296580{{!}}Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703{{!}}jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713{{!}}conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627{{!}}Add missing lazy img to carousel (T427821)]], [[gerrit:1295968{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T426799)]] * 07:48 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296516{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] (duration: 32m 13s) * 07:44 marostegui@dns1004: END - running authdns-update * 07:43 marostegui@dns1004: START - running authdns-update * 07:42 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1056 to es2 eqiad primary [[phab:T427875|T427875]]', diff saved to https://phabricator.wikimedia.org/P93637 and previous config saved to /var/cache/conftool/dbconfig/20260603-074250-marostegui.json * 07:37 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1049: repool after upgrade * 07:37 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:35 mszwarc@deploy1003: mszwarc, stran: Continuing with deployment * 07:35 mszwarc@deploy1003: mszwarc, stran: Backport for [[gerrit:1296516{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:32 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1049.eqiad.wmnet with OS trixie * 07:16 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296516{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] * 07:14 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1049.eqiad.wmnet with reason: host reimage * 07:07 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1049.eqiad.wmnet with reason: host reimage * 07:07 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2231: repool after maintenance * 07:04 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 06:57 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2231.codfw.wmnet with OS trixie * 06:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1049.eqiad.wmnet with OS trixie * 06:46 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1049: Upgrading es1049.eqiad.wmnet * 06:46 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2056 to es2 codfw primary [[phab:T427875|T427875]]', diff saved to https://phabricator.wikimedia.org/P93632 and previous config saved to /var/cache/conftool/dbconfig/20260603-064623-marostegui.json * 06:45 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1049: Upgrading es1049.eqiad.wmnet * 06:45 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:44 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1056: repool after upgrade * 06:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2231.codfw.wmnet with reason: host reimage * 06:36 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2231.codfw.wmnet with reason: host reimage * 06:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2231.codfw.wmnet with OS trixie * 06:09 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2231: Upgrading db2231.codfw.wmnet * 06:09 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2231: Upgrading db2231.codfw.wmnet * 06:09 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:59 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1056: repool after upgrade * 05:59 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 05:55 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1056.eqiad.wmnet with OS trixie * 05:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1056.eqiad.wmnet with reason: host reimage * 05:33 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1056.eqiad.wmnet with reason: host reimage * 05:18 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1056.eqiad.wmnet with OS trixie * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1056: Upgrading es1056.eqiad.wmnet * 05:17 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1056: Upgrading es1056.eqiad.wmnet * 05:16 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade == 2026-06-02 == * 22:21 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296689{{!}}hCaptcha: Correct inaccurate comment]] (duration: 06m 27s) * 22:18 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 22:18 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 22:17 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 22:17 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296689{{!}}hCaptcha: Correct inaccurate comment]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:15 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296689{{!}}hCaptcha: Correct inaccurate comment]] * 22:13 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296551{{!}}hCaptcha: Enable for badlogin on group0 wikis (T426875)]] (duration: 08m 31s) * 22:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 22:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 22:09 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 22:07 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296551{{!}}hCaptcha: Enable for badlogin on group0 wikis (T426875)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:05 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296551{{!}}hCaptcha: Enable for badlogin on group0 wikis (T426875)]] * 20:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93621 and previous config saved to /var/cache/conftool/dbconfig/20260602-203945-fceratto.json * 20:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93620 and previous config saved to /var/cache/conftool/dbconfig/20260602-202937-fceratto.json * 20:27 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1054.eqiad.wmnet * 20:27 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:27 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1054.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:26 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1054.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:20 jiji@cumin1003: START - Cookbook sre.dns.netbox * 20:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93619 and previous config saved to /var/cache/conftool/dbconfig/20260602-201929-fceratto.json * 20:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93618 and previous config saved to /var/cache/conftool/dbconfig/20260602-200922-fceratto.json * 20:03 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1054.eqiad.wmnet * 19:48 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1053.eqiad.wmnet * 19:48 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:48 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1053.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 19:37 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1053.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 19:09 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93617 and previous config saved to /var/cache/conftool/dbconfig/20260602-190907-fceratto.json * 19:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance * 19:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93616 and previous config saved to /var/cache/conftool/dbconfig/20260602-190811-fceratto.json * 19:05 dancy@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.5 refs [[phab:T423914|T423914]] * 18:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P93615 and previous config saved to /var/cache/conftool/dbconfig/20260602-185804-fceratto.json * 18:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P93614 and previous config saved to /var/cache/conftool/dbconfig/20260602-184757-fceratto.json * 18:38 jiji@cumin1003: START - Cookbook sre.dns.netbox * 18:38 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:38 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93612 and previous config saved to /var/cache/conftool/dbconfig/20260602-183749-fceratto.json * 18:37 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:37 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:33 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1053.eqiad.wmnet * 18:30 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1259 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93611 and previous config saved to /var/cache/conftool/dbconfig/20260602-183023-fceratto.json * 18:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance * 18:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93610 and previous config saved to /var/cache/conftool/dbconfig/20260602-182956-fceratto.json * 18:27 mutante: gerrit delete unused plugin projects: barricade, WikimediaBlocks and WikimediaWebSessions * 18:26 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1052.eqiad.wmnet * 18:26 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:26 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1052.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 18:25 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1052.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 18:25 dancy: Train is blocked at testwikis on https://phabricator.wikimedia.org/T427935 * 18:21 Daimona: Running query from [[phab:T427962|T427962]]#11978299 in x1.wikishared * 18:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P93609 and previous config saved to /var/cache/conftool/dbconfig/20260602-181949-fceratto.json * 18:16 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296615{{!}}feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614{{!}}feat(cleanMentorList): Add a feature flag (T427386)]] (duration: 34m 09s) * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 18:12 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 18:12 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 18:12 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 18:10 jiji@cumin1003: START - Cookbook sre.dns.netbox * 18:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P93608 and previous config saved to /var/cache/conftool/dbconfig/20260602-180941-fceratto.json * 18:08 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 18:07 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 18:06 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 18:06 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 18:05 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:05 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:05 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 18:05 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 18:04 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 18:02 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 18:02 swfrench-wmf: reverting shellbox to 2026-05-20-192555 due to errors in shellbox-syntaxhighlight * 18:02 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 18:01 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 18:01 urbanecm@deploy1003: urbanecm: Continuing with deployment * 18:01 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1296615{{!}}feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614{{!}}feat(cleanMentorList): Add a feature flag (T427386)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:00 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1052.eqiad.wmnet * 17:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93607 and previous config saved to /var/cache/conftool/dbconfig/20260602-175933-fceratto.json * 17:58 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:57 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:56 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1051.eqiad.wmnet * 17:56 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:56 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1051.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:55 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1051.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:53 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93605 and previous config saved to /var/cache/conftool/dbconfig/20260602-175227-fceratto.json * 17:52 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance * 17:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93604 and previous config saved to /var/cache/conftool/dbconfig/20260602-175157-fceratto.json * 17:51 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:51 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:50 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:50 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:50 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:49 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:49 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:48 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:48 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:47 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:44 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:42 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:42 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:42 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P93603 and previous config saved to /var/cache/conftool/dbconfig/20260602-174150-fceratto.json * 17:41 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1296615{{!}}feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614{{!}}feat(cleanMentorList): Add a feature flag (T427386)]] * 17:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P93602 and previous config saved to /var/cache/conftool/dbconfig/20260602-173143-fceratto.json * 17:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93601 and previous config saved to /var/cache/conftool/dbconfig/20260602-172135-fceratto.json * 17:14 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93600 and previous config saved to /var/cache/conftool/dbconfig/20260602-171422-fceratto.json * 17:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance * 17:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93599 and previous config saved to /var/cache/conftool/dbconfig/20260602-171354-fceratto.json * 17:04 jiji@cumin1003: START - Cookbook sre.dns.netbox * 17:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P93598 and previous config saved to /var/cache/conftool/dbconfig/20260602-170344-fceratto.json * 16:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P93597 and previous config saved to /var/cache/conftool/dbconfig/20260602-165336-fceratto.json * 16:49 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1051.eqiad.wmnet * 16:48 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1050.eqiad.wmnet * 16:48 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:48 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1050.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:47 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1050.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93596 and previous config saved to /var/cache/conftool/dbconfig/20260602-164328-fceratto.json * 16:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93595 and previous config saved to /var/cache/conftool/dbconfig/20260602-163622-fceratto.json * 16:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance * 16:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93594 and previous config saved to /var/cache/conftool/dbconfig/20260602-163550-fceratto.json * 16:34 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:34 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1072.eqiad.wmnet with OS trixie * 16:30 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:29 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:27 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2006.codfw.wmnet with OS trixie * 16:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P93593 and previous config saved to /var/cache/conftool/dbconfig/20260602-162542-fceratto.json * 16:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P93591 and previous config saved to /var/cache/conftool/dbconfig/20260602-161534-fceratto.json * 16:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1072.eqiad.wmnet with reason: host reimage * 16:10 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1071.eqiad.wmnet with OS trixie * 16:10 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296624{{!}}Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 06m 40s) * 16:09 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2006.codfw.wmnet with reason: host reimage * 16:05 kharlan@deploy1003: kharlan: Continuing with deployment * 16:05 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1072.eqiad.wmnet with reason: host reimage * 16:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93590 and previous config saved to /var/cache/conftool/dbconfig/20260602-160527-fceratto.json * 16:05 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2006.codfw.wmnet with reason: host reimage * 16:05 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296624{{!}}Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:03 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296624{{!}}Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] * 15:59 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295909{{!}}hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] (duration: 09m 48s) * 15:59 kharlan@deploy1003: kharlan: Rolling back deployment * 15:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93589 and previous config saved to /var/cache/conftool/dbconfig/20260602-155817-fceratto.json * 15:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance * 15:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93588 and previous config saved to /var/cache/conftool/dbconfig/20260602-155749-fceratto.json * 15:54 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1071.eqiad.wmnet with reason: host reimage * 15:53 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1072.eqiad.wmnet with OS trixie * 15:51 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1070.eqiad.wmnet with OS trixie * 15:51 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295909{{!}}hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:50 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1071.eqiad.wmnet with reason: host reimage * 15:49 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295909{{!}}hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] * 15:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P93587 and previous config saved to /var/cache/conftool/dbconfig/20260602-154742-fceratto.json * 15:47 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296558{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] (duration: 07m 24s) * 15:43 kharlan@deploy1003: kharlan: Continuing with deployment * 15:42 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296558{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:40 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296558{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] * 15:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P93586 and previous config saved to /var/cache/conftool/dbconfig/20260602-153734-fceratto.json * 15:37 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1071.eqiad.wmnet with OS trixie * 15:36 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1069.eqiad.wmnet with OS trixie * 15:35 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1070.eqiad.wmnet with reason: host reimage * 15:32 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:32 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:31 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1070.eqiad.wmnet with reason: host reimage * 15:30 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:29 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93585 and previous config saved to /var/cache/conftool/dbconfig/20260602-152726-fceratto.json * 15:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2158: Repooling * {{safesubst:SAL entry|1=15:22 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295502{{!}}Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106{{!}}Remove unused 'writeapi' right]], [[gerrit:1296566{{!}}Clean up bot password configuration]], [[gerrit:1296563{{!}}Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574{{!}}cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582{{!}}U}} * 15:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1069.eqiad.wmnet with reason: host reimage * 15:20 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93583 and previous config saved to /var/cache/conftool/dbconfig/20260602-152026-fceratto.json * 15:20 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance * 15:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93582 and previous config saved to /var/cache/conftool/dbconfig/20260602-151958-fceratto.json * 15:19 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:19 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:18 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1070.eqiad.wmnet with OS trixie * 15:18 dreamyjazz@deploy1003: matmarex, anzx, dreamyjazz: Continuing with deployment * 15:18 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 15:17 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:17 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:15 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1069.eqiad.wmnet with reason: host reimage * {{safesubst:SAL entry|1=15:15 dreamyjazz@deploy1003: matmarex, anzx, dreamyjazz: Backport for [[gerrit:1295502{{!}}Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106{{!}}Remove unused 'writeapi' right]], [[gerrit:1296566{{!}}Clean up bot password configuration]], [[gerrit:1296563{{!}}Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574{{!}}cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582}} * 15:14 jiji@cumin1003: START - Cookbook sre.dns.netbox * {{safesubst:SAL entry|1=15:13 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1295502{{!}}Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106{{!}}Remove unused 'writeapi' right]], [[gerrit:1296566{{!}}Clean up bot password configuration]], [[gerrit:1296563{{!}}Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574{{!}}cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582{{!}}Us}} * 15:12 jayme@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-main2006.codfw.wmnet with OS trixie * 15:12 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1068.eqiad.wmnet with OS trixie * 15:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P93580 and previous config saved to /var/cache/conftool/dbconfig/20260602-150951-fceratto.json * 15:09 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296514{{!}}[Growth] Set wgGEMentorshipCleanupEnabled to false on all wikis (T427386)]] (duration: 06m 22s) * 15:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1167: Repooling after Icing wait-for-green timeout * 15:06 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1050.eqiad.wmnet * 15:06 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1049.eqiad.wmnet * 15:06 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:06 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1049.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:05 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1049.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:02 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1296514{{!}}[Growth] Set wgGEMentorshipCleanupEnabled to false on all wikis (T427386)]] * 15:02 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1069.eqiad.wmnet with OS trixie * 15:01 jiji@cumin1003: START - Cookbook sre.dns.netbox * 14:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P93578 and previous config saved to /var/cache/conftool/dbconfig/20260602-145943-fceratto.json * 14:54 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage * 14:52 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:52 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:52 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1049.eqiad.wmnet * 14:51 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1067.eqiad.wmnet with OS trixie * 14:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:50 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage * 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93575 and previous config saved to /var/cache/conftool/dbconfig/20260602-144935-fceratto.json * 14:42 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for pc2021.codfw.wmnet * 14:42 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for pc2021.codfw.wmnet * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2250.codfw.wmnet * 14:41 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2250.codfw.wmnet * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2158.codfw.wmnet * 14:41 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2158.codfw.wmnet * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc2021: Repooling * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 14:41 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache * 14:41 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc2021: Repooling * 14:41 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93573 and previous config saved to /var/cache/conftool/dbconfig/20260602-144110-fceratto.json * 14:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance * 14:41 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2158: Repooling * 14:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93571 and previous config saved to /var/cache/conftool/dbconfig/20260602-144043-fceratto.json * 14:38 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:38 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:38 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:37 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1048.eqiad.wmnet * 14:37 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:37 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1048.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 14:37 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1068.eqiad.wmnet with OS trixie * 14:36 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1066.eqiad.wmnet with OS trixie * 14:34 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage * 14:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P93569 and previous config saved to /var/cache/conftool/dbconfig/20260602-143035-fceratto.json * 14:30 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage * 14:25 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1048.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 14:21 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1167: Repooling after Icing wait-for-green timeout * 14:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage * 14:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P93566 and previous config saved to /var/cache/conftool/dbconfig/20260602-142027-fceratto.json * 14:17 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1067.eqiad.wmnet with OS trixie * 14:17 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 14:17 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1167.eqiad.wmnet * 14:17 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1167.eqiad.wmnet * 14:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1065.eqiad.wmnet with OS trixie * 14:15 jayme@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2006.codfw.wmnet with OS trixie * 14:14 jiji@cumin1003: START - Cookbook sre.dns.netbox * 14:13 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage * 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93564 and previous config saved to /var/cache/conftool/dbconfig/20260602-141019-fceratto.json * 14:09 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments userOptions.php --delete --nowarn growthexperiments-homepage-variant # [[phab:T417621|T417621]] * 14:09 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1048.eqiad.wmnet * 14:08 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments userOptions.php --delete growthexperiments-homepage-variant # [[phab:T417621|T417621]] * 14:05 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 14:01 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93563 and previous config saved to /var/cache/conftool/dbconfig/20260602-140140-fceratto.json * 14:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 14:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1156.eqiad.wmnet with reason: Maintenance * 14:01 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1066.eqiad.wmnet with OS trixie * 14:00 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage * 14:00 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 14:00 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 14:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93562 and previous config saved to /var/cache/conftool/dbconfig/20260602-140022-fceratto.json * 14:00 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1064.eqiad.wmnet with OS trixie * 13:56 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage * 13:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1167.eqiad.wmnet with OS trixie * 13:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P93561 and previous config saved to /var/cache/conftool/dbconfig/20260602-135015-fceratto.json * 13:47 topranks: revert all config to normal on cr1-codfw and ssw1-a1-codfw * 13:43 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1065.eqiad.wmnet with OS trixie * 13:42 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1064.eqiad.wmnet with reason: host reimage * 13:40 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1063.eqiad.wmnet with OS trixie * 13:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P93560 and previous config saved to /var/cache/conftool/dbconfig/20260602-134007-fceratto.json * 13:38 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1167.eqiad.wmnet with reason: host reimage * 13:35 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs1002.eqiad.wmnet with OS trixie * 13:35 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs1003.eqiad.wmnet with OS trixie * 13:34 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:34 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:32 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1064.eqiad.wmnet with reason: host reimage * 13:31 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1167.eqiad.wmnet with reason: host reimage * 13:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93559 and previous config saved to /var/cache/conftool/dbconfig/20260602-132959-fceratto.json * 13:27 slyngshede@dns1004: END - running authdns-update * 13:25 slyngshede@dns1004: START - running authdns-update * 13:24 topranks: increase OSPF cost on ssw1-a1-codfw et-0/0/4 towards lsw1-a5-codfw [[phab:T427301|T427301]] * 13:23 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1063.eqiad.wmnet with reason: host reimage * 13:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93558 and previous config saved to /var/cache/conftool/dbconfig/20260602-132314-fceratto.json * 13:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1210.eqiad.wmnet with reason: Maintenance * 13:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93557 and previous config saved to /var/cache/conftool/dbconfig/20260602-132246-fceratto.json * 13:20 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1064.eqiad.wmnet with OS trixie * 13:19 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 13:19 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1062.eqiad.wmnet with OS trixie * 13:18 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1063.eqiad.wmnet with reason: host reimage * 13:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2049: repool after upgrade * 13:17 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 13:16 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1167.eqiad.wmnet with OS trixie * 13:15 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 13:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1167: Upgrading db1167.eqiad.wmnet * 13:13 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1167: Upgrading db1167.eqiad.wmnet * 13:13 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:12 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 13:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P93554 and previous config saved to /var/cache/conftool/dbconfig/20260602-131238-fceratto.json * 13:12 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 13:12 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 13:11 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 13:07 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1003.eqiad.wmnet with OS trixie * 13:07 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1002.eqiad.wmnet with OS trixie * 13:06 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1063.eqiad.wmnet with OS trixie * 13:04 jayme@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-main2006.codfw.wmnet with OS trixie * 13:04 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:04 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:03 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1022-1023].eqiad.wmnet with reason: Reimaging upstream servers * 13:03 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1001.eqiad.wmnet with OS trixie * 13:03 topranks: increase OSPF cost on ssw1-a1-codfw et-0/0/2 towards lsw1-a3-codfw [[phab:T427301|T427301]] * 13:03 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1062.eqiad.wmnet with reason: host reimage * 13:02 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Reimaging upstream servers * 13:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P93553 and previous config saved to /var/cache/conftool/dbconfig/20260602-130230-fceratto.json * 12:59 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1062.eqiad.wmnet with reason: host reimage * 12:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:57 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:57 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2161: Migration of db2161.codfw.wmnet completed * 12:54 topranks: shutdown sub-interfaces on cr1-codfw et-1/1/5 for row A/B vlans [[phab:T427301|T427301]] * 12:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 12:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93550 and previous config saved to /var/cache/conftool/dbconfig/20260602-125223-fceratto.json * 12:50 topranks: enable bgp graceful-shutdown in overlay on ssw1-a1-codfw [[phab:T427301|T427301]] * 12:49 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc1061.eqiad.wmnet with OS trixie * 12:48 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt * 12:48 ayounsi@cumin1003: START - Cookbook sre.hosts.remove-downtime for lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt * 12:47 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1062.eqiad.wmnet with OS trixie * 12:45 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93548 and previous config saved to /var/cache/conftool/dbconfig/20260602-124541-fceratto.json * 12:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance * 12:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93547 and previous config saved to /var/cache/conftool/dbconfig/20260602-124512-fceratto.json * 12:43 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc1060.eqiad.wmnet with OS trixie * 12:42 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 12:42 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mc1061.eqiad.wmnet with reason: host reimage * 12:42 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1061.eqiad.wmnet with reason: host reimage * 12:41 topranks: enable bgp graceful-shutdown in underlay on ssw1-a1-codfw [[phab:T427301|T427301]] * 12:35 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mc1060.eqiad.wmnet with reason: host reimage * 12:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P93545 and previous config saved to /var/cache/conftool/dbconfig/20260602-123505-fceratto.json * 12:33 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 12:33 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1060.eqiad.wmnet with reason: host reimage * 12:31 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2049: repool after upgrade * 12:31 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 12:29 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1061.eqiad.wmnet with OS trixie * 12:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2049.codfw.wmnet with OS trixie * 12:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P93542 and previous config saved to /var/cache/conftool/dbconfig/20260602-122459-fceratto.json * 12:24 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1059.eqiad.wmnet with OS trixie * 12:21 XioNoX: reboot lsw1-a3-codfw for software upgrade - [[phab:T427301|T427301]] * 12:20 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1060.eqiad.wmnet with OS trixie * 12:20 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 12:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1058.eqiad.wmnet with OS trixie * 12:17 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 12:16 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296532{{!}}hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533{{!}}hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] (duration: 09m 02s) * 12:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93539 and previous config saved to /var/cache/conftool/dbconfig/20260602-121451-fceratto.json * 12:11 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 12:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2049.codfw.wmnet with reason: host reimage * 12:11 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt with reason: Switch maintenance * 12:10 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2161: Migration of db2161.codfw.wmnet completed * 12:09 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Switch maintenance * 12:09 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296532{{!}}hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533{{!}}hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:08 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1200 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93537 and previous config saved to /var/cache/conftool/dbconfig/20260602-120755-fceratto.json * 12:07 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1059.eqiad.wmnet with reason: host reimage * 12:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance * 12:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93536 and previous config saved to /var/cache/conftool/dbconfig/20260602-120728-fceratto.json * 12:07 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 12:07 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296532{{!}}hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533{{!}}hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] * 12:05 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2049.codfw.wmnet with reason: host reimage * 12:04 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1058.eqiad.wmnet with reason: host reimage * 12:02 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1059.eqiad.wmnet with reason: host reimage * 12:01 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2161.codfw.wmnet with OS trixie * 12:00 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1058.eqiad.wmnet with reason: host reimage * 11:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P93535 and previous config saved to /var/cache/conftool/dbconfig/20260602-115721-fceratto.json * 11:55 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 11:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:55 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 11:53 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 11:53 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 11:53 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:50 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1059.eqiad.wmnet with OS trixie * 11:49 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1057.eqiad.wmnet with OS trixie * 11:49 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2049.codfw.wmnet with OS trixie * 11:48 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2049: Upgrading es2049.codfw.wmnet * 11:48 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2049: Upgrading es2049.codfw.wmnet * 11:47 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:47 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1058.eqiad.wmnet with OS trixie * 11:47 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2056: repool after upgrade * 11:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P93532 and previous config saved to /var/cache/conftool/dbconfig/20260602-114713-fceratto.json * 11:45 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1056.eqiad.wmnet with OS trixie * 11:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2161.codfw.wmnet with reason: host reimage * 11:40 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2161.codfw.wmnet with reason: host reimage * 11:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93531 and previous config saved to /var/cache/conftool/dbconfig/20260602-113705-fceratto.json * 11:33 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1057.eqiad.wmnet with reason: host reimage * 11:30 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1185 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93529 and previous config saved to /var/cache/conftool/dbconfig/20260602-113019-fceratto.json * 11:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance * 11:29 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1056.eqiad.wmnet with reason: host reimage * 11:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1161: Repooling * 11:26 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1161: Repooling * 11:23 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2161.codfw.wmnet with OS trixie * 11:22 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1057.eqiad.wmnet with reason: host reimage * 11:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2161: Upgrading db2161.codfw.wmnet * 11:21 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2161: Upgrading db2161.codfw.wmnet * 11:21 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1056.eqiad.wmnet with reason: host reimage * 11:21 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P93527 and previous config saved to /var/cache/conftool/dbconfig/20260602-111954-fceratto.json * 11:15 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2161 [[phab:T427892|T427892]]', diff saved to https://phabricator.wikimedia.org/P93525 and previous config saved to /var/cache/conftool/dbconfig/20260602-111511-cwilliams.json * 11:12 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2165 to s8 primary [[phab:T427892|T427892]]', diff saved to https://phabricator.wikimedia.org/P93524 and previous config saved to /var/cache/conftool/dbconfig/20260602-111200-cwilliams.json * 11:10 cezmunsta: Starting s8 codfw failover from db2161 to db2165 - [[phab:T427892|T427892]] * 11:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P93523 and previous config saved to /var/cache/conftool/dbconfig/20260602-110947-fceratto.json * 11:09 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1057.eqiad.wmnet with OS trixie * 11:09 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1056.eqiad.wmnet with OS trixie * 11:04 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2165 with weight 0 [[phab:T427892|T427892]]', diff saved to https://phabricator.wikimedia.org/P93522 and previous config saved to /var/cache/conftool/dbconfig/20260602-110420-cwilliams.json * 11:03 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s8 [[phab:T427892|T427892]] * 11:02 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2056: repool after upgrade * 11:01 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93520 and previous config saved to /var/cache/conftool/dbconfig/20260602-105939-fceratto.json * 10:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93519 and previous config saved to /var/cache/conftool/dbconfig/20260602-105239-fceratto.json * 10:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 10:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance * 10:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93518 and previous config saved to /var/cache/conftool/dbconfig/20260602-105202-fceratto.json * 10:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2056.codfw.wmnet with OS trixie * 10:42 moritzm: installing busybox security updates * 10:42 claime: Enabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 10:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P93517 and previous config saved to /var/cache/conftool/dbconfig/20260602-104154-fceratto.json * 10:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P93516 and previous config saved to /var/cache/conftool/dbconfig/20260602-103146-fceratto.json * 10:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2056.codfw.wmnet with reason: host reimage * 10:27 claime: Disabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 10:25 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2056.codfw.wmnet with reason: host reimage * 10:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93515 and previous config saved to /var/cache/conftool/dbconfig/20260602-102139-fceratto.json * 10:09 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2056.codfw.wmnet with OS trixie * 10:08 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2056: Upgrading es2056.codfw.wmnet * 10:08 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2056: Upgrading es2056.codfw.wmnet * 10:08 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:06 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply * 10:06 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply * 09:56 claime: Enabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 09:46 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on cumin2003.codfw.wmnet with reason: in setup * 09:45 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1187: Pooling * 09:37 claime: Running puppet on cp6010 and cp6011 - [[phab:T422937|T422937]] * 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow2004.codfw.wmnet to plain * 09:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93511 and previous config saved to /var/cache/conftool/dbconfig/20260602-093716-fceratto.json * 09:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance * 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow2004.codfw.wmnet to plain * 09:34 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of rpki2003.codfw.wmnet to plain * 09:34 claime: Disabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 09:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of rpki2003.codfw.wmnet to plain * 09:32 moritzm: temporarily remove ganeti2045 from the codfw cluster [[phab:T427357|T427357]] * 09:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1055.eqiad.wmnet with OS trixie * 09:15 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1187: Pooling * 09:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1055.eqiad.wmnet with reason: host reimage * 09:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1187 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93508 and previous config saved to /var/cache/conftool/dbconfig/20260602-091126-fceratto.json * 09:09 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1055.eqiad.wmnet with reason: host reimage * 09:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1187 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93506 and previous config saved to /var/cache/conftool/dbconfig/20260602-090432-fceratto.json * 09:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1187.eqiad.wmnet with reason: Maintenance * 08:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2250.codfw.wmnet with reason: rack A3 maintenance * 08:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:56 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1055.eqiad.wmnet with OS trixie * 08:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:54 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:54 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:53 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 08:52 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 08:51 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 08:50 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 08:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 08:41 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:39 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:37 urbanecm: Reset user email of Barras@votewiki to the one of Barras@SUL * 08:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance * 08:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93505 and previous config saved to /var/cache/conftool/dbconfig/20260602-083033-fceratto.json * 08:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:29 slyngs: IDP, new configuration in preparation for webauthn * 08:20 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P93504 and previous config saved to /var/cache/conftool/dbconfig/20260602-082026-fceratto.json * 08:19 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 08:18 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 08:18 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:17 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296488{{!}}Revert "translate: adding separate read/write endpoints" (T425377)]] (duration: 03m 33s) * 08:16 atsuko@deploy1003: atsuko: Rolling back deployment * 08:16 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2053: repool after upgrade * 08:15 atsuko@deploy1003: atsuko: Backport for [[gerrit:1296488{{!}}Revert "translate: adding separate read/write endpoints" (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:13 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1296488{{!}}Revert "translate: adding separate read/write endpoints" (T425377)]] * 08:11 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:10 marostegui: Install mariadb 10.11.17 on es2053 [[phab:T427345|T427345]] * 08:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P93502 and previous config saved to /var/cache/conftool/dbconfig/20260602-081018-fceratto.json * 08:09 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:09 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: Depool for rack maintenance * 08:03 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296262{{!}}translate: fixing missed variable in credentials formatting closure (T425377)]] (duration: 14m 47s) * 08:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93499 and previous config saved to /var/cache/conftool/dbconfig/20260602-080011-fceratto.json * 07:59 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 07:59 atsuko@deploy1003: atsuko: Rolling back deployment * 07:58 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 07:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1181 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93498 and previous config saved to /var/cache/conftool/dbconfig/20260602-075759-fceratto.json * 07:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1181.eqiad.wmnet with reason: Maintenance * 07:57 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling * 07:50 atsuko@deploy1003: atsuko: Backport for [[gerrit:1296262{{!}}translate: fixing missed variable in credentials formatting closure (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:49 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1296262{{!}}translate: fixing missed variable in credentials formatting closure (T425377)]] * 07:48 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1181: Pooling * 07:47 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1181: Pooling * 07:44 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1181: Reboot * 07:43 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1181: Reboot * 07:42 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1181.eqiad.wmnet with reason: Reboot * 07:41 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 07:41 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:41 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1181: Migration of db1181.eqiad.wmnet completed * 07:40 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294949{{!}}translate: adding separate read/write endpoints (T425377)]] (duration: 21m 01s) * 07:39 atsuko@deploy1003: atsuko: Rolling back deployment * 07:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93490 and previous config saved to /var/cache/conftool/dbconfig/20260602-073904-fceratto.json * 07:32 XioNoX: pfw1-eqiad# delete protocols bgp group Production family inet6 - [[phab:T423384|T423384]] * 07:30 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2053: repool after upgrade * 07:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2158.codfw.wmnet with reason: rack A3 maintenance * 07:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93487 and previous config saved to /var/cache/conftool/dbconfig/20260602-072856-fceratto.json * 07:28 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2158: rack A3 maintenance * 07:28 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2158: rack A3 maintenance * 07:27 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on pc2021.codfw.wmnet with reason: rack A3 maintenance * 07:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc2021: rack A3 maintenance * 07:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 07:25 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache * 07:25 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool pc2021: rack A3 maintenance * 07:23 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2241: Depool for rack maintenance * 07:23 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2241.codfw.wmnet * 07:23 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2241.codfw.wmnet * 07:21 atsuko@deploy1003: atsuko: Backport for [[gerrit:1294949{{!}}translate: adding separate read/write endpoints (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:20 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2053.codfw.wmnet with OS trixie * 07:19 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1294949{{!}}translate: adding separate read/write endpoints (T425377)]] * 07:15 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2241.codfw.wmnet with reason: Depool for rack maintenance * 07:14 marostegui: Install mariadb 10.11.17 on db2186 [[phab:T427345|T427345]] * 07:12 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: Depool for rack maintenance * 07:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2186.codfw.wmnet with reason: upgrade * 07:12 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2241: Depool for rack maintenance * 07:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2053.codfw.wmnet with reason: host reimage * 06:59 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2053.codfw.wmnet with reason: host reimage * 06:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93478 and previous config saved to /var/cache/conftool/dbconfig/20260602-065533-fceratto.json * 06:55 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1181: Migration of db1181.eqiad.wmnet completed * 06:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 06:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1181.eqiad.wmnet with OS trixie * 06:43 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2053.codfw.wmnet with OS trixie * 06:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2053: Upgrading es2053.codfw.wmnet * 06:41 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2053: Upgrading es2053.codfw.wmnet * 06:41 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:37 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 06:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 06:36 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 06:36 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1052: repool after upgrade * 06:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1181.eqiad.wmnet with reason: host reimage * 06:24 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1181.eqiad.wmnet with reason: host reimage * 06:22 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 06:21 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 06:16 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 06:15 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 06:08 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1181.eqiad.wmnet with OS trixie * 06:05 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1181: Upgrading db1181.eqiad.wmnet * 06:05 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1181: Upgrading db1181.eqiad.wmnet * 06:04 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:02 marostegui@dns1004: END - running authdns-update * 06:01 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1181 [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93473 and previous config saved to /var/cache/conftool/dbconfig/20260602-060157-marostegui.json * 06:01 marostegui@dns1004: START - running authdns-update * 06:00 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db1236 to s7 primary and set section read-write [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93472 and previous config saved to /var/cache/conftool/dbconfig/20260602-060041-marostegui.json * 06:00 marostegui@cumin1003: dbctl commit (dc=all): 'Set s7 eqiad as read-only for maintenance - [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93471 and previous config saved to /var/cache/conftool/dbconfig/20260602-060018-marostegui.json * 06:00 marostegui: Starting s7 eqiad failover from db1181 to db1236 - [[phab:T426088|T426088]] * 05:51 marostegui@cumin1003: dbctl commit (dc=all): 'Set db1236 with weight 0 [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93470 and previous config saved to /var/cache/conftool/dbconfig/20260602-055153-marostegui.json * 05:51 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 [[phab:T426088|T426088]] * 05:50 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1052: repool after upgrade * 05:50 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 05:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1052.eqiad.wmnet with OS trixie * 05:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1052.eqiad.wmnet with reason: host reimage * 05:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:22 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1052.eqiad.wmnet with reason: host reimage * 05:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:07 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1052.eqiad.wmnet with OS trixie * 05:06 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1052: Upgrading es1052.eqiad.wmnet * 05:06 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1052: Upgrading es1052.eqiad.wmnet * 05:05 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 04:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 04:49 ryankemper: [[phab:T425007|T425007]] (k8s) created 4 wdqs namespaces on `dse-k8s-codfw`'s `admin_ng` ns: `wdqs-[internal,external]` & `wdqs-[internal,external]-next`; certs issued * 04:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 04:40 ryankemper@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 04:36 ryankemper@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 04:05 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.2 (duration: 05m 33s) == 2026-06-01 == * 23:27 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295963{{!}}Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962{{!}}Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] (duration: 07m 17s) * 23:23 jdlrobson@deploy1003: mfossati, jdlrobson: Continuing with deployment * 23:22 jdlrobson@deploy1003: mfossati, jdlrobson: Backport for [[gerrit:1295963{{!}}Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962{{!}}Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:20 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1295963{{!}}Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962{{!}}Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] * 23:15 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296022{{!}}Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028{{!}}styles: Limit selector to badge client pref (T427407)]] (duration: 09m 33s) * 23:11 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 23:07 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1296022{{!}}Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028{{!}}styles: Limit selector to badge client pref (T427407)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:06 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1296022{{!}}Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028{{!}}styles: Limit selector to badge client pref (T427407)]] * 23:04 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6015.* * 22:36 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296024{{!}}Add maintenance script to scrape SVG render files]] (duration: 06m 22s) * 22:32 reedy@deploy1003: reedy: Continuing with deployment * 22:31 reedy@deploy1003: reedy: Backport for [[gerrit:1296024{{!}}Add maintenance script to scrape SVG render files]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:30 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1296024{{!}}Add maintenance script to scrape SVG render files]] * 22:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 22:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 22:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 21:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 21:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 21:51 sbassett: Deployed updated mitigation for [[phab:T326691|T326691]] * 21:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 21:35 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 21:35 maryum: Deployed security fix for [[phab:T427611|T427611]] * 21:35 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 21:33 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 21:32 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 21:27 maryum: Deployed security fix for [[phab:T427235|T427235]] * 21:13 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296002{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009{{!}}Redirect Special:AccountRecovery to the shared domain (T427692)]] (duration: 09m 20s) * 21:09 catrope@deploy1003: catrope, arlolra: Continuing with deployment * 21:09 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 21:09 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 21:08 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 21:07 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 21:07 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 21:06 catrope@deploy1003: catrope, arlolra: Backport for [[gerrit:1296002{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009{{!}}Redirect Special:AccountRecovery to the shared domain (T427692)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:04 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1296002{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009{{!}}Redirect Special:AccountRecovery to the shared domain (T427692)]] * 20:53 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 20:37 ryankemper@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wdqs1015.eqiad.wmnet with reason: [[phab:T427852|T427852]] hw failure * 20:26 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1285412{{!}}Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531{{!}}Enable AbuseFilter block action on nlwiki (T427384)]] (duration: 07m 48s) * 20:22 catrope@deploy1003: sfaci, xxblackburnxx, catrope: Continuing with deployment * 20:20 catrope@deploy1003: sfaci, xxblackburnxx, catrope: Backport for [[gerrit:1285412{{!}}Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531{{!}}Enable AbuseFilter block action on nlwiki (T427384)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:18 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1285412{{!}}Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531{{!}}Enable AbuseFilter block action on nlwiki (T427384)]] * 20:12 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295504{{!}}passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] (duration: 07m 37s) * 20:08 catrope@deploy1003: catrope: Continuing with deployment * 20:07 catrope@deploy1003: catrope: Backport for [[gerrit:1295504{{!}}passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:05 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1295504{{!}}passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] * 19:48 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 19:47 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 19:47 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 19:46 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 19:46 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 19:45 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 19:01 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: sync * 19:00 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: sync * 18:24 otto@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295950{{!}}mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] (duration: 06m 42s) * 18:20 otto@deploy1003: otto: Continuing with deployment * 18:19 otto@deploy1003: otto: Backport for [[gerrit:1295950{{!}}mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:17 otto@deploy1003: Started scap sync-world: Backport for [[gerrit:1295950{{!}}mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] * 18:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 18:05 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 18:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd2001.codfw.wmnet to plain * 18:02 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 18:02 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd2001.codfw.wmnet to plain * 18:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2003.codfw.wmnet to plain * 18:01 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply * 18:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2003.codfw.wmnet to plain * 17:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 17:58 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 17:53 jasmine@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2006.codfw.wmnet with OS trixie * 17:42 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295976{{!}}nlwiki: change to Wikipedia 25 logo (T424519)]] (duration: 07m 29s) * 17:37 samtar@deploy1003: chlod, samtar: Continuing with deployment * 17:36 samtar@deploy1003: chlod, samtar: Backport for [[gerrit:1295976{{!}}nlwiki: change to Wikipedia 25 logo (T424519)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:34 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1295976{{!}}nlwiki: change to Wikipedia 25 logo (T424519)]] * 17:20 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1236: Update * 17:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd2001.codfw.wmnet to drbd * 17:04 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling * 17:04 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 17:04 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1180: Pooling * 17:03 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 17:03 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling * 17:03 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 16:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd2001.codfw.wmnet to drbd * 16:58 Amir1: drop flaggedrevs tables on wikinews wikis ([[phab:T423577|T423577]]) * 16:57 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 16:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93462 and previous config saved to /var/cache/conftool/dbconfig/20260601-165717-fceratto.json * 16:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93460 and previous config saved to /var/cache/conftool/dbconfig/20260601-164709-fceratto.json * 16:42 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling * 16:37 ryankemper@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,cluster=wdqs-main,service=wdqs-main,name=wdqs1015.eqiad.wmnet * 16:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93458 and previous config saved to /var/cache/conftool/dbconfig/20260601-163701-fceratto.json * 16:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:35 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1236.eqiad.wmnet * 16:35 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1236.eqiad.wmnet * 16:35 ryankemper@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,cluster=wdqs,service=wdqs-main,name=wdqs1015.eqiad.wmnet * 16:34 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update * 16:34 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1236: Update * 16:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1236.eqiad.wmnet with reason: Kernel update [[phab:T426633|T426633]] * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:30 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1236.eqiad.wmnet * 16:30 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1236.eqiad.wmnet * 16:30 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update * 16:29 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1236: Update * 16:29 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update * 16:29 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2003.codfw.wmnet to drbd * 16:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93455 and previous config saved to /var/cache/conftool/dbconfig/20260601-162653-fceratto.json * 16:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 16:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1209: Migration of db1209.eqiad.wmnet completed * 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1236.eqiad.wmnet with reason: Kernel update [[phab:T426633|T426633]] * 16:09 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1236: Update * 16:09 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1236: Update * 16:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:06 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 16:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2003.codfw.wmnet to drbd * 16:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 16:03 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 16:02 moritzm: temporarily remove ganeti2027 from the codfw cluster [[phab:T427357|T427357]] * 15:56 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:56 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.depool (exit_code=97) depool db1224: Pooling * 15:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2005.codfw.wmnet with OS bullseye * 15:53 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1224: Pooling * 15:51 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 15:49 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling * 15:49 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:48 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet * 15:45 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling * 15:44 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2005.codfw.wmnet with reason: host reimage * 15:40 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:40 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling * 15:40 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet * 15:40 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet * 15:40 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet * 15:40 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet * 15:39 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 15:39 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 15:39 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1209: Migration of db1209.eqiad.wmnet completed * 15:39 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:38 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:38 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling * 15:37 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2005.codfw.wmnet with reason: host reimage * 15:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 15:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1209.eqiad.wmnet with OS trixie * 15:28 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295802{{!}}hCaptcha: Raise SiteVerify error threshold to 100]] (duration: 06m 15s) * 15:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93446 and previous config saved to /var/cache/conftool/dbconfig/20260601-152638-fceratto.json * 15:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 15:26 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:25 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet * 15:25 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet * 15:25 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling * 15:25 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:24 kharlan@deploy1003: kharlan: Continuing with deployment * 15:24 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295802{{!}}hCaptcha: Raise SiteVerify error threshold to 100]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:22 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host testvm2005.codfw.wmnet with OS bullseye * 15:22 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295802{{!}}hCaptcha: Raise SiteVerify error threshold to 100]] * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:20 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295946{{!}}hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] (duration: 08m 24s) * 15:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:16 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 15:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1209.eqiad.wmnet with reason: host reimage * 15:14 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1295946{{!}}hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:13 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:12 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1295946{{!}}hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] * 15:10 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1209.eqiad.wmnet with reason: host reimage * 15:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93445 and previous config saved to /var/cache/conftool/dbconfig/20260601-151024-fceratto.json * 15:08 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-reboot (exit_code=0) rolling reboot on A:sessionstore * 15:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93443 and previous config saved to /var/cache/conftool/dbconfig/20260601-150017-fceratto.json * 14:55 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1209.eqiad.wmnet with OS trixie * 14:52 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 14:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1209: Upgrading db1209.eqiad.wmnet * 14:52 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 14:52 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1209: Upgrading db1209.eqiad.wmnet * 14:52 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 14:51 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:51 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 14:50 atsuko@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 14:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93441 and previous config saved to /var/cache/conftool/dbconfig/20260601-145010-fceratto.json * 14:49 atsuko@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 14:49 atsuko@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 14:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:42 atsuko@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 14:41 atsuko@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93440 and previous config saved to /var/cache/conftool/dbconfig/20260601-144002-fceratto.json * 14:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:30 ladsgroup@deploy1003: Synchronized portals: Deploy portals ([[phab:T421797|T421797]]) (duration: 02m 43s) * 14:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:27 ladsgroup@deploy1003: Synchronized portals/wikipedia.org/assets: Deploy portals ([[phab:T421797|T421797]]) (duration: 06m 10s) * 14:25 sukhe@dns1004: END - running authdns-update * 14:23 sukhe@dns1004: START - running authdns-update * 14:22 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 14:21 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:16 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:12 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:12 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:11 Lucas_WMDE: UTC afternoon backport+config window done * 14:10 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295918{{!}}Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] (duration: 11m 06s) * 14:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:05 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, codenamenoreste: Continuing with deployment * 14:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, codenamenoreste: Backport for [[gerrit:1295918{{!}}Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:01 eevans@cumin1003: START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:sessionstore * 13:58 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1295918{{!}}Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] * 13:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:52 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1265.eqiad.wmnet with OS trixie * 13:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93439 and previous config saved to /var/cache/conftool/dbconfig/20260601-133947-fceratto.json * 13:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 13:37 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage * 13:35 atsukoito: restarted pybal.service on lvs2013 * 13:31 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage * 13:31 atsukoito: restarted pybal.service on lvs2014 * 13:24 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs-test2001.codfw.wmnet * 13:24 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:22 atsukoito: restarted pybal.service on lvs1019 * 13:22 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in eqiad/ml-serve-eqiad: maintenance * 13:21 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in eqiad/ml-serve-eqiad: maintenance * 13:20 atsukoito: restarted pybal.service on lvs1020 * 13:20 Msz2001: UTC afternoon backpot+config window done * 13:20 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295875{{!}}Add SetGlobalPreference maintenance script (T427476)]] (duration: 06m 22s) * 13:19 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs-test2001.codfw.wmnet * 13:18 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1265.eqiad.wmnet with OS trixie * 13:18 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:16 mszwarc@deploy1003: mszwarc: Continuing with deployment * 13:15 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1295875{{!}}Add SetGlobalPreference maintenance script (T427476)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:14 atsukoito: sudo cumin 'A:lvs-low-traffic-eqiad' 'systemctl restart pybal.service' * 13:14 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1295875{{!}}Add SetGlobalPreference maintenance script (T427476)]] * 13:12 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295536{{!}}swwiki: Enable the Visual Editor on the project namespace (T427117)]] (duration: 10m 06s) * 13:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93438 and previous config saved to /var/cache/conftool/dbconfig/20260601-130949-fceratto.json * 13:08 mszwarc@deploy1003: codenamenoreste, mszwarc: Continuing with deployment * 13:07 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 13:06 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . * 13:05 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 13:04 mszwarc@deploy1003: codenamenoreste, mszwarc: Backport for [[gerrit:1295536{{!}}swwiki: Enable the Visual Editor on the project namespace (T427117)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:04 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 13:04 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 13:03 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 13:02 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1295536{{!}}swwiki: Enable the Visual Editor on the project namespace (T427117)]] * 12:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93437 and previous config saved to /var/cache/conftool/dbconfig/20260601-125941-fceratto.json * 12:56 dpogorzelski@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=inference,name=eqiad * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'readability' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'logo-detection' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'edit-check' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . * 12:52 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:50 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:49 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93436 and previous config saved to /var/cache/conftool/dbconfig/20260601-124934-fceratto.json * 12:48 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:47 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:46 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:44 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:43 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:42 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:41 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93435 and previous config saved to /var/cache/conftool/dbconfig/20260601-123926-fceratto.json * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:29 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:28 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 12:28 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:27 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:27 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2005.codfw.wmnet to plain * 12:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2005.codfw.wmnet to plain * 12:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet * 12:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 12:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2005.codfw.wmnet to drbd * 12:20 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:17 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:15 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in eqiad/ml-serve-eqiad: maintenance * 12:15 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in eqiad/ml-serve-eqiad: maintenance * 12:11 dpogorzelski@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=inference,name=eqiad * 12:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2005.codfw.wmnet to drbd * 12:05 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet * 12:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 12:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2027.codfw.wmnet * 12:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 11:59 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in eqiad/ml-serve-eqiad: maintenance * 11:59 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in eqiad/ml-serve-eqiad: maintenance * 11:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93434 and previous config saved to /var/cache/conftool/dbconfig/20260601-113911-fceratto.json * 11:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 11:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93433 and previous config saved to /var/cache/conftool/dbconfig/20260601-113843-fceratto.json * 11:37 moritzm: installing Exim security updates * 11:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:32 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:32 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:32 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93432 and previous config saved to /var/cache/conftool/dbconfig/20260601-112835-fceratto.json * 11:25 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 11:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:22 moritzm: installing imagemagick security updates * 11:22 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:22 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:22 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 11:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93430 and previous config saved to /var/cache/conftool/dbconfig/20260601-111827-fceratto.json * 11:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:14 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 11:12 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 11:10 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93429 and previous config saved to /var/cache/conftool/dbconfig/20260601-110820-fceratto.json * 11:04 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 11:01 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1055: repool after upgrade * 11:01 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93427 and previous config saved to /var/cache/conftool/dbconfig/20260601-110121-fceratto.json * 11:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance * 10:54 marostegui@dns1004: END - running authdns-update * 10:52 marostegui@dns1004: START - running authdns-update * 10:48 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1050 to es1 eqiad primary [[phab:T427032|T427032]]', diff saved to https://phabricator.wikimedia.org/P93425 and previous config saved to /var/cache/conftool/dbconfig/20260601-104837-marostegui.json * 10:47 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2055 to es1 codfw primary [[phab:T427032|T427032]]', diff saved to https://phabricator.wikimedia.org/P93424 and previous config saved to /var/cache/conftool/dbconfig/20260601-104739-marostegui.json * 10:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1177: Migration of db1177.eqiad.wmnet completed * 10:40 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet * 10:34 kamila@cumin1003: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet * 10:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93421 and previous config saved to /var/cache/conftool/dbconfig/20260601-103316-fceratto.json * 10:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93418 and previous config saved to /var/cache/conftool/dbconfig/20260601-102308-fceratto.json * 10:16 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1055: repool after upgrade * 10:15 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1055.eqiad.wmnet with OS trixie * 10:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93415 and previous config saved to /var/cache/conftool/dbconfig/20260601-101300-fceratto.json * 10:09 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 10:07 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 10:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93414 and previous config saved to /var/cache/conftool/dbconfig/20260601-100252-fceratto.json * 10:00 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1177: Migration of db1177.eqiad.wmnet completed * 09:58 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1055.eqiad.wmnet with reason: host reimage * 09:56 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 09:54 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 09:53 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1055.eqiad.wmnet with reason: host reimage * 09:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1177.eqiad.wmnet with OS trixie * 09:51 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 09:50 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 09:39 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1055.eqiad.wmnet with OS trixie * 09:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1055: Upgrading es1055.eqiad.wmnet * 09:38 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1055: Upgrading es1055.eqiad.wmnet * 09:37 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1177.eqiad.wmnet with reason: host reimage * 09:31 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1177.eqiad.wmnet with reason: host reimage * 09:17 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1177.eqiad.wmnet with OS trixie * 09:15 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 09:14 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 09:13 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 09:12 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 09:12 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1177: Upgrading db1177.eqiad.wmnet * 09:11 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1177: Upgrading db1177.eqiad.wmnet * 09:11 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93410 and previous config saved to /var/cache/conftool/dbconfig/20260601-090237-fceratto.json * 09:02 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93409 and previous config saved to /var/cache/conftool/dbconfig/20260601-090209-fceratto.json * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P93408 and previous config saved to /var/cache/conftool/dbconfig/20260601-085202-fceratto.json * 08:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P93407 and previous config saved to /var/cache/conftool/dbconfig/20260601-084154-fceratto.json * 08:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93406 and previous config saved to /var/cache/conftool/dbconfig/20260601-083146-fceratto.json * 08:24 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93405 and previous config saved to /var/cache/conftool/dbconfig/20260601-082442-fceratto.json * 08:24 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1168.eqiad.wmnet with reason: Maintenance * 07:58 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295454{{!}}Disable the creation of synthetic main refs in production (T427484)]] (duration: 11m 26s) * 07:56 XioNoX: add no_p2p term to pfw1-codfw BGP_fundraising_export - [[phab:T423384|T423384]] * 07:52 wmde-fisch@deploy1003: lilients, wmde-fisch: Continuing with deployment * 07:51 wmde-fisch@deploy1003: lilients, wmde-fisch: Backport for [[gerrit:1295454{{!}}Disable the creation of synthetic main refs in production (T427484)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:47 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1295454{{!}}Disable the creation of synthetic main refs in production (T427484)]] * 07:45 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294826{{!}}Update VE core submodule to master (9cf5524e7) (T424232)]] (duration: 31m 34s) * 07:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 07:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 07:32 wmde-fisch@deploy1003: wmde-fisch: Continuing with deployment * 07:31 wmde-fisch@deploy1003: wmde-fisch: Backport for [[gerrit:1294826{{!}}Update VE core submodule to master (9cf5524e7) (T424232)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet * 07:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet * 07:13 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1294826{{!}}Update VE core submodule to master (9cf5524e7) (T424232)]] * 06:48 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 06:47 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. == 2026-05-31 == * 02:06 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 30s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-30 == * 16:21 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:21 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:21 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:21 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 06:39 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 06:39 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 06:39 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 06:38 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 27s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-29 == * 23:39 aokoth@cumin1003: END (PASS) - Cookbook sre.vrts.upgrade (exit_code=0) on VRTS host vrts1003.eqiad.wmnet * 23:37 aokoth@cumin1003: START - Cookbook sre.vrts.upgrade on VRTS host vrts1003.eqiad.wmnet * 21:42 catrope@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 21:41 catrope@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 17:40 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295487{{!}}Hide experiment if not active and no assigned group]] (duration: 06m 54s) * 17:35 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 17:34 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1295487{{!}}Hide experiment if not active and no assigned group]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:33 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1295487{{!}}Hide experiment if not active and no assigned group]] * 16:30 jgreen@dns1004: END - running authdns-update * 16:28 jgreen@dns1004: START - running authdns-update * 16:13 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:12 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 15:28 dancy@deploy1003: Installation of scap version "4.267.0" completed for 2 hosts * 15:26 dancy@deploy1003: Installing scap version "4.267.0" for 2 host(s) * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:15 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295466{{!}}GlobalPreferencesHandler: Cast auto-reveal expiry to int (T427625)]] (duration: 07m 58s) * 14:11 kharlan@deploy1003: kharlan: Continuing with deployment * 14:09 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295466{{!}}GlobalPreferencesHandler: Cast auto-reveal expiry to int (T427625)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:07 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295466{{!}}GlobalPreferencesHandler: Cast auto-reveal expiry to int (T427625)]] * 13:53 moritzm: imported OpenJDK 21 21.0.11+10-1~deb12u1 to component/jdk21 (backport of latest Java 21 security release for Bookworm) * 12:09 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader1006.wikimedia.org * 12:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1006.wikimedia.org with OS trixie * 11:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1006.wikimedia.org with reason: host reimage * 11:47 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1006.wikimedia.org with reason: host reimage * 11:36 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader1006.wikimedia.org with OS trixie * 11:15 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:15 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:13 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader1006.wikimedia.org on all recursors * 11:12 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader1006.wikimedia.org on all recursors * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:06 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:00 jmm@cumin2002: START - Cookbook sre.dns.netbox * 11:00 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader1006.wikimedia.org * 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader1005.wikimedia.org * 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1005.wikimedia.org with OS trixie * 10:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1005.wikimedia.org with reason: host reimage * 10:40 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2212: Pooling * 10:37 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1005.wikimedia.org with reason: host reimage * 10:27 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader1005.wikimedia.org with OS trixie * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 10:01 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:59 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:55 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 09:50 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 09:49 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:45 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:44 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup2014.codfw.wmnet with OS bookworm * 09:33 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:20 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup2014.codfw.wmnet with reason: host reimage * 09:12 jynus@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on backup2014.codfw.wmnet with reason: host reimage * 09:10 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 09:10 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 09:03 jelto@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM etherpad2002.codfw.wmnet * 08:59 jelto@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM etherpad2002.codfw.wmnet * 08:59 jelto: gnt-instance modify -B memory=4g,vcpus=1 etherpad2002.codfw.wmnet - [[phab:T427588|T427588]] * 08:54 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2014.codfw.wmnet with OS bookworm * 08:51 jelto@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM etherpad1004.eqiad.wmnet * 08:50 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams-internal: apply * 08:50 jynus@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host backup2014.codfw.wmnet with OS bookworm * 08:49 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams-internal: apply * 08:47 jelto@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM etherpad1004.eqiad.wmnet * 08:46 jelto: gnt-instance modify -B memory=4g,vcpus=1 etherpad1004.eqiad.wmnet - [[phab:T427588|T427588]] * 08:42 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2212: Pooling * 08:42 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 08:39 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2212: Pooling * 08:39 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 08:38 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams-internal: apply * 08:37 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams-internal: apply * 08:37 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams-internal: apply * 08:36 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams-internal: apply * 08:33 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2014.codfw.wmnet with OS bookworm * 08:31 jynus@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host backup2014.codfw.wmnet with OS bookworm * 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader1005.wikimedia.org on all recursors * 08:21 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader1005.wikimedia.org on all recursors * 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 08:21 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 08:18 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 08:17 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 08:16 jmm@cumin2002: START - Cookbook sre.dns.netbox * 08:16 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader1005.wikimedia.org * 08:05 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2212: Pooling * 07:59 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 07:59 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 07:54 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 07:54 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2212.codfw.wmnet * 07:54 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2212.codfw.wmnet * 07:22 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2014.codfw.wmnet with OS bookworm * 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader2006.wikimedia.org * 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader2006.wikimedia.org with OS trixie * 06:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader2006.wikimedia.org with reason: host reimage * 06:53 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader2006.wikimedia.org with reason: host reimage * 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader2006.wikimedia.org with OS trixie * 06:32 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:32 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader2006.wikimedia.org on all recursors * 06:31 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader2006.wikimedia.org on all recursors * 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:31 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:27 jmm@cumin2002: START - Cookbook sre.dns.netbox * 06:27 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader2006.wikimedia.org * 03:01 vriley@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts db1224.eqiad.wmnet * 03:00 vriley@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1224.eqiad.wmnet * 03:00 vriley@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1224.eqiad.wmnet * 02:56 vriley@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1224.eqiad.wmnet * 01:47 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5032.eqsin.wmnet with OS trixie * 01:18 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5032.eqsin.wmnet with reason: host reimage * 01:14 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5032.eqsin.wmnet with reason: host reimage * 00:31 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cp5032.eqsin.wmnet with OS trixie * 00:29 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host cp5032.eqsin.wmnet * 00:23 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 00:22 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply * 00:21 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 00:21 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply == 2026-05-28 == * 23:07 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 23:07 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new ae1.522 interface - pt1979@cumin2002" * 23:07 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new ae1.522 interface - pt1979@cumin2002" * 23:02 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 22:34 andrewbogott: reprepro includedeb trixie-wikimedia /home/andrew/magnum-cluster-api_0.36.6-1~wmf13u2_amd64.deb * 22:31 logmsgbot: dreamyjazz Deployed security patch for [[phab:T426388|T426388]] * 21:33 maryum: Deployed security fix for [[phab:T426867|T426867]] * 21:21 alexsanford: Deployed security fix for [[phab:T426889|T426889]] * 21:07 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host cp5032.eqsin.wmnet * 21:04 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "setup new eqsin vlan - pt1979@cumin2002 - [[phab:T427393|T427393]]" * 21:04 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "setup new eqsin vlan - pt1979@cumin2002 - [[phab:T427393|T427393]]" * 20:48 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295066{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T420336 T427098 T427354 T427082)]], [[gerrit:1295067{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T427082)]] (duration: 07m 34s) * 20:44 arlolra@deploy1003: arlolra: Continuing with deployment * 20:43 arlolra@deploy1003: arlolra: Backport for [[gerrit:1295066{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T420336 T427098 T427354 T427082)]], [[gerrit:1295067{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T427082)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:41 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1295066{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T420336 T427098 T427354 T427082)]], [[gerrit:1295067{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T427082)]] * 20:34 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293805{{!}}Deploy PRV to 7 wikis (T427331)]] (duration: 07m 20s) * 20:30 arlolra@deploy1003: arlolra: Continuing with deployment * 20:29 arlolra@deploy1003: arlolra: Backport for [[gerrit:1293805{{!}}Deploy PRV to 7 wikis (T427331)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:27 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1293805{{!}}Deploy PRV to 7 wikis (T427331)]] * 20:22 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291996{{!}}Replace deprecated Hooks::getInstance (T426981)]], [[gerrit:1294393{{!}}Permissions: Create wmf-officeit group on officewiki]], [[gerrit:1294229{{!}}Deploy IRS Direct Reporting feature to enwiki (T427369)]], [[gerrit:1295039{{!}}Add 2FA enforcement demotion config for phase 2 groups (T423119)]] (duration: 09m 07s) * 20:18 stran@deploy1003: alexsanford, stran, catrope, dreamyjazz: Continuing with deployment * 20:14 stran@deploy1003: alexsanford, stran, catrope, dreamyjazz: Backport for [[gerrit:1291996{{!}}Replace deprecated Hooks::getInstance (T426981)]], [[gerrit:1294393{{!}}Permissions: Create wmf-officeit group on officewiki]], [[gerrit:1294229{{!}}Deploy IRS Direct Reporting feature to enwiki (T427369)]], [[gerrit:1295039{{!}}Add 2FA enforcement demotion config for phase 2 groups (T423119)]] synced to the testservers (see https://wikitech. * 20:13 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5032.eqsin.wmnet with OS trixie * 20:13 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1291996{{!}}Replace deprecated Hooks::getInstance (T426981)]], [[gerrit:1294393{{!}}Permissions: Create wmf-officeit group on officewiki]], [[gerrit:1294229{{!}}Deploy IRS Direct Reporting feature to enwiki (T427369)]], [[gerrit:1295039{{!}}Add 2FA enforcement demotion config for phase 2 groups (T423119)]] * 19:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1018.eqiad.wmnet * 19:27 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1018.eqiad.wmnet * 19:09 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1018.eqiad.wmnet with reason: Kernel reboot * 19:09 brett: Stopping pybal/puppet/downtiming lvs1018.eqiad.wmnet for reboot * 19:05 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1019.eqiad.wmnet * 19:05 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1019.eqiad.wmnet * 18:52 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cp5032.eqsin.wmnet with OS trixie * 18:51 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:51 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change cp5032 IP - pt1979@cumin2002" * 18:51 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change cp5032 IP - pt1979@cumin2002" * 18:47 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 18:40 mutante: planet1003/planet2003 - apt-get upgrade - all pending package upgrades * 18:35 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: Kernel reboot * 18:34 brett: Stopping pybal/puppet/downtiming lvs1019.eqiad.wmnet for reboot and BIOS update/memory self-healing - [[phab:T426109|T426109]] * 18:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs2011.codfw.wmnet * 18:25 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs2011.codfw.wmnet * 18:19 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2011.codfw.wmnet with reason: Kernel reboot * 18:19 brett: Stopping pybal/puppet/downtiming lvs2011.codfw.wmnet for reboot * 18:09 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs2013.codfw.wmnet * 18:06 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs2013.codfw.wmnet * 18:00 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2013.codfw.wmnet with reason: Kernel reboot * 17:57 brett: Stopping pybal/puppet/downtiming lvs2013.codfw.wmnet for reboot * 17:19 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 16:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93393 and previous config saved to /var/cache/conftool/dbconfig/20260528-164514-fceratto.json * 16:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P93392 and previous config saved to /var/cache/conftool/dbconfig/20260528-163507-fceratto.json * 16:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P93391 and previous config saved to /var/cache/conftool/dbconfig/20260528-162459-fceratto.json * 16:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db1224.eqiad.wmnet with reason: unreachable [[phab:T427535|T427535]] * 16:17 swfrench-wmf: reprepro include xdebug_3.4.4-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:17 swfrench-wmf: reprepro include wikidiff2_1.14.1-2+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:17 swfrench-wmf: reprepro include php-yaml_2.2.4-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-xhprof_2.3.10-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-wmerrors_2.0.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-uuid_1.3.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-redis_6.2.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 swfrench-wmf: reprepro include php-pcov_1.0.12-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 swfrench-wmf: reprepro include php-memcached_3.3.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 16:15 swfrench-wmf: reprepro include php-luasandbox_4.1.2-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 16:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93390 and previous config saved to /var/cache/conftool/dbconfig/20260528-161452-fceratto.json * 16:14 swfrench-wmf: reprepro include php-imagick_3.7.0-13+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:14 swfrench-wmf: reprepro include php-excimer_1.2.5-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:09 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:09 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:06 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1251 ([[phab:T426633|T426633]])', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20260528-160646-fceratto.json * 16:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1251.eqiad.wmnet with reason: Maintenance * 16:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93388 and previous config saved to /var/cache/conftool/dbconfig/20260528-160613-fceratto.json * 15:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P93387 and previous config saved to /var/cache/conftool/dbconfig/20260528-155605-fceratto.json * 15:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P93386 and previous config saved to /var/cache/conftool/dbconfig/20260528-154557-fceratto.json * 15:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93385 and previous config saved to /var/cache/conftool/dbconfig/20260528-153550-fceratto.json * 15:27 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1235 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93384 and previous config saved to /var/cache/conftool/dbconfig/20260528-152736-fceratto.json * 15:27 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1235.eqiad.wmnet with reason: Maintenance * 15:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93383 and previous config saved to /var/cache/conftool/dbconfig/20260528-152708-fceratto.json * 15:20 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp5032.eqsin.wmnet with reason: Testing reimaging on new subnet * 15:18 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5032.* * 15:17 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P93382 and previous config saved to /var/cache/conftool/dbconfig/20260528-151701-fceratto.json * 15:17 jhathaway: dmarc ingress test on mx-in1001 * 15:14 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:14 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P93381 and previous config saved to /var/cache/conftool/dbconfig/20260528-150653-fceratto.json * 14:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93380 and previous config saved to /var/cache/conftool/dbconfig/20260528-145646-fceratto.json * 14:56 moritzm: installing nginx security updates * 14:49 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1234 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93379 and previous config saved to /var/cache/conftool/dbconfig/20260528-144936-fceratto.json * 14:49 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 14:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1234.eqiad.wmnet with reason: Maintenance * 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93378 and previous config saved to /var/cache/conftool/dbconfig/20260528-144909-fceratto.json * 14:48 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader2005.wikimedia.org * 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader2005.wikimedia.org with OS trixie * 14:47 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 14:39 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2189.codfw.wmnet * 14:39 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2189.codfw.wmnet * 14:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P93377 and previous config saved to /var/cache/conftool/dbconfig/20260528-143901-fceratto.json * 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader2005.wikimedia.org with reason: host reimage * 14:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P93376 and previous config saved to /var/cache/conftool/dbconfig/20260528-142854-fceratto.json * 14:28 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:28 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader2005.wikimedia.org with reason: host reimage * 14:27 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:19 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294998{{!}}ImageContentLookup: Fix issue created by strict types (T427505)]], [[gerrit:1295001{{!}}Enable hCaptcha for VisualEditor in group 1 (T425940)]] (duration: 11m 29s) * 14:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93375 and previous config saved to /var/cache/conftool/dbconfig/20260528-141846-fceratto.json * 14:15 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1232 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93374 and previous config saved to /var/cache/conftool/dbconfig/20260528-141029-fceratto.json * 14:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1232.eqiad.wmnet with reason: Maintenance * 14:10 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader2005.wikimedia.org with OS trixie * 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93373 and previous config saved to /var/cache/conftool/dbconfig/20260528-141001-fceratto.json * 14:09 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1294998{{!}}ImageContentLookup: Fix issue created by strict types (T427505)]], [[gerrit:1295001{{!}}Enable hCaptcha for VisualEditor in group 1 (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:08 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294998{{!}}ImageContentLookup: Fix issue created by strict types (T427505)]], [[gerrit:1295001{{!}}Enable hCaptcha for VisualEditor in group 1 (T425940)]] * 14:00 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down * 13:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P93371 and previous config saved to /var/cache/conftool/dbconfig/20260528-135951-fceratto.json * 13:58 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp6015.drmrs.wmnet,service=(cdn{{!}}ats-be) * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:55 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader2005.wikimedia.org on all recursors * 13:55 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader2005.wikimedia.org on all recursors * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:55 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P93370 and previous config saved to /var/cache/conftool/dbconfig/20260528-134944-fceratto.json * 13:40 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 13:40 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 13:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93369 and previous config saved to /var/cache/conftool/dbconfig/20260528-133936-fceratto.json * 13:39 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 13:38 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 13:36 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294986{{!}}Image Carousel: check candidate pages (T427336)]] (duration: 06m 40s) * 13:34 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 13:33 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 13:32 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93368 and previous config saved to /var/cache/conftool/dbconfig/20260528-133230-fceratto.json * 13:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance * 13:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93367 and previous config saved to /var/cache/conftool/dbconfig/20260528-133202-fceratto.json * 13:31 mlitn@deploy1003: mlitn: Continuing with deployment * 13:31 mlitn@deploy1003: mlitn: Backport for [[gerrit:1294986{{!}}Image Carousel: check candidate pages (T427336)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:29 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1294986{{!}}Image Carousel: check candidate pages (T427336)]] * 13:22 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 13:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P93366 and previous config saved to /var/cache/conftool/dbconfig/20260528-132155-fceratto.json * 13:21 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 13:17 elukey: clean up a lof ot stale Kafka ACLs on Kafka Jumbo - Details in [[phab:T425528|T425528]] * 13:14 jmm@cumin2002: START - Cookbook sre.dns.netbox * 13:14 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader2005.wikimedia.org * 13:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P93365 and previous config saved to /var/cache/conftool/dbconfig/20260528-131147-fceratto.json * 13:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93364 and previous config saved to /var/cache/conftool/dbconfig/20260528-130139-fceratto.json * 12:54 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1218 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93363 and previous config saved to /var/cache/conftool/dbconfig/20260528-125439-fceratto.json * 12:54 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1218.eqiad.wmnet with reason: Maintenance * 12:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93362 and previous config saved to /var/cache/conftool/dbconfig/20260528-125412-fceratto.json * 12:48 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 12:48 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 12:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P93361 and previous config saved to /var/cache/conftool/dbconfig/20260528-124404-fceratto.json * 12:44 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 12:43 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 12:39 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 12:38 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 12:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P93360 and previous config saved to /var/cache/conftool/dbconfig/20260528-123357-fceratto.json * 12:25 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest1006.eqiad.wmnet with OS trixie * 12:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93359 and previous config saved to /var/cache/conftool/dbconfig/20260528-122349-fceratto.json * 12:15 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93358 and previous config saved to /var/cache/conftool/dbconfig/20260528-121551-fceratto.json * 12:15 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance * 12:15 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie * 12:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93357 and previous config saved to /var/cache/conftool/dbconfig/20260528-121523-fceratto.json * 12:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P93356 and previous config saved to /var/cache/conftool/dbconfig/20260528-120515-fceratto.json * 12:02 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest1006.eqiad.wmnet with OS trixie * 12:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 12:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 12:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 12:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 11:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P93355 and previous config saved to /var/cache/conftool/dbconfig/20260528-115508-fceratto.json * 11:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93354 and previous config saved to /var/cache/conftool/dbconfig/20260528-114500-fceratto.json * 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93353 and previous config saved to /var/cache/conftool/dbconfig/20260528-113635-fceratto.json * 11:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 11:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance * 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93352 and previous config saved to /var/cache/conftool/dbconfig/20260528-113559-fceratto.json * 11:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P93351 and previous config saved to /var/cache/conftool/dbconfig/20260528-112551-fceratto.json * 11:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P93350 and previous config saved to /var/cache/conftool/dbconfig/20260528-111543-fceratto.json * 11:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93349 and previous config saved to /var/cache/conftool/dbconfig/20260528-110536-fceratto.json * 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1195 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93348 and previous config saved to /var/cache/conftool/dbconfig/20260528-105820-fceratto.json * 10:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie * 10:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1195.eqiad.wmnet with reason: Maintenance * 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93347 and previous config saved to /var/cache/conftool/dbconfig/20260528-105753-fceratto.json * 10:56 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 10:55 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 10:55 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 10:55 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 10:50 moritzm: update trixie netboot image for 13.5 point release [[phab:T427072|T427072]] * 10:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P93346 and previous config saved to /var/cache/conftool/dbconfig/20260528-104745-fceratto.json * 10:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P93345 and previous config saved to /var/cache/conftool/dbconfig/20260528-103738-fceratto.json * 10:29 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P13724 # [[phab:T406971|T406971]] * 10:28 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P14223 # [[phab:T422264|T422264]] * 10:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93344 and previous config saved to /var/cache/conftool/dbconfig/20260528-102730-fceratto.json * 10:26 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P1748 # [[phab:T422392|T422392]] * 10:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93343 and previous config saved to /var/cache/conftool/dbconfig/20260528-101900-fceratto.json * 10:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance * 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93342 and previous config saved to /var/cache/conftool/dbconfig/20260528-101829-fceratto.json * 10:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P93341 and previous config saved to /var/cache/conftool/dbconfig/20260528-100822-fceratto.json * 09:59 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290687{{!}}stream: webrequest.page_view (T426092 T426091)]] (duration: 06m 41s) * 09:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P93340 and previous config saved to /var/cache/conftool/dbconfig/20260528-095814-fceratto.json * 09:55 javiermonton@deploy1003: javiermonton: Continuing with deployment * 09:54 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1290687{{!}}stream: webrequest.page_view (T426092 T426091)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:52 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1290687{{!}}stream: webrequest.page_view (T426092 T426091)]] * 09:48 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294243{{!}}Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937{{!}}CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] (duration: 07m 37s) * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93339 and previous config saved to /var/cache/conftool/dbconfig/20260528-094807-fceratto.json * 09:44 dreamyjazz@deploy1003: dreamyjazz, stran: Continuing with deployment * 09:44 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:43 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:43 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:43 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:42 dreamyjazz@deploy1003: dreamyjazz, stran: Backport for [[gerrit:1294243{{!}}Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937{{!}}CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294243{{!}}Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937{{!}}CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] * 09:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93338 and previous config saved to /var/cache/conftool/dbconfig/20260528-093920-fceratto.json * 09:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: Maintenance * 09:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93337 and previous config saved to /var/cache/conftool/dbconfig/20260528-093849-fceratto.json * 09:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P93336 and previous config saved to /var/cache/conftool/dbconfig/20260528-092842-fceratto.json * 09:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance * 09:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93335 and previous config saved to /var/cache/conftool/dbconfig/20260528-092239-fceratto.json * 09:22 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pki-root1001.eqiad.wmnet * 09:22 elukey@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:22 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki-root1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1003" * 09:22 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki-root1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1003" * 09:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:18 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 09:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P93334 and previous config saved to /var/cache/conftool/dbconfig/20260528-091834-fceratto.json * 09:18 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 09:18 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 09:17 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1165: Reboot completed * 09:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 09:17 elukey@cumin1003: START - Cookbook sre.dns.netbox * 09:14 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 09:13 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:13 elukey@cumin1003: START - Cookbook sre.hosts.decommission for hosts pki-root1001.eqiad.wmnet * 09:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P93332 and previous config saved to /var/cache/conftool/dbconfig/20260528-091231-fceratto.json * 09:09 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:09 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93331 and previous config saved to /var/cache/conftool/dbconfig/20260528-090826-fceratto.json * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P93329 and previous config saved to /var/cache/conftool/dbconfig/20260528-090224-fceratto.json * 09:02 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Deploying to prod (duration: 02m 31s) * 09:01 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2216 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93328 and previous config saved to /var/cache/conftool/dbconfig/20260528-090114-fceratto.json * 09:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2216.codfw.wmnet with reason: Maintenance * 09:00 joal@deploy1003: Finished deploy [analytics/refinery@878cb24] (thin): Regular analytics weekly train THIN - 2[analytics/refinery@878cb24a] (duration: 02m 08s) * 08:59 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Deploying to prod * 08:58 joal@deploy1003: Started deploy [analytics/refinery@878cb24] (thin): Regular analytics weekly train THIN - 2[analytics/refinery@878cb24a] * 08:57 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Testing on backup host (duration: 00m 53s) * 08:56 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Testing on backup host * 08:56 joal@deploy1003: Finished deploy [analytics/refinery@878cb24]: Regular analytics weekly train - 2 [analytics/refinery@878cb24a] (duration: 06m 54s) * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93327 and previous config saved to /var/cache/conftool/dbconfig/20260528-085216-fceratto.json * 08:50 XioNoX: cr1-codfw# delete protocols bgp group fundraising family inet6 - [[phab:T423384|T423384]] * 08:49 joal@deploy1003: Started deploy [analytics/refinery@878cb24]: Regular analytics weekly train - 2 [analytics/refinery@878cb24a] * 08:49 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294925{{!}}hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] (duration: 09m 20s) * 08:49 joal@deploy1003: Finished deploy [analytics/refinery@878cb24] (hadoop-test): Regular analytics weekly train TEST -2 [analytics/refinery@878cb24a] (duration: 02m 00s) * 08:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93326 and previous config saved to /var/cache/conftool/dbconfig/20260528-084906-fceratto.json * 08:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1209.eqiad.wmnet with reason: Maintenance * 08:48 slyngshede@dns1004: END - running authdns-update * 08:47 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1165: Reboot completed * 08:47 joal@deploy1003: Started deploy [analytics/refinery@878cb24] (hadoop-test): Regular analytics weekly train TEST -2 [analytics/refinery@878cb24a] * 08:47 slyngs: Upgrade IDP to CAS 7.3.7.1 * 08:46 slyngshede@dns1004: START - running authdns-update * 08:45 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 08:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93324 and previous config saved to /var/cache/conftool/dbconfig/20260528-084149-fceratto.json * 08:41 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1294925{{!}}hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294925{{!}}hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] * 08:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet * 08:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet * 08:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93323 and previous config saved to /var/cache/conftool/dbconfig/20260528-083504-fceratto.json * 08:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1025].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 08:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1165.eqiad.wmnet with reason: Maintenance * 08:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93322 and previous config saved to /var/cache/conftool/dbconfig/20260528-083331-fceratto.json * 08:24 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1209: Test * 08:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P93320 and previous config saved to /var/cache/conftool/dbconfig/20260528-082324-fceratto.json * 08:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2189: repool after crash * 08:17 slyngshede@dns1004: END - running authdns-update * 08:16 slyngshede@dns1004: START - running authdns-update * 08:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P93318 and previous config saved to /var/cache/conftool/dbconfig/20260528-081316-fceratto.json * 08:10 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 08:09 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1209: Test * 08:05 hashar@deploy1003: Finished deploy [integration/docroot@2a51016]: build: update dependencies + eslint fix in comment. f021d3f..2a51016 (duration: 00m 13s) * 08:05 hashar@deploy1003: Started deploy [integration/docroot@2a51016]: build: update dependencies + eslint fix in comment. f021d3f..2a51016 * 08:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93315 and previous config saved to /var/cache/conftool/dbconfig/20260528-080309-fceratto.json * 07:56 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93314 and previous config saved to /var/cache/conftool/dbconfig/20260528-075631-fceratto.json * 07:56 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020,1022-1023].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 07:56 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance * 07:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93313 and previous config saved to /var/cache/conftool/dbconfig/20260528-075521-fceratto.json * 07:47 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab replica * 07:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93311 and previous config saved to /var/cache/conftool/dbconfig/20260528-074513-fceratto.json * 07:37 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2189: repool after crash * 07:36 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab replica * 07:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93309 and previous config saved to /var/cache/conftool/dbconfig/20260528-073506-fceratto.json * 07:34 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica * 07:29 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294808{{!}}Don't run the click intent experiment on mobile (T426743)]] (duration: 06m 29s) * 07:25 wmde-fisch@deploy1003: thiemowmde, wmde-fisch: Continuing with deployment * 07:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93308 and previous config saved to /var/cache/conftool/dbconfig/20260528-072458-fceratto.json * 07:24 wmde-fisch@deploy1003: thiemowmde, wmde-fisch: Backport for [[gerrit:1294808{{!}}Don't run the click intent experiment on mobile (T426743)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:24 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica * 07:23 tgr@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwikisource --logwiki=metawiki Ioed Renamed_user_4232d41570b9e8f46ef150e5e360e446 # [[phab:T427459|T427459]] * 07:22 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1294808{{!}}Don't run the click intent experiment on mobile (T426743)]] * 07:20 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1270986{{!}}Update wikimania wordmark for 2026 (T413331)]] (duration: 06m 54s) * 07:18 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93307 and previous config saved to /var/cache/conftool/dbconfig/20260528-071836-fceratto.json * 07:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1264.eqiad.wmnet with reason: Maintenance * 07:16 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1167: Reboot completed * 07:16 wmde-fisch@deploy1003: wmde-fisch, robertsky: Continuing with deployment * 07:15 wmde-fisch@deploy1003: wmde-fisch, robertsky: Backport for [[gerrit:1270986{{!}}Update wikimania wordmark for 2026 (T413331)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:13 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1270986{{!}}Update wikimania wordmark for 2026 (T413331)]] * 07:11 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289898{{!}}Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] (duration: 07m 15s) * 07:07 wmde-fisch@deploy1003: wmde-fisch, arthurtaylor: Continuing with deployment * 07:06 wmde-fisch@deploy1003: wmde-fisch, arthurtaylor: Backport for [[gerrit:1289898{{!}}Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:04 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1289898{{!}}Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] * 06:43 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1167: Reboot completed * 06:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93303 and previous config saved to /var/cache/conftool/dbconfig/20260528-064217-fceratto.json * 06:33 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93302 and previous config saved to /var/cache/conftool/dbconfig/20260528-063357-fceratto.json * 06:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 06:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance * 06:25 hashar: Restarting CI Jenkins for plugins upgrades * 06:16 fceratto@dns1005: END - running authdns-update * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1209 [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93301 and previous config saved to /var/cache/conftool/dbconfig/20260528-061609-fceratto.json * 06:14 fceratto@dns1005: START - running authdns-update * 06:11 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1193 to s8 primary and set section read-write [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93300 and previous config saved to /var/cache/conftool/dbconfig/20260528-061138-fceratto.json * 06:10 fceratto@cumin1003: dbctl commit (dc=all): 'Set s8 eqiad as read-only for maintenance - [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93299 and previous config saved to /var/cache/conftool/dbconfig/20260528-061048-fceratto.json * 06:10 federico3: Starting s8 eqiad failover from db1209 to db1193 - [[phab:T426095|T426095]] * 06:04 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1193 with weight 0 [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93298 and previous config saved to /var/cache/conftool/dbconfig/20260528-060412-fceratto.json * 06:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s8 [[phab:T426095|T426095]] * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 41s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:53 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 00:53 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new subnet in eqsin - pt1979@cumin2002" * 00:53 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new subnet in eqsin - pt1979@cumin2002" * 00:49 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 00:25 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294470{{!}}Activate conductwiki (T426984)]] (duration: 07m 12s) * 00:21 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 00:20 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1294470{{!}}Activate conductwiki (T426984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 00:18 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1294470{{!}}Activate conductwiki (T426984)]] * 00:12 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294438{{!}}Init conductwiki (T426984)]] (duration: 07m 25s) * 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 00:08 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 00:06 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1294438{{!}}Init conductwiki (T426984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 00:04 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1294438{{!}}Init conductwiki (T426984)]] * 00:04 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] == 2026-05-27 == * 23:13 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294432{{!}}Exclude more content from selection (T426308)]], [[gerrit:1285523{{!}}Remove MinervaNightMode config after skin cleanup (T426689)]] (duration: 08m 42s) * 23:09 jdlrobson@deploy1003: jdlrobson, h2o, egardner: Continuing with deployment * 23:06 jdlrobson@deploy1003: jdlrobson, h2o, egardner: Backport for [[gerrit:1294432{{!}}Exclude more content from selection (T426308)]], [[gerrit:1285523{{!}}Remove MinervaNightMode config after skin cleanup (T426689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:04 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294432{{!}}Exclude more content from selection (T426308)]], [[gerrit:1285523{{!}}Remove MinervaNightMode config after skin cleanup (T426689)]] * 22:58 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294435{{!}}passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] (duration: 07m 49s) * 22:55 ladsgroup@cumin1003: END (PASS) - Cookbook sre.mysql.sanitarium_restart (exit_code=0) * 22:54 catrope@deploy1003: catrope: Continuing with deployment * 22:52 catrope@deploy1003: catrope: Backport for [[gerrit:1294435{{!}}passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:50 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1294435{{!}}passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] * 22:46 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294360{{!}}Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322{{!}}Thumbnails are not being optimized in large mode (T427237)]] (duration: 06m 54s) * 22:42 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 22:41 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1294360{{!}}Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322{{!}}Thumbnails are not being optimized in large mode (T427237)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:40 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart * 22:40 ladsgroup@cumin1003: END (FAIL) - Cookbook sre.mysql.sanitarium_restart (exit_code=99) * 22:40 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart * 22:39 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294360{{!}}Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322{{!}}Thumbnails are not being optimized in large mode (T427237)]] * 22:39 ladsgroup@deploy1003: Finished scap sync-world: Add conduct.wikimedia.org ([[phab:T426984|T426984]]) (duration: 07m 16s) * 22:35 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 22:34 ladsgroup@deploy1003: ladsgroup: Add conduct.wikimedia.org ([[phab:T426984|T426984]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:33 ladsgroup@deploy1003: Started scap sync-world: Add conduct.wikimedia.org ([[phab:T426984|T426984]]) * 22:13 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294370{{!}}Carousel only on articles (T427336)]] (duration: 10m 00s) * 22:09 egardner@deploy1003: egardner: Continuing with deployment * 22:05 egardner@deploy1003: egardner: Backport for [[gerrit:1294370{{!}}Carousel only on articles (T427336)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:03 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1294370{{!}}Carousel only on articles (T427336)]] * 21:37 bking@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 15 days, 0:00:00 on relforge[1008-1010].eqiad.wmnet with reason: non-production environment * 21:20 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 21:20 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 21:20 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 21:19 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 21:04 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1288370{{!}}Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819{{!}}Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] (duration: 07m 38s) * 20:59 ebernhardson@deploy1003: matmarex, ebernhardson, pppery: Continuing with deployment * 20:58 ebernhardson@deploy1003: matmarex, ebernhardson, pppery: Backport for [[gerrit:1288370{{!}}Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819{{!}}Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:56 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1288370{{!}}Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819{{!}}Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] * 20:51 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294373{{!}}identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374{{!}}Revert^2 "cirrus: AB test query suggester variants" (T407432)]] (duration: 07m 30s) * 20:47 ebernhardson@deploy1003: ebernhardson: Continuing with deployment * 20:46 ebernhardson@deploy1003: ebernhardson: Backport for [[gerrit:1294373{{!}}identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374{{!}}Revert^2 "cirrus: AB test query suggester variants" (T407432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:44 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294373{{!}}identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374{{!}}Revert^2 "cirrus: AB test query suggester variants" (T407432)]] * 20:43 swfrench-wmf: reprepro include dh-php_5.5+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 20:39 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts lvs1016.eqiad.wmnet * 20:39 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:39 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brett@cumin2002" * 20:38 swfrench-wmf: reprepro include php-defaults_94+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 20:37 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brett@cumin2002" * 20:31 brett@cumin2002: START - Cookbook sre.dns.netbox * 20:27 swfrench-wmf: reprepro include php8.3_8.3.31-1+wmf12u2 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 20:25 brett@cumin2002: START - Cookbook sre.hosts.decommission for hosts lvs1016.eqiad.wmnet * 20:25 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294342{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344{{!}}frwiki: restrict Article Guidance experiment to junior editors (T426871)]] (duration: 08m 11s) * 20:21 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host lvs1016.eqiad.wmnet with OS bullseye * 20:21 sbisson@deploy1003: sbisson: Continuing with deployment * 20:20 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1020.eqiad.wmnet * 20:19 sbisson@deploy1003: sbisson: Backport for [[gerrit:1294342{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344{{!}}frwiki: restrict Article Guidance experiment to junior editors (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be v * 20:17 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294342{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344{{!}}frwiki: restrict Article Guidance experiment to junior editors (T426871)]] * 20:14 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet * 20:05 cmooney@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 12355 * 20:04 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 12355 * 19:51 brett@cumin2002: START - Cookbook sre.hosts.reimage for host lvs1016.eqiad.wmnet with OS bullseye * 19:48 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 19:45 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 19:45 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 19:32 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp6016.drmrs.wmnet,cp[1112,1114].eqiad.wmnet,cp[5024,5031-5032].eqsin.wmnet<nowiki>}</nowiki> and A:cp * 19:32 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5032.eqsin.wmnet * 19:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 19:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 19:01 joal@deploy1003: Finished deploy [analytics/refinery@96cf761] (thin): Regular analytics weekly train THIN [analytics/refinery@96cf761f] (duration: 02m 08s) * 18:59 joal@deploy1003: Started deploy [analytics/refinery@96cf761] (thin): Regular analytics weekly train THIN [analytics/refinery@96cf761f] * 18:58 joal@deploy1003: Finished deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] (duration: 05m 01s) * 18:53 joal@deploy1003: Started deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] * 18:53 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294376{{!}}Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375{{!}}auth: Mark the hidden token field used for reauth as skippable (T427398)]] (duration: 07m 41s) * 18:49 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5031.eqsin.wmnet * 18:49 catrope@deploy1003: catrope: Continuing with deployment * 18:47 catrope@deploy1003: catrope: Backport for [[gerrit:1294376{{!}}Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375{{!}}auth: Mark the hidden token field used for reauth as skippable (T427398)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:45 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1294376{{!}}Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375{{!}}auth: Mark the hidden token field used for reauth as skippable (T427398)]] * 18:40 joal@deploy1003: Finished deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] (duration: 01m 05s) * 18:39 joal@deploy1003: Started deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] * 18:37 joal@deploy1003: Finished deploy [analytics/refinery@96cf761] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96cf761f] (duration: 02m 04s) * 18:35 joal@deploy1003: Started deploy [analytics/refinery@96cf761] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96cf761f] * 18:29 swfrench@deploy1003: Finished scap sync-world: Helmfile-only deployment to clean up unused mesh listeners (duration: 06m 12s) * 18:25 swfrench@deploy1003: swfrench: Continuing with deployment * 18:24 swfrench@deploy1003: swfrench: Helmfile-only deployment to clean up unused mesh listeners synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:23 swfrench@deploy1003: Started scap sync-world: Helmfile-only deployment to clean up unused mesh listeners * 18:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93296 and previous config saved to /var/cache/conftool/dbconfig/20260527-181923-fceratto.json * 18:13 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 18:12 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 18:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 18:11 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 18:11 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 18:10 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 18:10 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 18:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93295 and previous config saved to /var/cache/conftool/dbconfig/20260527-180915-fceratto.json * 18:09 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 18:09 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293776{{!}}ProductionServices: Revert to discovery shellbox listeners]] (duration: 10m 24s) * 18:08 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1017.eqiad.wmnet * 18:08 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1017.eqiad.wmnet * 18:07 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5024.eqsin.wmnet * 18:03 swfrench@deploy1003: swfrench: Continuing with deployment * 18:02 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 18:02 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 18:02 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 18:01 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:01 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 18:01 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 18:00 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 18:00 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293776{{!}}ProductionServices: Revert to discovery shellbox listeners]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:00 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93294 and previous config saved to /var/cache/conftool/dbconfig/20260527-175908-fceratto.json * 17:58 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293776{{!}}ProductionServices: Revert to discovery shellbox listeners]] * 17:55 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93293 and previous config saved to /var/cache/conftool/dbconfig/20260527-174900-fceratto.json * 17:43 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293774{{!}}ProductionServices: Temporarily use shellbox in codfw]] (duration: 15m 01s) * 17:38 swfrench@deploy1003: swfrench: Continuing with deployment * 17:31 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293774{{!}}ProductionServices: Temporarily use shellbox in codfw]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:28 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293774{{!}}ProductionServices: Temporarily use shellbox in codfw]] * 17:25 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp1114.eqiad.wmnet * 17:18 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:15 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:15 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:14 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:05 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293775{{!}}ProductionServices: Temporarily use shellbox in eqiad]] (duration: 08m 44s) * 17:00 swfrench@deploy1003: swfrench: Continuing with deployment * 16:58 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293775{{!}}ProductionServices: Temporarily use shellbox in eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:56 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293775{{!}}ProductionServices: Temporarily use shellbox in eqiad]] * 16:53 atsuko@dns1004: END - running authdns-update * 16:51 atsuko@dns1004: START - running authdns-update * 16:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93292 and previous config saved to /var/cache/conftool/dbconfig/20260527-164846-fceratto.json * 16:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1264.eqiad.wmnet with reason: Maintenance * 16:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93291 and previous config saved to /var/cache/conftool/dbconfig/20260527-164815-fceratto.json * 16:43 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp1112.eqiad.wmnet * 16:41 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1017.eqiad.wmnet with reason: Setting up * 16:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P93290 and previous config saved to /var/cache/conftool/dbconfig/20260527-163808-fceratto.json * 16:37 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2163: Repooling after testing patch * 16:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P93287 and previous config saved to /var/cache/conftool/dbconfig/20260527-162800-fceratto.json * 16:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93285 and previous config saved to /var/cache/conftool/dbconfig/20260527-161753-fceratto.json * 16:14 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 16:13 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 16:13 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 16:12 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 16:11 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93284 and previous config saved to /var/cache/conftool/dbconfig/20260527-161101-fceratto.json * 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: Maintenance * 16:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93283 and previous config saved to /var/cache/conftool/dbconfig/20260527-161034-fceratto.json * 16:10 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 16:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1178: Recovering from failure in cookbook * 16:10 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 16:05 sukhe@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host durum5003.eqsin.wmnet with OS trixie * 16:03 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp6016.drmrs.wmnet * 16:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220', diff saved to https://phabricator.wikimedia.org/P93280 and previous config saved to /var/cache/conftool/dbconfig/20260527-160027-fceratto.json * 15:59 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1017.eqiad.wmnet * 15:53 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2163.codfw.wmnet * 15:53 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2163.codfw.wmnet * 15:52 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs1017.eqiad.wmnet * 15:52 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Repooling after testing patch * 15:52 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp6016.drmrs.wmnet,cp[1112,1114].eqiad.wmnet,cp[5024,5031-5032].eqsin.wmnet<nowiki>}</nowiki> and A:cp * 15:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2163: Testing cookbook * 15:50 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2163: Testing cookbook * 15:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220', diff saved to https://phabricator.wikimedia.org/P93276 and previous config saved to /var/cache/conftool/dbconfig/20260527-155019-fceratto.json * 15:45 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:45 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93274 and previous config saved to /var/cache/conftool/dbconfig/20260527-154011-fceratto.json * 15:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 15:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2163: Migration of db2163.codfw.wmnet completed * 15:32 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Migration of db2163.codfw.wmnet completed * 15:32 cwilliams@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2163: Migration of db2163.codfw.wmnet completed * 15:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1178: Recovering from failure in cookbook * 15:22 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1178.eqiad.wmnet * 15:22 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1178.eqiad.wmnet * 15:19 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 15:19 cdanis: 💙cdanis@cp4047.ulsfo.wmnet ~ 🕦☕ sudo apt install lua5.4-ciderbloom lua5.4-ciderbloom-dbgsym * 15:13 cdanis: 💙cdanis@cp5026.eqsin.wmnet ~ 🕚☕ sudo apt install lua5.4-ciderbloom lua5.4-ciderbloom-dbgsym * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:11 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:11 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:11 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Icinga wait failed during run * 15:10 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:10 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:10 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:09 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:09 cdanis: 💔cdanis@apt1002.wikimedia.org ~ 🕚☕ sudo -i reprepro --component main --restrict cidergrinder update trixie-wikimedia * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:05 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93268 and previous config saved to /var/cache/conftool/dbconfig/20260527-150508-fceratto.json * 15:05 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1220.eqiad.wmnet with reason: Maintenance * 15:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93267 and previous config saved to /var/cache/conftool/dbconfig/20260527-150438-fceratto.json * 14:59 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Migration of db2163.codfw.wmnet completed * 14:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P93264 and previous config saved to /var/cache/conftool/dbconfig/20260527-145430-fceratto.json * 14:54 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 14:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2163.codfw.wmnet with OS trixie * 14:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply * 14:50 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply * 14:46 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290926{{!}}Re-enable ReadingLists QuickSurvey (T426781)]] (duration: 08m 32s) * 14:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1178.eqiad.wmnet with OS trixie * 14:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P93263 and previous config saved to /var/cache/conftool/dbconfig/20260527-144423-fceratto.json * 14:42 aude@deploy1003: aude: Continuing with deployment * 14:40 aude@deploy1003: aude: Backport for [[gerrit:1290926{{!}}Re-enable ReadingLists QuickSurvey (T426781)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:38 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db2189.codfw.wmnet with reason: crashed [[phab:T427376|T427376]] * 14:38 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1290926{{!}}Re-enable ReadingLists QuickSurvey (T426781)]] * 14:35 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290924{{!}}Make logging of title and page ID optional (T426457)]] (duration: 11m 30s) * 14:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93262 and previous config saved to /var/cache/conftool/dbconfig/20260527-143416-fceratto.json * 14:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2163.codfw.wmnet with reason: host reimage * 14:29 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2163.codfw.wmnet with reason: host reimage * 14:29 aude@deploy1003: aude: Continuing with deployment * 14:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1178.eqiad.wmnet with reason: host reimage * 14:27 aude@deploy1003: aude: Backport for [[gerrit:1290924{{!}}Make logging of title and page ID optional (T426457)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:27 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93260 and previous config saved to /var/cache/conftool/dbconfig/20260527-142659-fceratto.json * 14:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1179.eqiad.wmnet with reason: Maintenance * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:23 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1290924{{!}}Make logging of title and page ID optional (T426457)]] * 14:22 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1178.eqiad.wmnet with reason: host reimage * 14:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1033.eqiad.wmnet with reason: Maintenance * 14:18 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294247{{!}}Update Direct Reporting email (T427358)]] (duration: 33m 01s) * 14:10 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2163.codfw.wmnet with OS trixie * 14:09 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1178.eqiad.wmnet with OS trixie * 14:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2163: Upgrading db2163.codfw.wmnet * 14:08 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2163: Upgrading db2163.codfw.wmnet * 14:08 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1178: Upgrading db1178.eqiad.wmnet * 14:07 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1178: Upgrading db1178.eqiad.wmnet * 14:06 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:06 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:06 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:06 stran@deploy1003: stran: Continuing with deployment * 14:02 stran@deploy1003: stran: Backport for [[gerrit:1294247{{!}}Update Direct Reporting email (T427358)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:56 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 13:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2164: Migration of db2164.codfw.wmnet completed * 13:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1192: Migration of db1192.eqiad.wmnet completed * 13:45 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1294247{{!}}Update Direct Reporting email (T427358)]] * 13:40 phuedx@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294217{{!}}ext.wikimediaEvents: Add hoisting error detection test (T427092)]] (duration: 11m 35s) * 13:36 phuedx@deploy1003: phuedx: Continuing with deployment * 13:30 phuedx@deploy1003: phuedx: Backport for [[gerrit:1294217{{!}}ext.wikimediaEvents: Add hoisting error detection test (T427092)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:28 phuedx@deploy1003: Started scap sync-world: Backport for [[gerrit:1294217{{!}}ext.wikimediaEvents: Add hoisting error detection test (T427092)]] * 13:21 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290781{{!}}mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264{{!}}MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] (duration: 13m 23s) * 13:15 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2189: Test * 13:15 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2189: Test * 13:15 mlitn@deploy1003: krinkle, mlitn: Continuing with deployment * 13:13 mlitn@deploy1003: krinkle, mlitn: Backport for [[gerrit:1290781{{!}}mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264{{!}}MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 13:10 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2164: Migration of db2164.codfw.wmnet completed * 13:08 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1290781{{!}}mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264{{!}}MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] * 13:06 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 13:05 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db2212.codfw.wmnet with reason: failed to reboot [[phab:T427388|T427388]] [[phab:T426633|T426633]] * 13:05 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1192: Migration of db1192.eqiad.wmnet completed * 13:01 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2164.codfw.wmnet with OS trixie * 12:57 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1192.eqiad.wmnet with OS trixie * 12:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2164.codfw.wmnet with reason: host reimage * 12:40 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1192.eqiad.wmnet with reason: host reimage * 12:40 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2164.codfw.wmnet with reason: host reimage * 12:35 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1192.eqiad.wmnet with reason: host reimage * 12:28 Amir1: deleting binlogs older than a year * 12:22 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2164.codfw.wmnet with OS trixie * 12:21 cmooney@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 36692 * 12:21 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1192.eqiad.wmnet with OS trixie * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1077 * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1080 * 12:20 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1077 * 12:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2164: Upgrading db2164.codfw.wmnet * 12:20 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 36692 * 12:20 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1080 * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1078 * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1079 * 12:20 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2164: Upgrading db2164.codfw.wmnet * 12:19 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:19 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1079 * 12:19 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1078 * 12:19 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:19 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1192: Upgrading db1192.eqiad.wmnet * 12:19 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:18 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1192: Upgrading db1192.eqiad.wmnet * 12:18 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:15 jclark@cumin1003: START - Cookbook sre.dns.netbox * 12:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2165: Migration of db2165.codfw.wmnet completed * 12:14 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:14 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:14 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:12 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db2189: Test * 12:11 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2189: Test * 12:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1193: Migration of db1193.eqiad.wmnet completed * 12:09 jclark@cumin1003: START - Cookbook sre.dns.netbox * 12:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93243 and previous config saved to /var/cache/conftool/dbconfig/20260527-120452-fceratto.json * 12:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: Maintenance * 12:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93242 and previous config saved to /var/cache/conftool/dbconfig/20260527-120205-fceratto.json * 12:01 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 11:58 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 11:58 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "is everything alright? /cc effie - ayounsi@cumin1003" * 11:58 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "is everything alright? /cc effie - ayounsi@cumin1003" * 11:56 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 11:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P93239 and previous config saved to /var/cache/conftool/dbconfig/20260527-115157-fceratto.json * 11:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P93237 and previous config saved to /var/cache/conftool/dbconfig/20260527-114149-fceratto.json * 11:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93235 and previous config saved to /var/cache/conftool/dbconfig/20260527-113142-fceratto.json * 11:29 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2165: Migration of db2165.codfw.wmnet completed * 11:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1193: Migration of db1193.eqiad.wmnet completed * 11:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93231 and previous config saved to /var/cache/conftool/dbconfig/20260527-112327-fceratto.json * 11:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2188.codfw.wmnet with reason: Maintenance * 11:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93230 and previous config saved to /var/cache/conftool/dbconfig/20260527-112257-fceratto.json * 11:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2165.codfw.wmnet with OS trixie * 11:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1193.eqiad.wmnet with OS trixie * 11:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P93229 and previous config saved to /var/cache/conftool/dbconfig/20260527-111250-fceratto.json * 11:10 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:10 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:08 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:08 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:02 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P93227 and previous config saved to /var/cache/conftool/dbconfig/20260527-110242-fceratto.json * 11:02 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:02 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 11:01 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 11:01 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2165.codfw.wmnet with reason: host reimage * 11:00 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2189', diff saved to https://phabricator.wikimedia.org/P93226 and previous config saved to /var/cache/conftool/dbconfig/20260527-110016-marostegui.json * 10:58 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1193.eqiad.wmnet with reason: host reimage * 10:57 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2165.codfw.wmnet with reason: host reimage * 10:56 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 10:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93225 and previous config saved to /var/cache/conftool/dbconfig/20260527-105235-fceratto.json * 10:52 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1193.eqiad.wmnet with reason: host reimage * 10:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1050: repool after maintenance * 10:45 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2176 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93223 and previous config saved to /var/cache/conftool/dbconfig/20260527-104518-fceratto.json * 10:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2176.codfw.wmnet with reason: Maintenance * 10:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93222 and previous config saved to /var/cache/conftool/dbconfig/20260527-104449-fceratto.json * 10:39 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2165.codfw.wmnet with OS trixie * 10:38 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1193.eqiad.wmnet with OS trixie * 10:36 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1193: Upgrading db1193.eqiad.wmnet * 10:35 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1193: Upgrading db1193.eqiad.wmnet * 10:35 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2165: Upgrading db2165.codfw.wmnet * 10:35 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2165: Upgrading db2165.codfw.wmnet * 10:34 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P93218 and previous config saved to /var/cache/conftool/dbconfig/20260527-103441-fceratto.json * 10:29 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:29 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P93217 and previous config saved to /var/cache/conftool/dbconfig/20260527-102434-fceratto.json * 10:22 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:21 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93215 and previous config saved to /var/cache/conftool/dbconfig/20260527-101426-fceratto.json * 10:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1203: Migration of db1203.eqiad.wmnet completed * 10:10 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2166: Migration of db2166.codfw.wmnet completed * 10:08 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93212 and previous config saved to /var/cache/conftool/dbconfig/20260527-100701-fceratto.json * 10:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance * 10:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93211 and previous config saved to /var/cache/conftool/dbconfig/20260527-100632-fceratto.json * 10:05 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1050: repool after maintenance * 10:04 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:02 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1050.eqiad.wmnet with OS trixie * 09:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P93208 and previous config saved to /var/cache/conftool/dbconfig/20260527-095624-fceratto.json * 09:47 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 09:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P93206 and previous config saved to /var/cache/conftool/dbconfig/20260527-094616-fceratto.json * 09:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1050.eqiad.wmnet with reason: host reimage * 09:43 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 09:41 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1050.eqiad.wmnet with reason: host reimage * 09:38 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 09:38 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 09:37 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 09:37 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 09:36 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 09:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93203 and previous config saved to /var/cache/conftool/dbconfig/20260527-093609-fceratto.json * 09:34 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 09:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93202 and previous config saved to /var/cache/conftool/dbconfig/20260527-092842-fceratto.json * 09:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance * 09:28 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1203: Migration of db1203.eqiad.wmnet completed * 09:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93200 and previous config saved to /var/cache/conftool/dbconfig/20260527-092814-fceratto.json * 09:27 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1050.eqiad.wmnet with OS trixie * 09:26 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1050: Upgrading es1050.eqiad.wmnet * 09:25 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1050: Upgrading es1050.eqiad.wmnet * 09:25 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1050: repool after maintenance * 09:25 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1050: repool after maintenance * 09:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2166: Migration of db2166.codfw.wmnet completed * 09:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2051: repool after maintenance * 09:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1203.eqiad.wmnet with OS trixie * 09:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P93196 and previous config saved to /var/cache/conftool/dbconfig/20260527-091806-fceratto.json * 09:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2166.codfw.wmnet with OS trixie * 09:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P93194 and previous config saved to /var/cache/conftool/dbconfig/20260527-090759-fceratto.json * 09:03 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp3074.* * 09:03 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp3066.* * 09:03 fabfur: repooling cp3074 and cp3066 ([[phab:T419825|T419825]]) * 09:02 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp6015.drmrs.wmnet * 09:02 slyngshede@cumin1003: START - Cookbook sre.hosts.remove-downtime for cp6015.drmrs.wmnet * 09:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1203.eqiad.wmnet with reason: host reimage * 09:02 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: name=cp6015.* * 08:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2166.codfw.wmnet with reason: host reimage * 08:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93193 and previous config saved to /var/cache/conftool/dbconfig/20260527-085751-fceratto.json * 08:55 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1203.eqiad.wmnet with reason: host reimage * 08:54 Emperor: restart swift on ms-fe2011 [[phab:T360913|T360913]] * 08:54 jayme@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:54 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2166.codfw.wmnet with reason: host reimage * 08:54 jayme@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 08:51 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 08:51 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 08:51 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp3066.* * 08:51 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp3074.* * 08:51 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 08:50 fabfur: depooling and installing haproxy-awslc on cp3074 and cp3066 ([[phab:T419825|T419825]]) * 08:50 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93191 and previous config saved to /var/cache/conftool/dbconfig/20260527-085024-fceratto.json * 08:50 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance * 08:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93190 and previous config saved to /var/cache/conftool/dbconfig/20260527-085005-fceratto.json * 08:41 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1203.eqiad.wmnet with OS trixie * 08:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P93189 and previous config saved to /var/cache/conftool/dbconfig/20260527-083957-fceratto.json * 08:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2051: repool after maintenance * 08:37 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 08:36 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1203: Upgrading db1203.eqiad.wmnet * 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1004.wikimedia.org * 08:36 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1203: Upgrading db1203.eqiad.wmnet * 08:36 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:35 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2166.codfw.wmnet with OS trixie * 08:35 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2051.codfw.wmnet with OS trixie * 08:34 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2166: Upgrading db2166.codfw.wmnet * 08:33 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2166: Upgrading db2166.codfw.wmnet * 08:33 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader1004.wikimedia.org * 08:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org * 08:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P93185 and previous config saved to /var/cache/conftool/dbconfig/20260527-082950-fceratto.json * 08:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org * 08:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93184 and previous config saved to /var/cache/conftool/dbconfig/20260527-081942-fceratto.json * 08:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2051.codfw.wmnet with reason: host reimage * 08:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2051.codfw.wmnet with reason: host reimage * 08:11 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 08:11 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2153 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93183 and previous config saved to /var/cache/conftool/dbconfig/20260527-081112-fceratto.json * 08:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance * 08:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93182 and previous config saved to /var/cache/conftool/dbconfig/20260527-081054-fceratto.json * 08:07 jmm@dns1004: END - running authdns-update * 08:05 jmm@dns1004: START - running authdns-update * 08:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248', diff saved to https://phabricator.wikimedia.org/P93181 and previous config saved to /var/cache/conftool/dbconfig/20260527-080046-fceratto.json * 07:59 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2051.codfw.wmnet with OS trixie * 07:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248', diff saved to https://phabricator.wikimedia.org/P93180 and previous config saved to /var/cache/conftool/dbconfig/20260527-075039-fceratto.json * 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1026.eqiad.wmnet * 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1026.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:43 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1026.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2051: Upgrading es2051.codfw.wmnet * 07:42 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2051: Upgrading es2051.codfw.wmnet * 07:41 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93178 and previous config saved to /var/cache/conftool/dbconfig/20260527-074031-fceratto.json * 07:40 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294125{{!}}Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126{{!}}Add script to demote ineligible members of restricted global groups (T425395)]] (duration: 06m 42s) * 07:36 mszwarc@deploy1003: mszwarc: Continuing with deployment * 07:35 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1294125{{!}}Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126{{!}}Add script to demote ineligible members of restricted global groups (T425395)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93177 and previous config saved to /var/cache/conftool/dbconfig/20260527-073504-fceratto.json * 07:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2248.codfw.wmnet with reason: Maintenance * 07:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93176 and previous config saved to /var/cache/conftool/dbconfig/20260527-073434-fceratto.json * 07:33 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1294125{{!}}Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126{{!}}Add script to demote ineligible members of restricted global groups (T425395)]] * 07:28 jmm@cumin2002: START - Cookbook sre.dns.netbox * 07:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P93175 and previous config saved to /var/cache/conftool/dbconfig/20260527-072426-fceratto.json * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) * 07:23 marostegui@cumin1003: Removing pc1014 from zarcillo [[phab:T427190|T427190]] * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1014.eqiad.wmnet * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 07:23 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 07:18 marostegui@cumin1003: START - Cookbook sre.dns.netbox * 07:15 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1026.eqiad.wmnet * 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1025.eqiad.wmnet * 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P93174 and previous config saved to /var/cache/conftool/dbconfig/20260527-071418-fceratto.json * 07:13 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1014.eqiad.wmnet * 07:13 marostegui@cumin1003: START - Cookbook sre.mysql.decommission * 07:13 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org * 07:07 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2055: repool after maintenance * 07:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org * 07:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1003.wikimedia.org * 07:06 jmm@cumin2002: START - Cookbook sre.dns.netbox * 07:06 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1190.eqiad.wmnet with reason: Maintenance on db1190 * 07:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93172 and previous config saved to /var/cache/conftool/dbconfig/20260527-070410-fceratto.json * 07:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader1003.wikimedia.org * 06:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93171 and previous config saved to /var/cache/conftool/dbconfig/20260527-065545-fceratto.json * 06:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2247.codfw.wmnet with reason: Maintenance * 06:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93170 and previous config saved to /var/cache/conftool/dbconfig/20260527-065526-fceratto.json * 06:54 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1025.eqiad.wmnet * 06:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P93168 and previous config saved to /var/cache/conftool/dbconfig/20260527-064519-fceratto.json * 06:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P93166 and previous config saved to /var/cache/conftool/dbconfig/20260527-063511-fceratto.json * 06:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93165 and previous config saved to /var/cache/conftool/dbconfig/20260527-062503-fceratto.json * 06:22 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2055: repool after maintenance * 06:21 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 06:21 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2055.codfw.wmnet with OS trixie * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2246 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93163 and previous config saved to /var/cache/conftool/dbconfig/20260527-061643-fceratto.json * 06:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2246.codfw.wmnet with reason: Maintenance * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93162 and previous config saved to /var/cache/conftool/dbconfig/20260527-061613-fceratto.json * 06:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P93161 and previous config saved to /var/cache/conftool/dbconfig/20260527-060606-fceratto.json * 06:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2055.codfw.wmnet with reason: host reimage * 05:56 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2055.codfw.wmnet with reason: host reimage * 05:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P93160 and previous config saved to /var/cache/conftool/dbconfig/20260527-055558-fceratto.json * 05:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93159 and previous config saved to /var/cache/conftool/dbconfig/20260527-054550-fceratto.json * 05:41 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2055.codfw.wmnet with OS trixie * 05:40 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2055: Upgrading es2055.codfw.wmnet * 05:40 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2055: Upgrading es2055.codfw.wmnet * 05:40 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:38 moritzm: remove ganeti1026 from eqiad Ganeti cluster [[phab:T424680|T424680]] * 05:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2245 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93157 and previous config saved to /var/cache/conftool/dbconfig/20260527-053727-fceratto.json * 05:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: Maintenance * 05:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93156 and previous config saved to /var/cache/conftool/dbconfig/20260527-053708-fceratto.json * 05:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P93155 and previous config saved to /var/cache/conftool/dbconfig/20260527-052700-fceratto.json * 05:26 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1014 from dbctl [[phab:T427270|T427270]]', diff saved to https://phabricator.wikimedia.org/P93154 and previous config saved to /var/cache/conftool/dbconfig/20260527-052624-marostegui.json * 05:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P93153 and previous config saved to /var/cache/conftool/dbconfig/20260527-051653-fceratto.json * 05:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93152 and previous config saved to /var/cache/conftool/dbconfig/20260527-050645-fceratto.json * 04:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2237 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93151 and previous config saved to /var/cache/conftool/dbconfig/20260527-045827-fceratto.json * 04:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: Maintenance * 04:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93150 and previous config saved to /var/cache/conftool/dbconfig/20260527-045759-fceratto.json * 04:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P93149 and previous config saved to /var/cache/conftool/dbconfig/20260527-044751-fceratto.json * 04:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P93148 and previous config saved to /var/cache/conftool/dbconfig/20260527-043744-fceratto.json * 04:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93147 and previous config saved to /var/cache/conftool/dbconfig/20260527-042737-fceratto.json * 04:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2236 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93146 and previous config saved to /var/cache/conftool/dbconfig/20260527-041921-fceratto.json * 04:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2236.codfw.wmnet with reason: Maintenance * 04:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93145 and previous config saved to /var/cache/conftool/dbconfig/20260527-041852-fceratto.json * 04:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P93144 and previous config saved to /var/cache/conftool/dbconfig/20260527-040844-fceratto.json * 03:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P93143 and previous config saved to /var/cache/conftool/dbconfig/20260527-035836-fceratto.json * 03:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93142 and previous config saved to /var/cache/conftool/dbconfig/20260527-034828-fceratto.json * 03:40 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93141 and previous config saved to /var/cache/conftool/dbconfig/20260527-034008-fceratto.json * 03:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2219.codfw.wmnet with reason: Maintenance * 03:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93140 and previous config saved to /var/cache/conftool/dbconfig/20260527-033938-fceratto.json * 03:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P93139 and previous config saved to /var/cache/conftool/dbconfig/20260527-032931-fceratto.json * 03:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P93138 and previous config saved to /var/cache/conftool/dbconfig/20260527-031923-fceratto.json * 03:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93137 and previous config saved to /var/cache/conftool/dbconfig/20260527-030915-fceratto.json * 03:00 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93136 and previous config saved to /var/cache/conftool/dbconfig/20260527-030045-fceratto.json * 03:00 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2210.codfw.wmnet with reason: Maintenance * 03:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93135 and previous config saved to /var/cache/conftool/dbconfig/20260527-030016-fceratto.json * 02:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P93134 and previous config saved to /var/cache/conftool/dbconfig/20260527-025008-fceratto.json * 02:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P93133 and previous config saved to /var/cache/conftool/dbconfig/20260527-024000-fceratto.json * 02:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93132 and previous config saved to /var/cache/conftool/dbconfig/20260527-022953-fceratto.json * 02:21 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93131 and previous config saved to /var/cache/conftool/dbconfig/20260527-022133-fceratto.json * 02:21 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2206.codfw.wmnet with reason: Maintenance * 02:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93130 and previous config saved to /var/cache/conftool/dbconfig/20260527-022100-fceratto.json * 02:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P93129 and previous config saved to /var/cache/conftool/dbconfig/20260527-021053-fceratto.json * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 29s) * 02:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P93128 and previous config saved to /var/cache/conftool/dbconfig/20260527-020045-fceratto.json * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93127 and previous config saved to /var/cache/conftool/dbconfig/20260527-015037-fceratto.json * 01:42 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93126 and previous config saved to /var/cache/conftool/dbconfig/20260527-014204-fceratto.json * 01:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2179.codfw.wmnet with reason: Maintenance * 01:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93125 and previous config saved to /var/cache/conftool/dbconfig/20260527-014134-fceratto.json * 01:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P93124 and previous config saved to /var/cache/conftool/dbconfig/20260527-013126-fceratto.json * 01:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P93123 and previous config saved to /var/cache/conftool/dbconfig/20260527-012119-fceratto.json * 01:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93122 and previous config saved to /var/cache/conftool/dbconfig/20260527-011111-fceratto.json * 01:02 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2172 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93121 and previous config saved to /var/cache/conftool/dbconfig/20260527-010234-fceratto.json * 01:02 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance * 01:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93120 and previous config saved to /var/cache/conftool/dbconfig/20260527-010205-fceratto.json * 00:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P93119 and previous config saved to /var/cache/conftool/dbconfig/20260527-005157-fceratto.json * 00:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P93118 and previous config saved to /var/cache/conftool/dbconfig/20260527-004149-fceratto.json * 00:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93117 and previous config saved to /var/cache/conftool/dbconfig/20260527-003141-fceratto.json * 00:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2155 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93116 and previous config saved to /var/cache/conftool/dbconfig/20260527-002309-fceratto.json * 00:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance * 00:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93115 and previous config saved to /var/cache/conftool/dbconfig/20260527-002228-fceratto.json * 00:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P93114 and previous config saved to /var/cache/conftool/dbconfig/20260527-001220-fceratto.json * 00:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P93113 and previous config saved to /var/cache/conftool/dbconfig/20260527-000209-fceratto.json == 2026-05-26 == * 23:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93112 and previous config saved to /var/cache/conftool/dbconfig/20260526-235201-fceratto.json * 23:44 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93111 and previous config saved to /var/cache/conftool/dbconfig/20260526-234451-fceratto.json * 23:44 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance * 23:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93110 and previous config saved to /var/cache/conftool/dbconfig/20260526-234421-fceratto.json * 23:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P93109 and previous config saved to /var/cache/conftool/dbconfig/20260526-233414-fceratto.json * 23:27 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5026.* * 23:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P93108 and previous config saved to /var/cache/conftool/dbconfig/20260526-232406-fceratto.json * 23:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93107 and previous config saved to /var/cache/conftool/dbconfig/20260526-231358-fceratto.json * 23:07 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5026.* * 23:06 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93106 and previous config saved to /var/cache/conftool/dbconfig/20260526-230650-fceratto.json * 23:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance * 23:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93105 and previous config saved to /var/cache/conftool/dbconfig/20260526-230620-fceratto.json * 22:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P93104 and previous config saved to /var/cache/conftool/dbconfig/20260526-225612-fceratto.json * 22:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P93103 and previous config saved to /var/cache/conftool/dbconfig/20260526-224604-fceratto.json * 22:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93101 and previous config saved to /var/cache/conftool/dbconfig/20260526-223556-fceratto.json * 22:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93100 and previous config saved to /var/cache/conftool/dbconfig/20260526-222848-fceratto.json * 22:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance * 22:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93099 and previous config saved to /var/cache/conftool/dbconfig/20260526-222828-fceratto.json * 22:23 robh@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts cp6015.drmrs.wmnet * 22:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P93098 and previous config saved to /var/cache/conftool/dbconfig/20260526-221819-fceratto.json * 22:10 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1009.eqiad.wmnet with OS trixie * 22:08 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1008.eqiad.wmnet with OS trixie * 22:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P93097 and previous config saved to /var/cache/conftool/dbconfig/20260526-220811-fceratto.json * 22:04 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293701{{!}}MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] (duration: 09m 30s) * 22:03 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1009.eqiad.wmnet with reason: host reimage * 22:00 egardner@deploy1003: egardner, mfossati: Continuing with deployment * 21:59 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1008.eqiad.wmnet with reason: host reimage * 21:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93096 and previous config saved to /var/cache/conftool/dbconfig/20260526-215803-fceratto.json * 21:57 egardner@deploy1003: egardner, mfossati: Backport for [[gerrit:1293701{{!}}MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:56 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp6015.drmrs.wmnet * 21:56 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1010.eqiad.wmnet with OS trixie * 21:56 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp6015.drmrs.wmnet * 21:55 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1293701{{!}}MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] * 21:54 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1009.eqiad.wmnet with reason: host reimage * 21:51 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1008.eqiad.wmnet with reason: host reimage * 21:50 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93095 and previous config saved to /var/cache/conftool/dbconfig/20260526-215043-fceratto.json * 21:50 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance * 21:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93094 and previous config saved to /var/cache/conftool/dbconfig/20260526-215011-fceratto.json * 21:49 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1010.eqiad.wmnet with reason: host reimage * 21:47 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp6015.drmrs.wmnet * 21:44 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1009 * 21:44 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host relforge1009 * 21:43 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host relforge1009 * 21:43 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) relforge1009.eqiad.wmnet 120.48.64.10.in-addr.arpa 0.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:43 bking@cumin2002: START - Cookbook sre.dns.wipe-cache relforge1009.eqiad.wmnet 120.48.64.10.in-addr.arpa 0.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:43 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:42 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1009 - bking@cumin2002" * 21:42 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1010.eqiad.wmnet with reason: host reimage * 21:42 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1009 - bking@cumin2002" * 21:41 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1008 * 21:40 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host relforge1008 * 21:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P93093 and previous config saved to /var/cache/conftool/dbconfig/20260526-214003-fceratto.json * 21:36 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host relforge1008 * 21:36 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) relforge1008.eqiad.wmnet 100.32.64.10.in-addr.arpa 0.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:36 bking@cumin2002: START - Cookbook sre.dns.wipe-cache relforge1008.eqiad.wmnet 100.32.64.10.in-addr.arpa 0.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:36 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:36 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1008 - bking@cumin2002" * 21:36 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1008 - bking@cumin2002" * 21:35 bking@cumin2002: START - Cookbook sre.dns.netbox * 21:32 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1010 * 21:32 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1010 * 21:31 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1010.eqiad.wmnet with OS trixie * 21:31 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1009 * 21:30 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1009.eqiad.wmnet with OS trixie * 21:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P93092 and previous config saved to /var/cache/conftool/dbconfig/20260526-212955-fceratto.json * 21:29 bking@cumin2002: START - Cookbook sre.dns.netbox * 21:29 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1008 * 21:29 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1008.eqiad.wmnet with OS trixie * 21:27 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist "all.dblist - mediamoderation-continuous-scan.dblist - preinstall.dblist" extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` in tmux session - [[phab:T421688|T421688]] * 21:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93091 and previous config saved to /var/cache/conftool/dbconfig/20260526-211948-fceratto.json * 21:19 jhathaway: dmarc ingress test run mx-in1001 * 21:15 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_codfw and A:cp * 21:15 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2057.codfw.wmnet * 21:14 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_codfw and A:cp * 21:14 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2058.codfw.wmnet * 21:12 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2222 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93090 and previous config saved to /var/cache/conftool/dbconfig/20260526-211238-fceratto.json * 21:12 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance * 21:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93089 and previous config saved to /var/cache/conftool/dbconfig/20260526-211207-fceratto.json * 21:06 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 21:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P93088 and previous config saved to /var/cache/conftool/dbconfig/20260526-210159-fceratto.json * 20:55 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on phab2003.codfw.wmnet with reason: WIP * 20:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P93087 and previous config saved to /var/cache/conftool/dbconfig/20260526-205152-fceratto.json * 20:50 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:50 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 20:50 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 20:45 dzahn@cumin2002: START - Cookbook sre.dns.netbox * 20:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93086 and previous config saved to /var/cache/conftool/dbconfig/20260526-204143-fceratto.json * 20:38 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2055.codfw.wmnet * 20:34 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93085 and previous config saved to /var/cache/conftool/dbconfig/20260526-203430-fceratto.json * 20:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance * 20:34 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2056.codfw.wmnet * 20:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93084 and previous config saved to /var/cache/conftool/dbconfig/20260526-203357-fceratto.json * 20:32 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:32 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:32 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:31 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P93083 and previous config saved to /var/cache/conftool/dbconfig/20260526-202349-fceratto.json * 20:18 alexsanford@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293161{{!}}Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794{{!}}Re-enable ReadingLists survey on beta cluster (T426781)]] (duration: 09m 14s) * 20:14 alexsanford@deploy1003: alexsanford, aude: Continuing with deployment * 20:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P93082 and previous config saved to /var/cache/conftool/dbconfig/20260526-201341-fceratto.json * 20:11 alexsanford@deploy1003: alexsanford, aude: Backport for [[gerrit:1293161{{!}}Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794{{!}}Re-enable ReadingLists survey on beta cluster (T426781)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:09 alexsanford@deploy1003: Started scap sync-world: Backport for [[gerrit:1293161{{!}}Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794{{!}}Re-enable ReadingLists survey on beta cluster (T426781)]] * 20:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93081 and previous config saved to /var/cache/conftool/dbconfig/20260526-200333-fceratto.json * 19:59 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2053.codfw.wmnet * 19:58 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2029.codfw.wmnet with OS trixie * 19:57 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2028.codfw.wmnet with OS trixie * 19:56 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2208 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93080 and previous config saved to /var/cache/conftool/dbconfig/20260526-195632-fceratto.json * 19:56 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance * 19:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93079 and previous config saved to /var/cache/conftool/dbconfig/20260526-195557-fceratto.json * 19:55 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2054.codfw.wmnet * 19:51 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:51 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P93078 and previous config saved to /var/cache/conftool/dbconfig/20260526-194549-fceratto.json * 19:45 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 19:44 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:43 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2029 * 19:43 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 * 19:43 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 * 19:43 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 19:40 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb2014.codfw.wmnet with OS trixie * 19:40 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:40 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb2013.codfw.wmnet with OS trixie * 19:40 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:39 brett@cumin2002: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 19:38 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 19:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P93077 and previous config saved to /var/cache/conftool/dbconfig/20260526-193541-fceratto.json * 19:35 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:35 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 19:30 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 19:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93076 and previous config saved to /var/cache/conftool/dbconfig/20260526-192533-fceratto.json * 19:24 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:21 dzahn@cumin2002: START - Cookbook sre.dns.netbox * 19:20 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2051.codfw.wmnet * 19:19 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:19 brett@cumin2002: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 19:18 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93075 and previous config saved to /var/cache/conftool/dbconfig/20260526-191818-fceratto.json * 19:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance * 19:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93074 and previous config saved to /var/cache/conftool/dbconfig/20260526-191748-fceratto.json * 19:16 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2052.codfw.wmnet * 19:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P93073 and previous config saved to /var/cache/conftool/dbconfig/20260526-190740-fceratto.json * 19:07 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb2014.codfw.wmnet with reason: host reimage * 19:03 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb2013.codfw.wmnet with reason: host reimage * 18:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1026.eqiad.wmnet * 18:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P93072 and previous config saved to /var/cache/conftool/dbconfig/20260526-185732-fceratto.json * 18:56 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2014.codfw.wmnet with reason: host reimage * 18:56 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2013.codfw.wmnet with reason: host reimage * 18:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93071 and previous config saved to /var/cache/conftool/dbconfig/20260526-184724-fceratto.json * 18:44 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb2014.codfw.wmnet with OS trixie * 18:43 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb2013.codfw.wmnet with OS trixie * 18:41 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host rdb2014.codfw.wmnet with OS trixie * 18:41 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2049.codfw.wmnet * 18:40 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93070 and previous config saved to /var/cache/conftool/dbconfig/20260526-184009-fceratto.json * 18:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance * 18:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93069 and previous config saved to /var/cache/conftool/dbconfig/20260526-183939-fceratto.json * 18:37 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2050.codfw.wmnet * 18:30 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 18:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P93068 and previous config saved to /var/cache/conftool/dbconfig/20260526-182931-fceratto.json * 18:29 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:29 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_magru-v4 - dzahn@cumin2002" * 18:29 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_magru-v4 - dzahn@cumin2002" * 18:24 dzahn@cumin2002: START - Cookbook sre.dns.netbox * 18:21 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 18:21 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 18:21 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 18:20 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 18:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P93066 and previous config saved to /var/cache/conftool/dbconfig/20260526-181923-fceratto.json * 18:15 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 18:15 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 18:15 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 18:15 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 18:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93065 and previous config saved to /var/cache/conftool/dbconfig/20260526-180915-fceratto.json * 18:02 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93064 and previous config saved to /var/cache/conftool/dbconfig/20260526-180205-fceratto.json * 18:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2159.codfw.wmnet with reason: Maintenance * 18:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93063 and previous config saved to /var/cache/conftool/dbconfig/20260526-180132-fceratto.json * 18:00 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2047.codfw.wmnet * 17:59 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2048.codfw.wmnet * 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:54 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:54 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:54 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P93062 and previous config saved to /var/cache/conftool/dbconfig/20260526-175124-fceratto.json * 17:42 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293779{{!}}Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] (duration: 07m 25s) * 17:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P93060 and previous config saved to /var/cache/conftool/dbconfig/20260526-174117-fceratto.json * 17:39 mvernon@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ms-be2089.codfw.wmnet * 17:37 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 17:37 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:36 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:36 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:36 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1293779{{!}}Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:36 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:34 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1293779{{!}}Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] * 17:33 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:33 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:33 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:33 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93059 and previous config saved to /var/cache/conftool/dbconfig/20260526-173109-fceratto.json * 17:27 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:26 jclark@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:25 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:25 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:25 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:24 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:24 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1001 to eqiad - jclark@cumin1003" * 17:24 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:24 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1001 to eqiad - jclark@cumin1003" * 17:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93058 and previous config saved to /var/cache/conftool/dbconfig/20260526-172332-fceratto.json * 17:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2227.codfw.wmnet with reason: Maintenance * 17:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93057 and previous config saved to /var/cache/conftool/dbconfig/20260526-172303-fceratto.json * 17:21 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2045.codfw.wmnet * 17:20 jclark@cumin1003: START - Cookbook sre.dns.netbox * 17:20 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2046.codfw.wmnet * 17:18 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:17 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:17 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:17 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:17 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:17 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:17 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:17 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:16 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:15 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 17:14 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:14 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:14 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:14 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:13 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:13 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:13 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:13 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:13 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P93056 and previous config saved to /var/cache/conftool/dbconfig/20260526-171255-fceratto.json * 17:11 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:11 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:11 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:11 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:07 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:05 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:05 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:05 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:05 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P93055 and previous config saved to /var/cache/conftool/dbconfig/20260526-170247-fceratto.json * 17:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:57 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:55 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:52 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93054 and previous config saved to /var/cache/conftool/dbconfig/20260526-165240-fceratto.json * 16:50 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:50 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:50 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:50 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:45 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:45 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:45 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:45 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:45 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:44 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:44 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2209 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93053 and previous config saved to /var/cache/conftool/dbconfig/20260526-164421-fceratto.json * 16:44 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:44 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1002 to eqiad - jclark@cumin1003" * 16:44 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2209.codfw.wmnet with reason: Maintenance * 16:44 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1002 to eqiad - jclark@cumin1003" * 16:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93052 and previous config saved to /var/cache/conftool/dbconfig/20260526-164352-fceratto.json * 16:42 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2043.codfw.wmnet * 16:41 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2044.codfw.wmnet * 16:40 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:40 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:40 jclark@cumin1003: START - Cookbook sre.dns.netbox * 16:40 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:40 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:40 brett: reboot lvs 101[345].eqiad.wmnet * 16:39 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:39 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:39 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:39 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:37 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:37 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 16:37 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:37 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:37 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:37 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 16:37 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 16:36 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 16:36 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:36 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:36 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:36 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:35 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 16:34 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 16:34 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:34 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:34 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:34 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:33 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_codfw and A:cp * 16:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P93051 and previous config saved to /var/cache/conftool/dbconfig/20260526-163344-fceratto.json * 16:33 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_codfw and A:cp * 16:31 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:31 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:30 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:30 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P93050 and previous config saved to /var/cache/conftool/dbconfig/20260526-162336-fceratto.json * 16:13 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2089.codfw.wmnet * 16:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93049 and previous config saved to /var/cache/conftool/dbconfig/20260526-161328-fceratto.json * 16:11 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:11 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:10 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:10 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:07 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search,name=eqiad * 16:06 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:06 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:06 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:06 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93047 and previous config saved to /var/cache/conftool/dbconfig/20260526-160450-fceratto.json * 16:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2194.codfw.wmnet with reason: Maintenance * 16:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93046 and previous config saved to /var/cache/conftool/dbconfig/20260526-160420-fceratto.json * 16:03 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:03 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:03 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] (duration: 00m 28s) * 16:02 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] * 16:00 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:00 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:00 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:00 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:55 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:55 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:55 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:55 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] (duration: 00m 22s) * 15:55 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:55 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] * 15:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P93045 and previous config saved to /var/cache/conftool/dbconfig/20260526-155413-fceratto.json * 15:46 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search,name=eqiad * 15:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P93044 and previous config saved to /var/cache/conftool/dbconfig/20260526-154405-fceratto.json * 15:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93043 and previous config saved to /var/cache/conftool/dbconfig/20260526-153357-fceratto.json * 15:30 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:30 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:30 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:30 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93042 and previous config saved to /var/cache/conftool/dbconfig/20260526-152629-fceratto.json * 15:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2190.codfw.wmnet with reason: Maintenance * 15:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93041 and previous config saved to /var/cache/conftool/dbconfig/20260526-152559-fceratto.json * 15:24 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:24 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:24 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:24 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:23 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:22 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:22 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:22 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P93040 and previous config saved to /var/cache/conftool/dbconfig/20260526-151552-fceratto.json * 15:12 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2196: Rack maintenance completed * 15:10 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2196.codfw.wmnet * 15:10 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2196.codfw.wmnet * 15:07 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw * 15:06 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2222: Rack maintenance completed * 15:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P93037 and previous config saved to /var/cache/conftool/dbconfig/20260526-150546-fceratto.json * 15:04 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2221: Rack maintenance completed * 15:04 brennen@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab1004 for [[phab:T427286|T427286]] (duration: 00m 39s) * 15:03 brennen@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab1004 for [[phab:T427286|T427286]] * 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2002 for [[phab:T427286|T427286]] (duration: 00m 45s) * 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2002 for [[phab:T427286|T427286]] * 15:02 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2002.codfw.wmnet with reason: Phabricator deploy * 15:01 bjensen: uploading prometheus-memcached-exporter_0.16.0-1_amd64 on apt1002 * 15:01 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator deploy * 15:00 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2223: switch maintenance * 14:56 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2196: Rack maintenance completed * 14:55 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2221.codfw.wmnet * 14:55 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2221.codfw.wmnet * 14:55 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2222.codfw.wmnet * 14:55 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2222.codfw.wmnet * 14:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93033 and previous config saved to /var/cache/conftool/dbconfig/20260526-145538-fceratto.json * 14:55 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet * 14:54 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1026.eqiad.wmnet * 14:52 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet * 14:52 moritzm: remove ganeti1025 from eqiad Ganeti cluster [[phab:T424680|T424680]] * 14:51 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2030.codfw.wmnet to cluster codfw and group A * 14:51 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2222: Rack maintenance completed * 14:49 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:49 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2221: Rack maintenance completed * 14:49 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2030.codfw.wmnet to cluster codfw and group A * 14:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2029.codfw.wmnet to cluster codfw and group A * 14:47 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2029.codfw.wmnet to cluster codfw and group A * 14:47 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2177 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93030 and previous config saved to /var/cache/conftool/dbconfig/20260526-144718-fceratto.json * 14:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance * 14:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93029 and previous config saved to /var/cache/conftool/dbconfig/20260526-144651-fceratto.json * 14:45 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=wdqs-scholarly,name=codfw * 14:45 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=wdqs-scholarly,name=codfw * 14:43 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw * 14:40 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:40 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2167: Migration of db2167.codfw.wmnet completed * 14:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P93026 and previous config saved to /var/cache/conftool/dbconfig/20260526-143643-fceratto.json * 14:31 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1054.eqiad.wmnet with OS trixie * 14:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P93023 and previous config saved to /var/cache/conftool/dbconfig/20260526-142636-fceratto.json * 14:26 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:25 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:24 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1014: Rack maintenance completed * 14:24 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.parsercache (exit_code=99) * 14:24 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache * 14:24 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc1014: Rack maintenance completed * 14:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1025.eqiad.wmnet * 14:19 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for backup2015.codfw.wmnet,db2197.codfw.wmnet * 14:19 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for backup2015.codfw.wmnet,db2197.codfw.wmnet * 14:18 jynus: restarting mediabackups@codfw after maintenance on a codfw backup media storage server [[phab:T426199|T426199]] * 14:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93021 and previous config saved to /var/cache/conftool/dbconfig/20260526-141628-fceratto.json * 14:16 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:14 fabfur: repooled cp2043 ([[phab:T426199|T426199]]) * 14:14 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2223: switch maintenance * 14:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1054.eqiad.wmnet with reason: host reimage * 14:14 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2043.* * 14:13 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293710{{!}}Site info should output thumblimits as array (T427066)]] (duration: 06m 40s) * 14:12 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:10 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1054.eqiad.wmnet with reason: host reimage * 14:10 fabfur@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2011.codfw.wmnet * 14:10 fabfur@cumin1003: START - Cookbook sre.hosts.remove-downtime for lvs2011.codfw.wmnet * 14:09 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 14:09 fabfur: restoring lvs2011 as primary ([[phab:T426199|T426199]]) * 14:08 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1293710{{!}}Site info should output thumblimits as array (T427066)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:08 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 14:08 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 14:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93017 and previous config saved to /var/cache/conftool/dbconfig/20260526-140748-fceratto.json * 14:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance * 14:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93016 and previous config saved to /var/cache/conftool/dbconfig/20260526-140718-fceratto.json * 14:07 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1293710{{!}}Site info should output thumblimits as array (T427066)]] * 14:05 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.decommission (exit_code=99) * 14:05 marostegui@cumin1003: Removing pc1013 from zarcillo [[phab:T427190|T427190]] * 14:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1013.eqiad.wmnet * 14:04 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:04 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 14:04 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 14:00 marostegui@cumin1003: START - Cookbook sre.dns.netbox * 13:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P93014 and previous config saved to /var/cache/conftool/dbconfig/20260526-135711-fceratto.json * 13:56 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1054.eqiad.wmnet with OS trixie * 13:55 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2167: Migration of db2167.codfw.wmnet completed * 13:53 Amir1: drop flaggedrevs tables on cawikinews ([[phab:T423577|T423577]]) * 13:49 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1013.eqiad.wmnet * 13:49 marostegui@cumin1003: START - Cookbook sre.mysql.decommission * 13:48 Lucas_WMDE: UTC afternoon backport+config window done * 13:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P93012 and previous config saved to /var/cache/conftool/dbconfig/20260526-134703-fceratto.json * 13:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2167.codfw.wmnet with OS trixie * 13:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93011 and previous config saved to /var/cache/conftool/dbconfig/20260526-133656-fceratto.json * 13:36 XioNoX: reboot lsw1-a2-codfw for software upgrade - [[phab:T426199|T426199]] * 13:36 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2223: switch maintenance * 13:35 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2223: switch maintenance * 13:35 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2222: switch maintenance * 13:35 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2222: switch maintenance * 13:35 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2221: switch maintenance * 13:35 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293662{{!}}Enable IRS Direct Reporting on testwiki (T425025)]] (duration: 09m 28s) * 13:34 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2221: switch maintenance * 13:34 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2196: switch maintenance * 13:34 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2196: switch maintenance * 13:31 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 13:30 stran@deploy1003: stran: Continuing with deployment * 13:29 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 13:29 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93006 and previous config saved to /var/cache/conftool/dbconfig/20260526-132927-fceratto.json * 13:29 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2167.codfw.wmnet with reason: host reimage * 13:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance * 13:29 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 34 hosts with reason: Switch maintenance * 13:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93005 and previous config saved to /var/cache/conftool/dbconfig/20260526-132857-fceratto.json * 13:28 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lsw1-a2-codfw,lsw1-a2-codfw IPv6,lsw1-a2-codfw.mgmt with reason: Switch maintenance * 13:27 stran@deploy1003: stran: Backport for [[gerrit:1293662{{!}}Enable IRS Direct Reporting on testwiki (T425025)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:25 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1293662{{!}}Enable IRS Direct Reporting on testwiki (T425025)]] * 13:25 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2167.codfw.wmnet with reason: host reimage * 13:22 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293706{{!}}Disable the `no` language code for translation (T424613)]] (duration: 08m 30s) * 13:22 ladsgroup@dns1004: END - running authdns-update * 13:20 ladsgroup@dns1004: START - running authdns-update * 13:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P93004 and previous config saved to /var/cache/conftool/dbconfig/20260526-131850-fceratto.json * 13:18 lucaswerkmeister-wmde@deploy1003: jhsoby, lucaswerkmeister-wmde: Continuing with deployment * 13:16 lucaswerkmeister-wmde@deploy1003: jhsoby, lucaswerkmeister-wmde: Backport for [[gerrit:1293706{{!}}Disable the `no` language code for translation (T424613)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:14 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1293706{{!}}Disable the `no` language code for translation (T424613)]] * 13:12 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293177{{!}}Instrumentation: log new articles namespace and source (T422146)]] (duration: 07m 09s) * 13:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P93003 and previous config saved to /var/cache/conftool/dbconfig/20260526-130842-fceratto.json * 13:08 sbisson@deploy1003: sbisson: Continuing with deployment * 13:07 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2167.codfw.wmnet with OS trixie * 13:07 sbisson@deploy1003: sbisson: Backport for [[gerrit:1293177{{!}}Instrumentation: log new articles namespace and source (T422146)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:05 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2167: Upgrading db2167.codfw.wmnet * 13:05 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1293177{{!}}Instrumentation: log new articles namespace and source (T422146)]] * 13:04 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2167: Upgrading db2167.codfw.wmnet * 13:04 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:04 kart_: Update Recommendation API to 2026-05-26-074931-production * 13:03 kartik@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 13:00 topranks: deactivate CR BGP to doh2002 to test backup path via doh2001 * 12:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93000 and previous config saved to /var/cache/conftool/dbconfig/20260526-125834-fceratto.json * 12:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2226 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92999 and previous config saved to /var/cache/conftool/dbconfig/20260526-125135-fceratto.json * 12:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance * 12:51 kartik@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92998 and previous config saved to /var/cache/conftool/dbconfig/20260526-125105-fceratto.json * 12:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P92997 and previous config saved to /var/cache/conftool/dbconfig/20260526-124059-fceratto.json * 12:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2003.wikimedia.org * 12:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1214: Migration of db1214.eqiad.wmnet completed * 12:33 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc2003.wikimedia.org * 12:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P92995 and previous config saved to /var/cache/conftool/dbconfig/20260526-123052-fceratto.json * 12:26 fabfur: depooled cp204 for network activity ([[phab:T426199|T426199]]) * 12:26 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2043.* * 12:24 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-a1-codfw,ssw1-a1-codfw IPv6,ssw1-a1-codfw.mgmt with reason: Switch maintenance * 12:24 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mirror1001.wikimedia.org * 12:23 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:23 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:22 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92993 and previous config saved to /var/cache/conftool/dbconfig/20260526-122044-fceratto.json * 12:20 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:19 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mirror1001.wikimedia.org * 12:13 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2225 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92991 and previous config saved to /var/cache/conftool/dbconfig/20260526-121336-fceratto.json * 12:13 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance * 12:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92990 and previous config saved to /var/cache/conftool/dbconfig/20260526-121306-fceratto.json * 12:09 fabfur@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2011.codfw.wmnet with reason: Planned downtime for rack maintenance * 12:08 fabfur: downtime, disable puppet and stop pybal for rack maintenance ([[phab:T426199|T426199]]) * 12:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2181: Migration of db2181.codfw.wmnet completed * 12:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P92987 and previous config saved to /var/cache/conftool/dbconfig/20260526-120258-fceratto.json * 12:01 XioNoX: start ssw1-a1-codfw network maintenance (no impact expected as the spines are redundant) * 11:59 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293167{{!}}hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055{{!}}hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] (duration: 15m 26s) * 11:56 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on backup2015.codfw.wmnet,db2197.codfw.wmnet with reason: network maintenance * 11:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1005.eqiad.wmnet * 11:55 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment * 11:54 jynus: stopping mediabackups@codfw for maintenance on a codfw backup media storage server [[phab:T426199|T426199]] * 11:54 jmm@dns1004: END - running authdns-update * 11:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P92985 and previous config saved to /var/cache/conftool/dbconfig/20260526-115251-fceratto.json * 11:52 jmm@dns1004: START - running authdns-update * 11:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1005.eqiad.wmnet * 11:49 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1214: Migration of db1214.eqiad.wmnet completed * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1004.eqiad.wmnet * 11:47 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet * 11:46 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1293167{{!}}hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055{{!}}hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:45 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1004.eqiad.wmnet * 11:44 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1293167{{!}}hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055{{!}}hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] * 11:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92983 and previous config saved to /var/cache/conftool/dbconfig/20260526-114243-fceratto.json * 11:42 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet * 11:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1214.eqiad.wmnet with OS trixie * 11:35 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293691{{!}}Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] (duration: 06m 46s) * 11:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92981 and previous config saved to /var/cache/conftool/dbconfig/20260526-113542-fceratto.json * 11:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance * 11:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92980 and previous config saved to /var/cache/conftool/dbconfig/20260526-113521-fceratto.json * 11:31 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 11:31 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1293691{{!}}Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1222: Migration of db1222.eqiad.wmnet completed * 11:29 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1293691{{!}}Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] * 11:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P92978 and previous config saved to /var/cache/conftool/dbconfig/20260526-112513-fceratto.json * 11:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1214.eqiad.wmnet with reason: host reimage * 11:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc4 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92977 and previous config saved to /var/cache/conftool/dbconfig/20260526-112326-marostegui.json * 11:22 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2181: Migration of db2181.codfw.wmnet completed * 11:22 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1024 to dbctl [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92975 and previous config saved to /var/cache/conftool/dbconfig/20260526-112215-marostegui.json * 11:20 fceratto@cumin1003: dbctl commit (dc=all): 'Switchover es2042 es2041 for [[phab:T426199|T426199]]', diff saved to https://phabricator.wikimedia.org/P92974 and previous config saved to /var/cache/conftool/dbconfig/20260526-112028-fceratto.json * 11:17 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1214.eqiad.wmnet with reason: host reimage * 11:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P92972 and previous config saved to /var/cache/conftool/dbconfig/20260526-111506-fceratto.json * 11:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2181.codfw.wmnet with OS trixie * 11:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92971 and previous config saved to /var/cache/conftool/dbconfig/20260526-110458-fceratto.json * 11:02 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1214.eqiad.wmnet with OS trixie * 11:00 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293095{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] (duration: 15m 50s) * 11:00 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1214: Upgrading db1214.eqiad.wmnet * 10:59 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1214: Upgrading db1214.eqiad.wmnet * 10:59 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92968 and previous config saved to /var/cache/conftool/dbconfig/20260526-105755-fceratto.json * 10:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance * 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92967 and previous config saved to /var/cache/conftool/dbconfig/20260526-105726-fceratto.json * 10:56 jiji@deploy1003: jiji: Continuing with deployment * 10:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2181.codfw.wmnet with reason: host reimage * 10:51 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2181.codfw.wmnet with reason: host reimage * 10:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P92966 and previous config saved to /var/cache/conftool/dbconfig/20260526-104718-fceratto.json * 10:46 jiji@deploy1003: jiji: Backport for [[gerrit:1293095{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:44 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1293095{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] * 10:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P92964 and previous config saved to /var/cache/conftool/dbconfig/20260526-103711-fceratto.json * 10:36 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2181.codfw.wmnet with OS trixie * 10:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply * 10:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply * 10:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92963 and previous config saved to /var/cache/conftool/dbconfig/20260526-102703-fceratto.json * 10:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1226: Migration of db1226.eqiad.wmnet completed * 10:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2181: Upgrading db2181.codfw.wmnet * 10:24 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2181: Upgrading db2181.codfw.wmnet * 10:24 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92960 and previous config saved to /var/cache/conftool/dbconfig/20260526-101936-fceratto.json * 10:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance * 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92959 and previous config saved to /var/cache/conftool/dbconfig/20260526-101842-fceratto.json * 10:16 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-codfw@codfw * 10:16 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 10:15 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 10:10 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293668{{!}}hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] (duration: 06m 42s) * 10:09 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-codfw@codfw * 10:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92957 and previous config saved to /var/cache/conftool/dbconfig/20260526-100834-fceratto.json * 10:06 kharlan@deploy1003: kharlan: Continuing with deployment * 10:05 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293668{{!}}hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:03 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293668{{!}}hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] * 10:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2195: Migration of db2195.codfw.wmnet completed * 10:01 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>kubestage200*<nowiki>}</nowiki> and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw) * 10:01 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet * 10:01 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet * 10:00 jmm@cumin2002: END (PASS) - Cookbook sre.netbox.restart-reboot (exit_code=0) rolling reboot on A:netbox * 09:58 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 09:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92955 and previous config saved to /var/cache/conftool/dbconfig/20260526-095827-fceratto.json * 09:58 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 09:58 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 09:57 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 09:56 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-eqiad@eqiad * 09:56 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:55 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 09:55 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:55 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:55 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet * 09:54 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet * 09:54 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet * 09:54 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet * 09:53 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>kubestage100*<nowiki>}</nowiki> and (A:wikikube-staging-master-eqiad or A:wikikube-staging-worker-eqiad) * 09:53 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet * 09:53 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet * 09:52 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-eqiad@eqiad * 09:52 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293665{{!}}hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] (duration: 08m 07s) * 09:51 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2043.* * 09:51 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2044.* * 09:48 fabfur: repooling cp2043 and cp2044 (haproxy-awslc) ([[phab:T419825|T419825]]) * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92953 and previous config saved to /var/cache/conftool/dbconfig/20260526-094819-fceratto.json * 09:47 kharlan@deploy1003: kharlan: Continuing with deployment * 09:46 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet * 09:45 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293665{{!}}hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:44 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs3009.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:44 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293665{{!}}hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] * 09:41 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet * 09:41 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet * 09:41 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet * 09:41 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92951 and previous config saved to /var/cache/conftool/dbconfig/20260526-094115-fceratto.json * 09:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2229.codfw.wmnet with reason: Maintenance * 09:41 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs3009.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92950 and previous config saved to /var/cache/conftool/dbconfig/20260526-094045-fceratto.json * 09:40 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1226: Migration of db1226.eqiad.wmnet completed * 09:39 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-codfw@codfw * 09:39 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:38 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:34 fabfur: depooling cp2044 to install haproxy-awslc ([[phab:T419825|T419825]]) * 09:34 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet * 09:34 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet * 09:34 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2044.* * 09:33 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet * 09:33 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet * 09:33 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet * 09:33 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2043.* * 09:32 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293661{{!}}hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] (duration: 06m 52s) * 09:32 fabfur: depooling cp2043 to install haproxy-awslc ([[phab:T419825|T419825]]) * 09:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1226.eqiad.wmnet with OS trixie * 09:30 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-codfw@codfw * 09:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P92947 and previous config saved to /var/cache/conftool/dbconfig/20260526-093031-fceratto.json * 09:29 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet * 09:29 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet * 09:29 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet * 09:28 kharlan@deploy1003: kharlan: Continuing with deployment * 09:28 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs3008.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:28 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293661{{!}}hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:27 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet * 09:26 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet * 09:26 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet * 09:26 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet * 09:26 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293661{{!}}hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] * 09:25 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs3008.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:25 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs3010.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet * 09:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet * 09:21 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs3010.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:20 fabfur: start rebooting esams liberica instances ([[phab:T426563|T426563]]) * 09:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P92946 and previous config saved to /var/cache/conftool/dbconfig/20260526-092024-fceratto.json * 09:20 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet * 09:16 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2195: Migration of db2195.codfw.wmnet completed * 09:15 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet * 09:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1226.eqiad.wmnet with reason: host reimage * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>kubestage100*<nowiki>}</nowiki> and (A:wikikube-staging-master-eqiad or A:wikikube-staging-worker-eqiad) * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>kubestage200*<nowiki>}</nowiki> and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw) * 09:14 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293658{{!}}Fix TypeError in Mandatory2FAChecker (T427251)]] (duration: 06m 47s) * 09:10 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1226.eqiad.wmnet with reason: host reimage * 09:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92944 and previous config saved to /var/cache/conftool/dbconfig/20260526-091016-fceratto.json * 09:09 mszwarc@deploy1003: mszwarc: Continuing with deployment * 09:09 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1293658{{!}}Fix TypeError in Mandatory2FAChecker (T427251)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2195.codfw.wmnet with OS trixie * 09:07 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293658{{!}}Fix TypeError in Mandatory2FAChecker (T427251)]] * 09:06 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs4009.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 09:03 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92943 and previous config saved to /var/cache/conftool/dbconfig/20260526-090315-fceratto.json * 09:03 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2224.codfw.wmnet with reason: Maintenance * 09:03 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs4009.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92942 and previous config saved to /var/cache/conftool/dbconfig/20260526-090256-fceratto.json * 08:57 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs4008.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 08:56 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox.discovery.wmnet. on all recursors * 08:56 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors * 08:55 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1226.eqiad.wmnet with OS trixie * 08:53 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs4008.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 08:53 fabfur: start rebooting ulsfo liberica instances ([[phab:T426563|T426563]]) * 08:53 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293594{{!}}Allow to remove passkeys when there's only one standard 2FA method (T426872)]] (duration: 07m 23s) * 08:53 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs5005.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:53 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1226: Upgrading db1226.eqiad.wmnet * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P92941 and previous config saved to /var/cache/conftool/dbconfig/20260526-085248-fceratto.json * 08:51 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox.discovery.wmnet. on all recursors * 08:51 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors * 08:51 jmm@cumin2002: START - Cookbook sre.netbox.restart-reboot rolling reboot on A:netbox * 08:50 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1226: Upgrading db1226.eqiad.wmnet * 08:50 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs5005.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:50 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2195.codfw.wmnet with reason: host reimage * 08:49 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1222: Migration of db1222.eqiad.wmnet completed * 08:48 mszwarc@deploy1003: mszwarc: Continuing with deployment * 08:47 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1293594{{!}}Allow to remove passkeys when there's only one standard 2FA method (T426872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:46 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293594{{!}}Allow to remove passkeys when there's only one standard 2FA method (T426872)]] * 08:43 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs5004.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet * 08:43 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2195.codfw.wmnet with reason: host reimage * 08:43 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1292032{{!}}Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964{{!}}hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] (duration: 09m 56s) * 08:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P92939 and previous config saved to /var/cache/conftool/dbconfig/20260526-084240-fceratto.json * 08:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1222.eqiad.wmnet with OS trixie * 08:40 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs5004.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:40 fabfur: start rebooting eqsin liberica instances ([[phab:T426563|T426563]]) * 08:39 kartik@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 08:39 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet * 08:39 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 08:39 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs5006.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:35 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs5006.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1024.eqiad.wmnet * 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1024.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 08:35 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1292032{{!}}Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964{{!}}hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:33 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs6002.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:33 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1292032{{!}}Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964{{!}}hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] * 08:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92938 and previous config saved to /var/cache/conftool/dbconfig/20260526-083233-fceratto.json * 08:30 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs6002.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:25 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2217 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92937 and previous config saved to /var/cache/conftool/dbconfig/20260526-082531-fceratto.json * 08:25 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2217.codfw.wmnet with reason: Maintenance * 08:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92936 and previous config saved to /var/cache/conftool/dbconfig/20260526-082458-fceratto.json * 08:23 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2195.codfw.wmnet with OS trixie * 08:23 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage * 08:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2195: Upgrading db2195.codfw.wmnet * 08:20 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2195: Upgrading db2195.codfw.wmnet * 08:19 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:18 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage * 08:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P92934 and previous config saved to /var/cache/conftool/dbconfig/20260526-081451-fceratto.json * 08:13 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs6001.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 08:10 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs6001.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:09 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1024.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 08:04 jmm@cumin2002: START - Cookbook sre.dns.netbox * 08:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P92932 and previous config saved to /var/cache/conftool/dbconfig/20260526-080443-fceratto.json * 08:01 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1222.eqiad.wmnet with OS trixie * 08:00 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs6003.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet * 07:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet * 07:59 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1024.eqiad.wmnet * 07:59 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1023.eqiad.wmnet * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1023.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:59 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 07:59 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:58 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1023.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:56 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs6003.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 07:56 fabfur: start rebooting drmrs liberica instances ([[phab:T426563|T426563]]) * 07:56 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs7002.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:54 jmm@cumin2002: START - Cookbook sre.dns.netbox * 07:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92931 and previous config saved to /var/cache/conftool/dbconfig/20260526-075435-fceratto.json * 07:52 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs7002.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:51 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1047.eqiad.wmnet * 07:51 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:51 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1047.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:49 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1023.eqiad.wmnet * 07:47 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2193 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92930 and previous config saved to /var/cache/conftool/dbconfig/20260526-074739-fceratto.json * 07:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2193.codfw.wmnet with reason: Maintenance * 07:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92929 and previous config saved to /var/cache/conftool/dbconfig/20260526-074710-fceratto.json * 07:46 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet * 07:45 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet * 07:45 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:45 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs7001.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:44 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1025.eqiad.wmnet * 07:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:43 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:41 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs7001.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:40 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1046.eqiad.wmnet * 07:40 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1046.eqiad.wmnet * 07:38 arthurtaylor@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291951{{!}}Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] (duration: 12m 01s) * 07:38 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1047.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P92928 and previous config saved to /var/cache/conftool/dbconfig/20260526-073702-fceratto.json * 07:37 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet * 07:36 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet * 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance * 07:35 fabfur: start rebooting magru liberica instances ([[phab:T426563|T426563]]) * 07:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92926 and previous config saved to /var/cache/conftool/dbconfig/20260526-073459-fceratto.json * 07:32 arthurtaylor@deploy1003: arthurtaylor: Continuing with deployment * 07:31 arthurtaylor@deploy1003: arthurtaylor: Backport for [[gerrit:1291951{{!}}Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:30 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1046.eqiad.wmnet * 07:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20260526-072643-fceratto.json * 07:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1046.eqiad.wmnet * 07:26 arthurtaylor@deploy1003: Started scap sync-world: Backport for [[gerrit:1291951{{!}}Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] * 07:25 jiji@cumin1003: START - Cookbook sre.dns.netbox * 07:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P92924 and previous config saved to /var/cache/conftool/dbconfig/20260526-072452-fceratto.json * 07:24 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet * 07:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1047.eqiad.wmnet * 07:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1047.eqiad.wmnet * 07:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92923 and previous config saved to /var/cache/conftool/dbconfig/20260526-071635-fceratto.json * 07:15 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet * 07:15 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1026.eqiad.wmnet * 07:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P92922 and previous config saved to /var/cache/conftool/dbconfig/20260526-071444-fceratto.json * 07:13 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet * 07:11 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1025.eqiad.wmnet * 07:10 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1025.eqiad.wmnet * 07:09 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92921 and previous config saved to /var/cache/conftool/dbconfig/20260526-070946-fceratto.json * 07:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2180.codfw.wmnet with reason: Maintenance * 07:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92920 and previous config saved to /var/cache/conftool/dbconfig/20260526-070916-fceratto.json * 07:09 moritzm: failover Ganeti master in eqiad to ganeti1048 * 07:09 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1047.eqiad.wmnet * 07:07 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1046.eqiad.wmnet * 07:07 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:06 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1046.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org * 07:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92919 and previous config saved to /var/cache/conftool/dbconfig/20260526-070436-fceratto.json * 07:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet * 07:04 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1046.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet * 07:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org * 06:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P92918 and previous config saved to /var/cache/conftool/dbconfig/20260526-065909-fceratto.json * 06:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org * 06:58 jiji@cumin1003: START - Cookbook sre.dns.netbox * 06:58 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet * 06:55 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet * 06:53 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1046.eqiad.wmnet * 06:53 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1045.eqiad.wmnet * 06:53 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:53 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1045.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 06:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org * 06:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P92917 and previous config saved to /var/cache/conftool/dbconfig/20260526-064901-fceratto.json * 06:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92916 and previous config saved to /var/cache/conftool/dbconfig/20260526-064833-fceratto.json * 06:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1222.eqiad.wmnet with reason: Maintenance * 06:47 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1222: Switchover * 06:41 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92914 and previous config saved to /var/cache/conftool/dbconfig/20260526-063853-fceratto.json * 06:35 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 06:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92912 and previous config saved to /var/cache/conftool/dbconfig/20260526-063155-fceratto.json * 06:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance * 06:28 fceratto@cumin1003: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance * 06:23 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1222: Switchover * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1222 [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92910 and previous config saved to /var/cache/conftool/dbconfig/20260526-061656-fceratto.json * 06:15 fceratto@dns1005: END - running authdns-update * 06:14 fceratto@dns1005: START - running authdns-update * 06:11 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1162 to s2 primary and set section read-write [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92909 and previous config saved to /var/cache/conftool/dbconfig/20260526-061114-fceratto.json * 06:10 fceratto@cumin1003: dbctl commit (dc=all): 'Set s2 eqiad as read-only for maintenance - [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92908 and previous config saved to /var/cache/conftool/dbconfig/20260526-061021-fceratto.json * 06:10 federico3: Starting s2 eqiad failover from db1222 to db1162 - [[phab:T425622|T425622]] * 06:04 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1162 with weight 0 [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92907 and previous config saved to /var/cache/conftool/dbconfig/20260526-060443-fceratto.json * 06:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s2 [[phab:T425622|T425622]] * 06:02 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:02 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:01 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:00 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 05:15 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1014.eqiad.wmnet: Maintenance on pc4 * 05:15 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 05:15 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 05:15 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1014.eqiad.wmnet: Maintenance on pc4 * 05:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2024.codfw.wmnet,pc[1014,1024].eqiad.wmnet with reason: Maintenance on pc4 * 04:37 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 04:34 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.1 (duration: 02m 32s) * 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] (duration: 36m 24s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 20s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-25 == * 21:00 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1045.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:49 jiji@cumin1003: START - Cookbook sre.dns.netbox * 20:38 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1045.eqiad.wmnet * 20:37 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1044.eqiad.wmnet * 20:37 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:37 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1044.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:25 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1044.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:15 moritzm: truncate krb5kdc.log1 (which made log rotation fail) * 20:06 jiji@cumin1003: START - Cookbook sre.dns.netbox * 19:57 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1044.eqiad.wmnet * 19:25 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1043.eqiad.wmnet * 19:25 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:25 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1043.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 19:22 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1043.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 18:49 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_eqiad * 18:49 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1115.eqiad.wmnet * 18:34 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5023.eqsin.wmnet [reason: manually pooling after reboot as icinga was down] * 18:33 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5030.eqsin.wmnet [reason: manually pooling after reboot as icinga was down] * 18:22 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp5030*<nowiki>}</nowiki> and A:cp * 18:22 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5030.eqsin.wmnet * 18:15 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp5023*<nowiki>}</nowiki> and A:cp * 18:15 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5023.eqsin.wmnet * 18:10 jiji@cumin1003: START - Cookbook sre.dns.netbox * 18:10 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp5030*<nowiki>}</nowiki> and A:cp * 18:09 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp1113*<nowiki>}</nowiki> and A:cp * 18:09 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet * 18:09 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet * 18:03 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp1113*<nowiki>}</nowiki> and A:cp * 18:02 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp5023*<nowiki>}</nowiki> and A:cp * 18:01 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-text_eqiad * 18:01 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-upload_eqsin * 18:01 sukhe: sre.cdn.roll-reboot cookbooks stalled due to icinga reboot * 18:00 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-text_eqsin * 17:35 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1043.eqiad.wmnet * 17:31 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1110.eqiad.wmnet [reason: manually pooling after reboot as icinga was down] * 17:30 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1042.eqiad.wmnet * 17:30 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:30 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1042.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:29 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1111.eqiad.wmnet * 17:28 sukhe: sukhe@alert1002:~$ sudo systemctl restart icinga.service * 17:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92903 and previous config saved to /var/cache/conftool/dbconfig/20260525-171310-fceratto.json * 17:11 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1042.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:06 jiji@cumin1003: START - Cookbook sre.dns.netbox * 17:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P92902 and previous config saved to /var/cache/conftool/dbconfig/20260525-170302-fceratto.json * 16:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P92901 and previous config saved to /var/cache/conftool/dbconfig/20260525-165255-fceratto.json * 16:51 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1042.eqiad.wmnet * 16:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92900 and previous config saved to /var/cache/conftool/dbconfig/20260525-164247-fceratto.json * 16:42 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1041.eqiad.wmnet * 16:42 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:42 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1041.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:41 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1041.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:40 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5021.eqsin.wmnet * 16:39 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5029.eqsin.wmnet * 16:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92899 and previous config saved to /var/cache/conftool/dbconfig/20260525-163559-fceratto.json * 16:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2158.codfw.wmnet with reason: Maintenance * 16:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92898 and previous config saved to /var/cache/conftool/dbconfig/20260525-163512-fceratto.json * 16:34 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1108.eqiad.wmnet * 16:30 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1109.eqiad.wmnet * 16:26 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249', diff saved to https://phabricator.wikimedia.org/P92897 and previous config saved to /var/cache/conftool/dbconfig/20260525-162505-fceratto.json * 16:20 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1041.eqiad.wmnet * 16:20 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1040.eqiad.wmnet * 16:20 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:20 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1040.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:16 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1040.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249', diff saved to https://phabricator.wikimedia.org/P92896 and previous config saved to /var/cache/conftool/dbconfig/20260525-161457-fceratto.json * 16:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92895 and previous config saved to /var/cache/conftool/dbconfig/20260525-160450-fceratto.json * 16:02 jiji@cumin1003: START - Cookbook sre.dns.netbox * 15:59 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2249 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92894 and previous config saved to /var/cache/conftool/dbconfig/20260525-155930-fceratto.json * 15:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2249.codfw.wmnet with reason: Maintenance * 15:57 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5020.eqsin.wmnet * 15:57 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5028.eqsin.wmnet * 15:52 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1106.eqiad.wmnet * 15:51 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1107.eqiad.wmnet * 15:29 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1040.eqiad.wmnet * 15:29 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1039.eqiad.wmnet * 15:29 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:29 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1039.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:27 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1039.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:17 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1013 from dbctl [[phab:T427190|T427190]]', diff saved to https://phabricator.wikimedia.org/P92893 and previous config saved to /var/cache/conftool/dbconfig/20260525-151718-marostegui.json * 15:15 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5019.eqsin.wmnet * 15:15 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5027.eqsin.wmnet * 15:12 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1104.eqiad.wmnet * 15:11 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1105.eqiad.wmnet * 15:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92892 and previous config saved to /var/cache/conftool/dbconfig/20260525-150309-fceratto.json * 14:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P92891 and previous config saved to /var/cache/conftool/dbconfig/20260525-145301-fceratto.json * 14:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P92890 and previous config saved to /var/cache/conftool/dbconfig/20260525-144253-fceratto.json * 14:33 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1102.eqiad.wmnet * 14:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92889 and previous config saved to /var/cache/conftool/dbconfig/20260525-143246-fceratto.json * 14:32 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5026.eqsin.wmnet * 14:32 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5018.eqsin.wmnet * 14:31 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1103.eqiad.wmnet * 14:25 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2228 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92888 and previous config saved to /var/cache/conftool/dbconfig/20260525-142551-fceratto.json * 14:25 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance * 14:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92887 and previous config saved to /var/cache/conftool/dbconfig/20260525-142520-fceratto.json * 14:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P92885 and previous config saved to /var/cache/conftool/dbconfig/20260525-141513-fceratto.json * 14:12 jiji@cumin1003: START - Cookbook sre.dns.netbox * 14:06 sukhe: curl localhost:9090/pools/inference-staging-grpc_30051 shows ml-staging200[1-3].codfw.wmnet as enabled and pooled: [[phab:T424049|T424049]] * 14:05 sukhe: sukhe@lvs2013:~$ sudo systemctl restart pybal.service: [[phab:T424049|T424049]] * 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P92884 and previous config saved to /var/cache/conftool/dbconfig/20260525-140505-fceratto.json * 14:03 sukhe: sudo cumin 'A:lvs and A:lvs-low-traffic-codfw' 'run-puppet-agent --enable "adding new ml-serve (grpc) [[phab:T424049|T424049]]"' * 14:02 sukhe: sukhe@lvs2014:~$ sudo systemctl restart pybal.service": [[phab:T424049|T424049]] * 14:02 sukhe: sukhe@lvs2014:~$ sudo systemctl restart pybal.service * 14:00 sukhe: sudo cumin 'A:lvs and A:lvs-secondary-codfw' 'run-puppet-agent --enable "adding new ml-serve (grpc) [[phab:T424049|T424049]]"' * 13:59 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1039.eqiad.wmnet * 13:58 sukhe: sudo cumin 'A:lvs and A:eqiad' 'run-puppet-agent --enable "adding new ml-serve (grpc) [[phab:T424049|T424049]]": NOOP change, since service is codfw only * 13:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92882 and previous config saved to /var/cache/conftool/dbconfig/20260525-135458-fceratto.json * 13:52 Msz2001: Everything deployed, UTC afternoon config+backport window done * 13:52 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293119{{!}}Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] (duration: 09m 43s) * 13:51 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1101.eqiad.wmnet * 13:51 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1100.eqiad.wmnet * 13:50 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5025.eqsin.wmnet * 13:50 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5017.eqsin.wmnet * 13:49 kart_: Updated Recommendation API to 2026-05-21-044522-production * 13:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2223 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92881 and previous config saved to /var/cache/conftool/dbconfig/20260525-134807-fceratto.json * 13:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2223.codfw.wmnet with reason: Maintenance * 13:47 mszwarc@deploy1003: vadymts1, mszwarc: Continuing with deployment * 13:47 kartik@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 13:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92880 and previous config saved to /var/cache/conftool/dbconfig/20260525-134737-fceratto.json * 13:45 mszwarc@deploy1003: vadymts1, mszwarc: Backport for [[gerrit:1293119{{!}}Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:45 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1162: Reboot * 13:43 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293119{{!}}Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] * 13:40 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqiad * 13:39 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqiad * 13:38 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290813{{!}}Article Guidance: enable experiment on phase 2 wikis (T426871)]] (duration: 08m 14s) * 13:38 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqsin * 13:38 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqsin * 13:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P92878 and previous config saved to /var/cache/conftool/dbconfig/20260525-133729-fceratto.json * 13:34 sbisson@deploy1003: sbisson: Continuing with deployment * 13:33 kartik@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 13:32 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1038.eqiad.wmnet * 13:32 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:32 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1038.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 13:31 sbisson@deploy1003: sbisson: Backport for [[gerrit:1290813{{!}}Article Guidance: enable experiment on phase 2 wikis (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:30 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1290813{{!}}Article Guidance: enable experiment on phase 2 wikis (T426871)]] * 13:27 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293094{{!}}Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953{{!}}Update logo, wordmark and tagline for zghwiki (T426406)]] (duration: 07m 43s) * 13:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P92876 and previous config saved to /var/cache/conftool/dbconfig/20260525-132722-fceratto.json * 13:23 mszwarc@deploy1003: mszwarc, jhsoby: Continuing with deployment * 13:21 mszwarc@deploy1003: mszwarc, jhsoby: Backport for [[gerrit:1293094{{!}}Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953{{!}}Update logo, wordmark and tagline for zghwiki (T426406)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:20 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1038.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 13:20 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293094{{!}}Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953{{!}}Update logo, wordmark and tagline for zghwiki (T426406)]] * 13:19 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291966{{!}}Modify various configurations for English Wikibooks (T426992)]] (duration: 15m 53s) * 13:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92875 and previous config saved to /var/cache/conftool/dbconfig/20260525-131714-fceratto.json * 13:12 mszwarc@deploy1003: vadymts1, mszwarc: Continuing with deployment * 13:12 jiji@cumin1003: START - Cookbook sre.dns.netbox * 13:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92873 and previous config saved to /var/cache/conftool/dbconfig/20260525-131023-fceratto.json * 13:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2211.codfw.wmnet with reason: Maintenance * 13:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92872 and previous config saved to /var/cache/conftool/dbconfig/20260525-130950-fceratto.json * 13:07 mszwarc@deploy1003: vadymts1, mszwarc: Backport for [[gerrit:1291966{{!}}Modify various configurations for English Wikibooks (T426992)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:03 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1291966{{!}}Modify various configurations for English Wikibooks (T426992)]] * 12:59 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1162: Reboot * 12:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P92870 and previous config saved to /var/cache/conftool/dbconfig/20260525-125942-fceratto.json * 12:59 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1162: Reboot * 12:59 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1162: Reboot * 12:58 kart_: Updated cxserver to 2026-05-24-103047-production ([[phab:T426808|T426808]], [[phab:T373418|T373418]]) * 12:56 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply * 12:56 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply * 12:54 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db1162: Reboot * 12:54 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1162: Reboot * 12:54 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply * 12:53 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply * 12:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1162.eqiad.wmnet with reason: Reboot * 12:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P92868 and previous config saved to /var/cache/conftool/dbconfig/20260525-124934-fceratto.json * 12:40 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply * 12:39 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply * 12:39 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1038.eqiad.wmnet * 12:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92867 and previous config saved to /var/cache/conftool/dbconfig/20260525-123927-fceratto.json * 12:32 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2192 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92866 and previous config saved to /var/cache/conftool/dbconfig/20260525-123239-fceratto.json * 12:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2192.codfw.wmnet with reason: Maintenance * 12:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92865 and previous config saved to /var/cache/conftool/dbconfig/20260525-123208-fceratto.json * 12:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P92864 and previous config saved to /var/cache/conftool/dbconfig/20260525-122201-fceratto.json * 12:17 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1037.eqiad.wmnet * 12:17 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:17 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1037.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 12:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P92863 and previous config saved to /var/cache/conftool/dbconfig/20260525-121153-fceratto.json * 12:10 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1037.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 12:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92862 and previous config saved to /var/cache/conftool/dbconfig/20260525-120145-fceratto.json * 11:58 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2178 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92861 and previous config saved to /var/cache/conftool/dbconfig/20260525-115504-fceratto.json * 11:54 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2178.codfw.wmnet with reason: Maintenance * 11:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92860 and previous config saved to /var/cache/conftool/dbconfig/20260525-115434-fceratto.json * 11:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P92859 and previous config saved to /var/cache/conftool/dbconfig/20260525-114426-fceratto.json * 11:43 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1037.eqiad.wmnet * 11:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P92858 and previous config saved to /var/cache/conftool/dbconfig/20260525-113419-fceratto.json * 11:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2160.codfw.wmnet with OS trixie * 11:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92857 and previous config saved to /var/cache/conftool/dbconfig/20260525-112411-fceratto.json * 11:17 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2171 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92856 and previous config saved to /var/cache/conftool/dbconfig/20260525-111717-fceratto.json * 11:17 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: Maintenance * 11:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92855 and previous config saved to /var/cache/conftool/dbconfig/20260525-111648-fceratto.json * 11:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P92854 and previous config saved to /var/cache/conftool/dbconfig/20260525-110640-fceratto.json * 11:05 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2160.codfw.wmnet with reason: host reimage * 11:00 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2160.codfw.wmnet with reason: host reimage * 10:58 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:57 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:56 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P92853 and previous config saved to /var/cache/conftool/dbconfig/20260525-105633-fceratto.json * 10:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92852 and previous config saved to /var/cache/conftool/dbconfig/20260525-104625-fceratto.json * 10:43 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2160.codfw.wmnet with OS trixie * 10:41 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc3 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92851 and previous config saved to /var/cache/conftool/dbconfig/20260525-104141-marostegui.json * 10:40 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1023 to pc3 as master [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92850 and previous config saved to /var/cache/conftool/dbconfig/20260525-104055-marostegui.json * 10:40 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1023 to dbctl', diff saved to https://phabricator.wikimedia.org/P92849 and previous config saved to /var/cache/conftool/dbconfig/20260525-104027-marostegui.json * 10:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92848 and previous config saved to /var/cache/conftool/dbconfig/20260525-103944-fceratto.json * 10:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance * 10:31 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply * 10:30 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply * 10:27 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 10:18 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 10:16 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1011.eqiad.wmnet * 10:08 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1011.eqiad.wmnet * 10:08 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1007.eqiad.wmnet * 09:59 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1007.eqiad.wmnet * 09:59 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1006.eqiad.wmnet * 09:57 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:49 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1006.eqiad.wmnet * 09:48 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:46 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:45 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:40 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:40 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:28 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:17 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:13 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92847 and previous config saved to /var/cache/conftool/dbconfig/20260525-091302-fceratto.json * 09:12 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231', diff saved to https://phabricator.wikimedia.org/P92846 and previous config saved to /var/cache/conftool/dbconfig/20260525-090255-fceratto.json * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231', diff saved to https://phabricator.wikimedia.org/P92845 and previous config saved to /var/cache/conftool/dbconfig/20260525-085247-fceratto.json * 08:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92844 and previous config saved to /var/cache/conftool/dbconfig/20260525-084239-fceratto.json * 08:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92843 and previous config saved to /var/cache/conftool/dbconfig/20260525-083540-fceratto.json * 08:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2231.codfw.wmnet with reason: Maintenance * 08:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92842 and previous config saved to /var/cache/conftool/dbconfig/20260525-083511-fceratto.json * 08:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215', diff saved to https://phabricator.wikimedia.org/P92841 and previous config saved to /var/cache/conftool/dbconfig/20260525-082504-fceratto.json * 08:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215', diff saved to https://phabricator.wikimedia.org/P92840 and previous config saved to /var/cache/conftool/dbconfig/20260525-081456-fceratto.json * 08:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92839 and previous config saved to /var/cache/conftool/dbconfig/20260525-080448-fceratto.json * 07:57 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2215 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92838 and previous config saved to /var/cache/conftool/dbconfig/20260525-075739-fceratto.json * 07:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2215.codfw.wmnet with reason: Maintenance * 07:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92837 and previous config saved to /var/cache/conftool/dbconfig/20260525-075708-fceratto.json * 07:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196', diff saved to https://phabricator.wikimedia.org/P92836 and previous config saved to /var/cache/conftool/dbconfig/20260525-074700-fceratto.json * 07:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196', diff saved to https://phabricator.wikimedia.org/P92835 and previous config saved to /var/cache/conftool/dbconfig/20260525-073653-fceratto.json * 07:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92834 and previous config saved to /var/cache/conftool/dbconfig/20260525-072645-fceratto.json * 07:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92833 and previous config saved to /var/cache/conftool/dbconfig/20260525-071953-fceratto.json * 07:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2196.codfw.wmnet with reason: Maintenance * 07:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92832 and previous config saved to /var/cache/conftool/dbconfig/20260525-071924-fceratto.json * 07:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186', diff saved to https://phabricator.wikimedia.org/P92831 and previous config saved to /var/cache/conftool/dbconfig/20260525-070917-fceratto.json * 07:03 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2233.codfw.wmnet with OS trixie * 06:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186', diff saved to https://phabricator.wikimedia.org/P92830 and previous config saved to /var/cache/conftool/dbconfig/20260525-065909-fceratto.json * 06:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92829 and previous config saved to /var/cache/conftool/dbconfig/20260525-064902-fceratto.json * 06:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92828 and previous config saved to /var/cache/conftool/dbconfig/20260525-064305-fceratto.json * 06:42 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance * 06:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2233.codfw.wmnet with reason: host reimage * 06:35 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2233.codfw.wmnet with reason: host reimage * 06:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2233.codfw.wmnet with OS trixie * 06:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2233.codfw.wmnet with reason: Reimage to Trixie * 06:17 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 06:17 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:15 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2160.codfw.wmnet with reason: Reboot upgrade m2 * 06:15 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2233.codfw.wmnet with reason: Reboot upgrade m2 * 06:08 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy1027.eqiad.wmnet with reason: Reboot * 05:18 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2023.codfw.wmnet,pc[1013,1023].eqiad.wmnet with reason: Maintenance on pc3 * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1013.eqiad.wmnet: Maintenance on pc3 * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 05:17 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 05:17 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1013.eqiad.wmnet: Maintenance on pc3 * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 43s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-24 == * 19:08 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down * 02:06 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 23s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-23 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 35s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-22 == * 23:39 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 23:39 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 23:39 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 23:39 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 23:38 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 23:37 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 23:37 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 23:37 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 22:20 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 22:12 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 22:11 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 20:29 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 20:28 inflatador: bking@deploy1003 set eqiad prod cirrus `node_concurrent_recoveries` up to 7 from 4 [[phab:T426585|T426585]] * 20:27 inflatador: bking@deploy1003 set codfw prod cirrus `node_concurrent_recoveries` back down to 4 from 7 [[phab:T426585|T426585]] * 18:39 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 17:34 topranks: enable ttl protection on esams CRs IBGP session * 17:28 topranks: enable ttl protection on ulsfo CRs IBGP session * 16:50 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 16:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 16:16 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 16:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 16:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 15:58 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 15:14 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 15:02 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 15:02 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 14:34 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudnet2008-dev.codfw.wmnet * 14:34 andrew@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:34 andrew@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2008-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 14:33 andrew@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2008-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 14:33 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb[1020,1022-1025].eqiad.wmnet * 14:29 andrew@cumin2002: START - Cookbook sre.dns.netbox * 14:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 14:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 14:23 andrew@cumin2002: START - Cookbook sre.hosts.decommission for hosts cloudnet2008-dev.codfw.wmnet * 14:23 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudnet2007-dev.codfw.wmnet * 14:23 andrew@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:23 andrew@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2007-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 14:03 andrew@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2007-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 13:59 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb[1020,1022-1025].eqiad.wmnet * 13:58 andrew@cumin2002: START - Cookbook sre.dns.netbox * 13:53 andrew@cumin2002: START - Cookbook sre.hosts.decommission for hosts cloudnet2007-dev.codfw.wmnet * 13:52 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1018.eqiad.wmnet * 13:50 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 13:50 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 13:46 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1018.eqiad.wmnet * 13:25 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for clouddb1018.eqiad.wmnet * 13:25 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1018.eqiad.wmnet * 13:25 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for 6 hosts * 13:16 inflatador: bking@deploy1002 set search_codfw cluster recovery settings from 4 to 7 [[phab:T426560|T426560]] * 13:15 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for 6 hosts * 13:15 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 13:11 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp5017.eqsin.wmnet<nowiki>}</nowiki> and A:cp * 13:11 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5017.eqsin.wmnet * 13:10 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1017.eqiad.wmnet * 13:09 elukey: uploaded spicerack_12.6.0 to apt.wikimedia.org bookworm-wikimedia * 13:08 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for clouddb1017.eqiad.wmnet * 12:59 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp5017.eqsin.wmnet<nowiki>}</nowiki> and A:cp * 12:57 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp308[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 12:57 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3081.esams.wmnet * 12:54 isaranto@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 12:41 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 12:15 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3080.esams.wmnet * 12:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 12:03 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp308[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[2-3].esams.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3073.esams.wmnet * 11:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2154: Migration of db2154.codfw.wmnet completed * 11:19 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3072.esams.wmnet * 11:15 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 11:11 fnegri@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb1017.eqiad.wmnet with reason: Rebooting clouddb1017 * 11:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1172: Migration of db1172.eqiad.wmnet completed * 11:07 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[2-3].esams.wmnet<nowiki>}</nowiki> and A:cp * 11:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1058.eqiad.wmnet * 11:01 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 11:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3079.esams.wmnet * 10:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1058.eqiad.wmnet * 10:55 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 10:55 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 10:48 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 10:47 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 10:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1024.eqiad.wmnet * 10:43 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:43 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:43 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:42 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:42 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:42 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2154: Migration of db2154.codfw.wmnet completed * 10:42 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:41 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1024.eqiad.wmnet * 10:37 moritzm: remove ganeti1024 foom eqiad Ganeti cluster [[phab:T424680|T424680]] * 10:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2154.codfw.wmnet with OS trixie * 10:31 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2010.codfw.wmnet with OS trixie * 10:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1024.eqiad.wmnet * 10:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1172: Migration of db1172.eqiad.wmnet completed * 10:19 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3078.esams.wmnet * 10:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2154.codfw.wmnet with reason: host reimage * 10:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1172.eqiad.wmnet with OS trixie * 10:15 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1017.eqiad.wmnet * 10:13 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2154.codfw.wmnet with reason: host reimage * 10:07 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 10:06 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 10:06 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3071.esams.wmnet * 09:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1172.eqiad.wmnet with reason: host reimage * 09:56 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2154.codfw.wmnet with OS trixie * 09:55 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage * 09:53 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1172.eqiad.wmnet with reason: host reimage * 09:51 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage * 09:39 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2154: Upgrading db2154.codfw.wmnet * 09:39 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2154: Upgrading db2154.codfw.wmnet * 09:38 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:38 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1172.eqiad.wmnet with OS trixie * 09:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1172: Upgrading db1172.eqiad.wmnet * 09:34 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1172: Upgrading db1172.eqiad.wmnet * 09:34 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:34 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2009.codfw.wmnet with OS trixie * 09:33 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS trixie * 09:26 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 09:26 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 09:26 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3070.esams.wmnet * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 09:16 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie * 09:14 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 09:11 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[6-7].esams.wmnet<nowiki>}</nowiki> and A:cp * 09:11 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3077.esams.wmnet * 09:04 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 09:03 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie * 08:47 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 08:46 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie * 08:40 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 08:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 08:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 08:30 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3076.esams.wmnet * 08:18 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[6-7].esams.wmnet<nowiki>}</nowiki> and A:cp * 08:15 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti1058.eqiad.wmnet on all recursors * 08:15 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:15 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change records for ganeti1058 - cmooney@cumin1003" * 08:15 cmooney@cumin1003: START - Cookbook sre.dns.wipe-cache ganeti1058.eqiad.wmnet on all recursors * 08:15 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change records for ganeti1058 - cmooney@cumin1003" * 08:09 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 08:07 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp306[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 08:07 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3069.esams.wmnet * 08:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 08:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 07:31 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1024.eqiad.wmnet * 07:26 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3068.esams.wmnet * 07:14 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp306[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 07:11 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1057.eqiad.wmnet to cluster eqiad and group A * 07:10 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3075.esams.wmnet<nowiki>}</nowiki> and A:cp * 07:10 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3075.esams.wmnet * 07:06 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1057.eqiad.wmnet to cluster eqiad and group A * 07:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1057.eqiad.wmnet * 07:02 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1057 * 07:01 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1057 * 06:58 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3075.esams.wmnet<nowiki>}</nowiki> and A:cp * 06:58 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3067.esams.wmnet<nowiki>}</nowiki> and A:cp * 06:58 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3067.esams.wmnet * 06:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1057.eqiad.wmnet * 06:46 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3067.esams.wmnet<nowiki>}</nowiki> and A:cp * 06:13 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1024.eqiad.wmnet * 06:08 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1024.eqiad.wmnet * 06:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org * 06:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org * 05:25 marostegui@dns1004: END - running authdns-update * 05:24 marostegui@dns1004: START - running authdns-update * 05:23 marostegui: Failover m5-master [[phab:T426633|T426633]] * 05:19 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy1028.eqiad.wmnet with reason: Reboot * 05:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy2005.codfw.wmnet with reason: Reboot * 05:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1012.eqiad.wmnet * 05:11 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 05:11 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1012.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 05:06 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1012.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 05:03 marostegui@cumin1003: START - Cookbook sre.dns.netbox * 04:56 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1012.eqiad.wmnet == 2026-05-21 == * 23:43 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290954{{!}}Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957{{!}}Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958{{!}}Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960{{!}}Drop unused wpReportIncidentLocalLinks]] (duration: 06m 42s) * 23:38 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 23:38 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1290954{{!}}Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957{{!}}Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958{{!}}Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960{{!}}Drop unused wpReportIncidentLocalLinks]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified * 23:36 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1290954{{!}}Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957{{!}}Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958{{!}}Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960{{!}}Drop unused wpReportIncidentLocalLinks]] * 22:26 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul2002.codfw.wmnet with OS trixie * 22:08 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul2002.codfw.wmnet with reason: host reimage * 22:03 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul2002.codfw.wmnet with reason: host reimage * 22:02 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 21:49 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 21:49 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 21:44 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host zuul2002.codfw.wmnet with OS trixie * 21:25 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:25 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 20:26 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 20:16 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 19:22 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-reboot (exit_code=0) rolling reboot on A:restbase * 19:10 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:59 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:53 papaul: rebooting msw1-codfw * 18:50 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:39 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:52 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:52 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:50 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:49 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:49 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:48 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:46 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 17:46 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 17:43 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:43 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:43 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:42 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:42 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:41 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:41 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:41 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:41 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:41 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:41 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:41 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:40 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:40 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:40 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:39 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 * 17:39 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:38 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down * 17:37 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 17:36 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:36 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:30 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:25 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:25 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:24 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:23 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:22 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1016.eqiad.wmnet * 17:22 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:14 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2031.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:14 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2030.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:13 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1016.eqiad.wmnet * 17:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:08 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repool pc2 ([[phab:T421705|T421705]])', diff saved to https://phabricator.wikimedia.org/P92810 and previous config saved to /var/cache/conftool/dbconfig/20260521-170823-ladsgroup.json * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:07 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2031.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:07 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2030.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:06 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:03 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:03 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:00 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2029 * 16:58 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2031 * 16:58 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:58 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 * 16:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 * 16:55 papaul: rebooting msw-d3-codfw * 16:55 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 16:52 papaul: rebooting msw-c7-codfw * 16:51 papaul: rebooting msw-c6-codfw * 16:48 papaul: rebooting msw-b7-codfw * 16:48 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1014.eqiad.wmnet * 16:45 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1014.eqiad.wmnet * 16:43 papaul: rebooting msw-b6-codfw * 16:40 papaul: rebooting msw-a1-codfw * 16:37 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 16:37 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1014.eqiad.wmnet * 16:37 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:36 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:35 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 16:35 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2030 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2030 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 * 16:34 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 16:34 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:33 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2028 to codfw - jhancock@cumin2002" * 16:33 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2028 to codfw - jhancock@cumin2002" * 16:26 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 16:24 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on pc1022.eqiad.wmnet with reason: Move to nftables * 16:24 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on pc2022.codfw.wmnet with reason: Move to nftables * 16:18 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2048: Repooling * 16:18 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depool pc2 ([[phab:T421705|T421705]])', diff saved to https://phabricator.wikimedia.org/P92807 and previous config saved to /var/cache/conftool/dbconfig/20260521-161808-ladsgroup.json * 16:15 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:15 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:15 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:15 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:52 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 15:42 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es2048: Repooling * 15:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92804 and previous config saved to /var/cache/conftool/dbconfig/20260521-154108-fceratto.json * 15:39 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:38 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:34 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:34 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:34 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:34 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:34 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92803 and previous config saved to /var/cache/conftool/dbconfig/20260521-153400-fceratto.json * 15:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2048.codfw.wmnet with reason: Maintenance * 15:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92802 and previous config saved to /var/cache/conftool/dbconfig/20260521-153331-fceratto.json * 15:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:25 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:24 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:24 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:24 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:24 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040', diff saved to https://phabricator.wikimedia.org/P92801 and previous config saved to /var/cache/conftool/dbconfig/20260521-152323-fceratto.json * 15:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1045.eqiad.wmnet * 15:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1045.eqiad.wmnet * 15:19 claime: Enabling puppet on A:cp-text - [[phab:T426323|T426323]] * 15:15 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1045.eqiad.wmnet * 15:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040', diff saved to https://phabricator.wikimedia.org/P92800 and previous config saved to /var/cache/conftool/dbconfig/20260521-151316-fceratto.json * 15:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1014.eqiad.wmnet * 15:11 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1045.eqiad.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2034.codfw.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2034.codfw.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1037.eqiad.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1037.eqiad.wmnet * 15:07 elukey@cumin1003: END (PASS) - Cookbook sre.misc-clusters.restart-reboot-config-master (exit_code=0) rolling reboot on A:config-master * 15:06 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1014.eqiad.wmnet * 15:05 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors * 15:05 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors * 15:04 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290805{{!}}hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] (duration: 10m 11s) * 15:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92799 and previous config saved to /var/cache/conftool/dbconfig/20260521-150308-fceratto.json * 15:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1037.eqiad.wmnet * 15:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2034.codfw.wmnet * 15:00 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors * 15:00 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors * 15:00 elukey@cumin1003: START - Cookbook sre.misc-clusters.restart-reboot-config-master rolling reboot on A:config-master * 15:00 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 15:00 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-lab1002.eqiad.wmnet * 14:59 elukey@cumin1003: END (PASS) - Cookbook sre.pki.restart-reboot (exit_code=0) rolling reboot on A:pki * 14:57 claime: Disabling puppet on A:cp-text - [[phab:T426323|T426323]] * 14:56 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1290805{{!}}hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:55 klausman@cumin1003: START - Cookbook sre.hosts.reboot-single for host ml-lab1002.eqiad.wmnet * 14:54 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-build1001.eqiad.wmnet * 14:54 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1290805{{!}}hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] * 14:54 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet * 14:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1013.eqiad.wmnet * 14:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1037.eqiad.wmnet * 14:53 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1028.eqiad.wmnet * 14:53 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>ml-serve1001.eqiad.wmnet<nowiki>}</nowiki> and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) * 14:53 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1001.eqiad.wmnet * 14:53 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1001.eqiad.wmnet * 14:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1028.eqiad.wmnet * 14:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92798 and previous config saved to /var/cache/conftool/dbconfig/20260521-145132-fceratto.json * 14:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2040.codfw.wmnet with reason: Maintenance * 14:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92797 and previous config saved to /var/cache/conftool/dbconfig/20260521-145103-fceratto.json * 14:50 klausman@cumin1003: START - Cookbook sre.hosts.reboot-single for host ml-build1001.eqiad.wmnet * 14:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: Migration of db2241.codfw.wmnet completed * 14:48 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1001.eqiad.wmnet * 14:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1013.eqiad.wmnet * 14:46 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1028.eqiad.wmnet * 14:45 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:44 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:42 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1001.eqiad.wmnet * 14:42 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>ml-serve1001.eqiad.wmnet<nowiki>}</nowiki> and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) * 14:42 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1028.eqiad.wmnet * 14:42 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-worker-eqiad * 14:42 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1011.eqiad.wmnet * 14:42 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1011.eqiad.wmnet * 14:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039', diff saved to https://phabricator.wikimedia.org/P92795 and previous config saved to /var/cache/conftool/dbconfig/20260521-144055-fceratto.json * 14:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1012.eqiad.wmnet * 14:38 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet. on all recursors * 14:37 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet. on all recursors * 14:37 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1011.eqiad.wmnet * 14:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1027.eqiad.wmnet * 14:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1027.eqiad.wmnet * 14:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1011.eqiad.wmnet * 14:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1012.eqiad.wmnet * 14:32 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1010.eqiad.wmnet * 14:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1010.eqiad.wmnet * 14:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039', diff saved to https://phabricator.wikimedia.org/P92793 and previous config saved to /var/cache/conftool/dbconfig/20260521-143045-fceratto.json * 14:30 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet. on all recursors * 14:30 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet. on all recursors * 14:29 elukey@cumin1003: START - Cookbook sre.pki.restart-reboot rolling reboot on A:pki * 14:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1027.eqiad.wmnet * 14:27 slyngshede@cumin1003: END (FAIL) - Cookbook sre.cdn.roll-reboot (exit_code=1) rolling reboot on P<nowiki>{</nowiki>cp601[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 14:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1027.eqiad.wmnet * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1054.eqiad.wmnet * 14:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1054.eqiad.wmnet * 14:24 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1010.eqiad.wmnet * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1011.eqiad.wmnet * 14:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92792 and previous config saved to /var/cache/conftool/dbconfig/20260521-142037-fceratto.json * 14:19 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1054.eqiad.wmnet * 14:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1054.eqiad.wmnet * 14:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1053.eqiad.wmnet * 14:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1053.eqiad.wmnet * 14:14 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1010.eqiad.wmnet * 14:14 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1009.eqiad.wmnet * 14:14 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1009.eqiad.wmnet * 14:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:13 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1011.eqiad.wmnet * 14:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:12 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2218: repool after maintenance * 14:11 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1053.eqiad.wmnet * 14:09 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92789 and previous config saved to /var/cache/conftool/dbconfig/20260521-140906-fceratto.json * 14:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2039.codfw.wmnet with reason: Maintenance * 14:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92788 and previous config saved to /var/cache/conftool/dbconfig/20260521-140837-fceratto.json * 14:08 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1009.eqiad.wmnet * 14:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:07 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1053.eqiad.wmnet * 14:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1035.eqiad.wmnet * 14:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1035.eqiad.wmnet * 14:04 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2241: Migration of db2241.codfw.wmnet completed * 14:03 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1009.eqiad.wmnet * 14:03 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1008.eqiad.wmnet * 14:03 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1008.eqiad.wmnet * 14:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2241.codfw.wmnet with OS trixie * 13:59 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 13:59 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1035.eqiad.wmnet * 13:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92786 and previous config saved to /var/cache/conftool/dbconfig/20260521-135830-fceratto.json * 13:58 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1008.eqiad.wmnet * 13:53 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1008.eqiad.wmnet * 13:53 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1007.eqiad.wmnet * 13:53 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1007.eqiad.wmnet * 13:51 Lucas_WMDE: UTC afternoon backport+config window done * 13:51 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290743{{!}}composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347{{!}}Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342{{!}}fix: simplify to show only one icon type for password reveal (T419413)]] (duration: 07m 20s) * 13:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92784 and previous config saved to /var/cache/conftool/dbconfig/20260521-134822-fceratto.json * 13:48 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1007.eqiad.wmnet * 13:47 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 13:46 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Continuing with deployment * 13:45 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 13:45 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Backport for [[gerrit:1290743{{!}}composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347{{!}}Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342{{!}}fix: simplify to show only one icon type for password reveal (T419413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes * 13:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2241.codfw.wmnet with reason: host reimage * 13:44 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 13:43 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1290743{{!}}composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347{{!}}Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342{{!}}fix: simplify to show only one icon type for password reveal (T419413)]] * 13:43 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 13:43 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1007.eqiad.wmnet * 13:42 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1006.eqiad.wmnet * 13:42 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1006.eqiad.wmnet * 13:41 dbrant@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290035{{!}}docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] (duration: 06m 52s) * 13:41 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 13:40 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2241.codfw.wmnet with reason: host reimage * 13:39 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1035.eqiad.wmnet * 13:38 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in codfw/ml-serve-codfw: maintenance * 13:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92782 and previous config saved to /var/cache/conftool/dbconfig/20260521-133815-fceratto.json * 13:37 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1006.eqiad.wmnet * 13:37 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in codfw/ml-serve-codfw: maintenance * 13:37 dbrant@deploy1003: dbrant: Continuing with deployment * 13:36 dbrant@deploy1003: dbrant: Backport for [[gerrit:1290035{{!}}docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1032.eqiad.wmnet * 13:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1032.eqiad.wmnet * 13:35 dbrant@deploy1003: Started scap sync-world: Backport for [[gerrit:1290035{{!}}docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] * 13:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1006.eqiad.wmnet * 13:32 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1005.eqiad.wmnet * 13:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1005.eqiad.wmnet * 13:31 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290014{{!}}Enable AG on phase 2 wikis (T426871)]] (duration: 09m 11s) * 13:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92781 and previous config saved to /var/cache/conftool/dbconfig/20260521-133116-fceratto.json * 13:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1048.eqiad.wmnet with reason: Maintenance * 13:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92780 and previous config saved to /var/cache/conftool/dbconfig/20260521-133048-fceratto.json * 13:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1032.eqiad.wmnet * 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1032.eqiad.wmnet * 13:27 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1005.eqiad.wmnet * 13:27 sbisson@deploy1003: sbisson: Continuing with deployment * 13:27 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2218: repool after maintenance * 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1031.eqiad.wmnet * 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1031.eqiad.wmnet * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:25 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2241.codfw.wmnet with OS trixie * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:24 sbisson@deploy1003: sbisson: Backport for [[gerrit:1290014{{!}}Enable AG on phase 2 wikis (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:23 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: Upgrading db2241.codfw.wmnet * 13:23 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2241: Upgrading db2241.codfw.wmnet * 13:23 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:22 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1290014{{!}}Enable AG on phase 2 wikis (T426871)]] * 13:22 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1005.eqiad.wmnet * 13:22 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1004.eqiad.wmnet * 13:22 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1004.eqiad.wmnet * 13:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040', diff saved to https://phabricator.wikimedia.org/P92778 and previous config saved to /var/cache/conftool/dbconfig/20260521-132041-fceratto.json * 13:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1031.eqiad.wmnet * 13:20 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290088{{!}}Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032{{!}}Enable 'flood' user group at en.wikiversity (T426882)]] (duration: 11m 55s) * 13:18 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet * 13:17 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1018.eqiad.wmnet with OS trixie * 13:16 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1031.eqiad.wmnet * 13:16 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1039: Repooling * 13:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1030.eqiad.wmnet * 13:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1030.eqiad.wmnet * 13:15 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Continuing with deployment * 13:15 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1004.eqiad.wmnet * 13:14 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet * 13:11 eevans@cumin1003: START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:restbase * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . * 13:10 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1004.eqiad.wmnet * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . * 13:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040', diff saved to https://phabricator.wikimedia.org/P92776 and previous config saved to /var/cache/conftool/dbconfig/20260521-131033-fceratto.json * 13:10 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1003.eqiad.wmnet * 13:10 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1003.eqiad.wmnet * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 13:10 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2241 [[phab:T426936|T426936]]', diff saved to https://phabricator.wikimedia.org/P92775 and previous config saved to /var/cache/conftool/dbconfig/20260521-131025-cwilliams.json * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'readability' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'logo-detection' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . * 13:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1030.eqiad.wmnet * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . * 13:10 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Backport for [[gerrit:1290088{{!}}Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032{{!}}Enable 'flood' user group at en.wikiversity (T426882)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'llm' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . * 13:08 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1290088{{!}}Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032{{!}}Enable 'flood' user group at en.wikiversity (T426882)]] * 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet * 13:06 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp601[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 13:06 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3074.esams.wmnet<nowiki>}</nowiki> and A:cp * 13:06 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3074.esams.wmnet * 13:06 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2162 to x3 primary [[phab:T426936|T426936]]', diff saved to https://phabricator.wikimedia.org/P92774 and previous config saved to /var/cache/conftool/dbconfig/20260521-130609-cwilliams.json * 13:04 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 13:04 cezmunsta: Starting x3 codfw failover from db2241 to db2162 - [[phab:T426936|T426936]] * 13:04 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1003.eqiad.wmnet * 13:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1030.eqiad.wmnet * 13:03 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet * 13:00 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 13:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92772 and previous config saved to /var/cache/conftool/dbconfig/20260521-130018-fceratto.json * 12:59 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1003.eqiad.wmnet * 12:59 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1018.eqiad.wmnet with reason: host reimage * 12:59 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1002.eqiad.wmnet * 12:59 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1002.eqiad.wmnet * 12:58 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:57 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:56 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2162 with weight 0 [[phab:T426936|T426936]]', diff saved to https://phabricator.wikimedia.org/P92771 and previous config saved to /var/cache/conftool/dbconfig/20260521-125645-cwilliams.json * 12:56 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Primary switchover x3 [[phab:T426936|T426936]] * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:55 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1029.eqiad.wmnet * 12:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1029.eqiad.wmnet * 12:54 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3074.esams.wmnet<nowiki>}</nowiki> and A:cp * 12:54 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1002.eqiad.wmnet * 12:54 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp600[7-8].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 12:54 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6008.drmrs.wmnet * 12:53 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:52 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1018.eqiad.wmnet with reason: host reimage * 12:51 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:49 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1002.eqiad.wmnet * 12:49 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-worker-eqiad * 12:48 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1029.eqiad.wmnet * 12:48 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3066.esams.wmnet<nowiki>}</nowiki> and A:cp * 12:48 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3066.esams.wmnet * 12:47 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:47 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92770 and previous config saved to /var/cache/conftool/dbconfig/20260521-124707-fceratto.json * 12:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1040.eqiad.wmnet with reason: Maintenance * 12:46 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es1039: Repooling * 12:46 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:45 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1029.eqiad.wmnet * 12:45 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:44 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:43 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:43 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290727{{!}}hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] (duration: 07m 54s) * 12:42 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92768 and previous config saved to /var/cache/conftool/dbconfig/20260521-124014-fceratto.json * 12:39 kharlan@deploy1003: kharlan: Continuing with deployment * 12:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1052.eqiad.wmnet * 12:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1052.eqiad.wmnet * 12:37 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1018.eqiad.wmnet with OS trixie * 12:37 kharlan@deploy1003: kharlan: Backport for [[gerrit:1290727{{!}}hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:36 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:36 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3066.esams.wmnet<nowiki>}</nowiki> and A:cp * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:35 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1290727{{!}}hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:34 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1017.eqiad.wmnet with OS trixie * 12:34 kart_: Updated cxserver to 2026-05-20-034002-production ([[phab:T388690|T388690]], [[phab:T404295|T404295]], [[phab:T391703|T391703]], [[phab:T426605|T426605]]) * 12:34 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet * 12:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1052.eqiad.wmnet * 12:30 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply * 12:30 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply * 12:30 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet * 12:29 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:29 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92767 and previous config saved to /var/cache/conftool/dbconfig/20260521-122905-fceratto.json * 12:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1039.eqiad.wmnet with reason: Maintenance * 12:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92766 and previous config saved to /var/cache/conftool/dbconfig/20260521-122839-fceratto.json * 12:27 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply * 12:27 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply * 12:26 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:23 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-staging-worker * 12:23 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2003.codfw.wmnet * 12:23 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2003.codfw.wmnet * 12:22 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1052.eqiad.wmnet * 12:21 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply * 12:21 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply * 12:21 moritzm: installing nginx security updates * 12:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1051.eqiad.wmnet * 12:20 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in codfw/ml-serve-codfw: maintenance * 12:19 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1017.eqiad.wmnet with reason: host reimage * 12:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1051.eqiad.wmnet * 12:19 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in codfw/ml-serve-codfw: maintenance * 12:19 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in codfw/ml-staging-codfw: maintenance * 12:19 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in codfw/ml-staging-codfw: maintenance * 12:19 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in codfw/ml-staging-codfw: maintenance * 12:18 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in codfw/ml-staging-codfw: maintenance * 12:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047', diff saved to https://phabricator.wikimedia.org/P92765 and previous config saved to /var/cache/conftool/dbconfig/20260521-121832-fceratto.json * 12:17 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2003.codfw.wmnet * 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet * 12:15 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1017.eqiad.wmnet with reason: host reimage * 12:14 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1051.eqiad.wmnet * 12:13 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6007.drmrs.wmnet * 12:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet * 12:10 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1051.eqiad.wmnet * 12:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047', diff saved to https://phabricator.wikimedia.org/P92764 and previous config saved to /var/cache/conftool/dbconfig/20260521-120824-fceratto.json * 12:07 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2003.codfw.wmnet * 12:07 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2002.codfw.wmnet * 12:07 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2002.codfw.wmnet * 12:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1050.eqiad.wmnet * 12:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1050.eqiad.wmnet * 12:02 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp600[7-8].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp601[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6014.drmrs.wmnet * 12:00 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1017.eqiad.wmnet with OS trixie * 12:00 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2002.codfw.wmnet * 11:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1002.wikimedia.org * 11:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92763 and previous config saved to /var/cache/conftool/dbconfig/20260521-115817-fceratto.json * 11:57 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1050.eqiad.wmnet * 11:53 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt1002.wikimedia.org * 11:51 taavi: disabling puppet on C:bird to roll out {{Gerrit|1289919}} * 11:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92762 and previous config saved to /var/cache/conftool/dbconfig/20260521-115112-fceratto.json * 11:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2047.codfw.wmnet with reason: Maintenance * 11:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1050.eqiad.wmnet * 11:50 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2002.codfw.wmnet * 11:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92761 and previous config saved to /var/cache/conftool/dbconfig/20260521-115043-fceratto.json * 11:50 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2001.codfw.wmnet * 11:50 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2001.codfw.wmnet * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1049.eqiad.wmnet * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt2002.wikimedia.org * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1049.eqiad.wmnet * 11:45 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2001.codfw.wmnet * 11:45 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker-exp1001.eqiad.wmnet * 11:44 kartik@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 11:44 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1049.eqiad.wmnet * 11:43 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt2002.wikimedia.org * 11:42 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet * 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet * 11:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037', diff saved to https://phabricator.wikimedia.org/P92760 and previous config saved to /var/cache/conftool/dbconfig/20260521-114036-fceratto.json * 11:39 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker-exp1001.eqiad.wmnet * 11:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker-exp2001.codfw.wmnet * 11:38 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host testreduce1002.eqiad.wmnet * 11:37 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1049.eqiad.wmnet * 11:36 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet * 11:36 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet * 11:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1038.eqiad.wmnet * 11:35 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2001.codfw.wmnet * 11:35 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-staging-worker * 11:35 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet * 11:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1038.eqiad.wmnet * 11:34 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host testreduce1002.eqiad.wmnet * 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker-exp2001.codfw.wmnet * 11:32 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet * 11:31 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet * 11:30 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt-staging2001.codfw.wmnet * 11:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037', diff saved to https://phabricator.wikimedia.org/P92759 and previous config saved to /var/cache/conftool/dbconfig/20260521-113028-fceratto.json * 11:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2014.codfw.wmnet * 11:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1038.eqiad.wmnet * 11:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt-staging2001.codfw.wmnet * 11:26 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet * 11:24 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1038.eqiad.wmnet * 11:24 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1034.eqiad.wmnet * 11:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1034.eqiad.wmnet * 11:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2014.codfw.wmnet * 11:20 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6013.drmrs.wmnet * 11:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92758 and previous config saved to /var/cache/conftool/dbconfig/20260521-112021-fceratto.json * 11:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1034.eqiad.wmnet * 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling reboot on A:ldap-replicas-eqiad * 11:13 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2013.codfw.wmnet * 11:11 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1034.eqiad.wmnet * 11:09 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp601[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92757 and previous config saved to /var/cache/conftool/dbconfig/20260521-110851-fceratto.json * 11:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2037.codfw.wmnet with reason: Maintenance * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92756 and previous config saved to /var/cache/conftool/dbconfig/20260521-110822-fceratto.json * 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1033.eqiad.wmnet * 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1033.eqiad.wmnet * 11:05 jmm@cumin2002: START - Cookbook sre.ldap.roll-restart-reboot-replica rolling reboot on A:ldap-replicas-eqiad * 11:05 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2013.codfw.wmnet * 11:04 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp600[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 11:04 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6006.drmrs.wmnet * 11:02 jmm@cumin2002: END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling reboot on A:ldap-replicas-codfw * 11:00 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1033.eqiad.wmnet * 10:59 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1016.eqiad.wmnet with reason: host reimage * 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036', diff saved to https://phabricator.wikimedia.org/P92753 and previous config saved to /var/cache/conftool/dbconfig/20260521-105815-fceratto.json * 10:57 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1033.eqiad.wmnet * 10:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1044.eqiad.wmnet * 10:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1044.eqiad.wmnet * 10:55 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1016.eqiad.wmnet with reason: host reimage * 10:54 jmm@cumin2002: START - Cookbook sre.ldap.roll-restart-reboot-replica rolling reboot on A:ldap-replicas-codfw * 10:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2012.codfw.wmnet * 10:51 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:51 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:51 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1044.eqiad.wmnet * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036', diff saved to https://phabricator.wikimedia.org/P92752 and previous config saved to /var/cache/conftool/dbconfig/20260521-104807-fceratto.json * 10:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2012.codfw.wmnet * 10:46 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1044.eqiad.wmnet * 10:44 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290709{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] (duration: 08m 02s) * 10:43 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:41 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:40 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet * 10:40 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:39 jiji@deploy1003: jiji: Continuing with deployment * 10:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92751 and previous config saved to /var/cache/conftool/dbconfig/20260521-103759-fceratto.json * 10:37 jiji@deploy1003: jiji: Backport for [[gerrit:1290709{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:36 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1290709{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] * 10:35 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet * 10:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1043.eqiad.wmnet * 10:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1043.eqiad.wmnet * 10:34 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:29 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 10:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1043.eqiad.wmnet * 10:27 dcausse: [[phab:T423993|T423993]]: reindexing all archive indices * 10:27 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . * 10:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92749 and previous config saved to /var/cache/conftool/dbconfig/20260521-102630-fceratto.json * 10:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance * 10:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1043.eqiad.wmnet * 10:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92748 and previous config saved to /var/cache/conftool/dbconfig/20260521-102601-fceratto.json * 10:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2011.codfw.wmnet * 10:24 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6005.drmrs.wmnet * 10:22 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1042.eqiad.wmnet * 10:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1042.eqiad.wmnet * 10:17 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2011.codfw.wmnet * 10:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1042.eqiad.wmnet * 10:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047', diff saved to https://phabricator.wikimedia.org/P92747 and previous config saved to /var/cache/conftool/dbconfig/20260521-101552-fceratto.json * 10:15 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:14 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . * 10:13 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1042.eqiad.wmnet * 10:13 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1041.eqiad.wmnet * 10:12 moritzm: installing postgresql security updates * 10:12 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp600[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 10:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1041.eqiad.wmnet * 10:10 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet * 10:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org * 10:09 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 10:08 fnegri@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb1013.eqiad.wmnet * 10:08 fnegri@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb1013.eqiad.wmnet * 10:07 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1013.eqiad.wmnet * 10:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1041.eqiad.wmnet * 10:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047', diff saved to https://phabricator.wikimedia.org/P92746 and previous config saved to /var/cache/conftool/dbconfig/20260521-100545-fceratto.json * 10:05 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet * 10:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1041.eqiad.wmnet * 10:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1040.eqiad.wmnet * 10:04 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet * 10:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1040.eqiad.wmnet * 10:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org * 10:01 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1013.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 10:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1040.eqiad.wmnet * 10:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org * 09:59 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet * 09:58 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-codfw * 09:58 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2005.codfw.wmnet * 09:58 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2005.codfw.wmnet * 09:56 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1040.eqiad.wmnet * 09:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1039.eqiad.wmnet * 09:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1039.eqiad.wmnet * 09:56 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 09:56 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:55 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ml-serve1013.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 09:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92745 and previous config saved to /var/cache/conftool/dbconfig/20260521-095536-fceratto.json * 09:54 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1384.eqiad.wmnet * 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org * 09:54 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:54 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:52 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2005.codfw.wmnet * 09:52 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2005.codfw.wmnet * 09:52 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply * 09:52 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2004.codfw.wmnet * 09:52 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2004.codfw.wmnet * 09:51 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply * 09:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1039.eqiad.wmnet * 09:49 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1384.eqiad.wmnet * 09:49 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 09:49 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1383.eqiad.wmnet * 09:48 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1039.eqiad.wmnet * 09:48 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1036.eqiad.wmnet * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92744 and previous config saved to /var/cache/conftool/dbconfig/20260521-094829-fceratto.json * 09:48 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1036.eqiad.wmnet * 09:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1047.eqiad.wmnet with reason: Maintenance * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92743 and previous config saved to /var/cache/conftool/dbconfig/20260521-094801-fceratto.json * 09:47 fnegri@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1013.eqiad.wmnet * 09:47 fnegri@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1013.eqiad.wmnet with reason: Rebooting clouddb1013 [[phab:T426563|T426563]] * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2004.codfw.wmnet * 09:45 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2004.codfw.wmnet * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet * 09:45 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet * 09:45 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet * 09:44 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1383.eqiad.wmnet * 09:44 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 09:44 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1382.eqiad.wmnet * 09:42 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2002.codfw.wmnet * 09:40 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1036.eqiad.wmnet * 09:39 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet * 09:38 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1382.eqiad.wmnet * 09:38 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1381.eqiad.wmnet * 09:38 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1036.eqiad.wmnet * 09:38 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2003.codfw.wmnet * 09:38 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2003.codfw.wmnet * 09:38 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2002.codfw.wmnet * 09:38 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2002.codfw.wmnet * 09:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037', diff saved to https://phabricator.wikimedia.org/P92742 and previous config saved to /var/cache/conftool/dbconfig/20260521-093754-fceratto.json * 09:37 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 09:37 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet * 09:37 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet * 09:37 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet * 09:37 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet * 09:36 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host build2002.codfw.wmnet * 09:36 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 09:35 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp601[1-2].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 09:35 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6012.drmrs.wmnet * 09:34 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet * 09:33 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet * 09:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1381.eqiad.wmnet * 09:33 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1380.eqiad.wmnet * 09:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1023.eqiad.wmnet * 09:31 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet * 09:31 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2002.codfw.wmnet * 09:31 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2002.codfw.wmnet * 09:31 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2001.codfw.wmnet * 09:31 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2001.codfw.wmnet * 09:30 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet * 09:30 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet * 09:30 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet * 09:30 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet * 09:29 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet * 09:29 jayme@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad * 09:29 jayme@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw * 09:29 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet * 09:28 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet * 09:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037', diff saved to https://phabricator.wikimedia.org/P92741 and previous config saved to /var/cache/conftool/dbconfig/20260521-092746-fceratto.json * 09:27 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1380.eqiad.wmnet * 09:27 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1379.eqiad.wmnet * 09:27 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet * 09:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1023.eqiad.wmnet * 09:25 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet * 09:24 jayme@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw * 09:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1056.eqiad.wmnet to cluster eqiad and group A * 09:23 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet * 09:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad * 09:22 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1379.eqiad.wmnet * 09:22 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1378.eqiad.wmnet * 09:21 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2001.codfw.wmnet * 09:21 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2001.codfw.wmnet * 09:21 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-codfw * 09:21 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1056.eqiad.wmnet to cluster eqiad and group A * 09:20 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 09:18 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-jumbo1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:18 moritzm: remove ganeti1023 foom eqiad Ganeti cluster [[phab:T424680|T424680]] * 09:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92740 and previous config saved to /var/cache/conftool/dbconfig/20260521-091738-fceratto.json * 09:16 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1378.eqiad.wmnet * 09:16 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1377.eqiad.wmnet * 09:12 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1377.eqiad.wmnet * 09:12 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1376.eqiad.wmnet * 09:07 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1036: Repooling * 09:07 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1376.eqiad.wmnet * 09:07 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1375.eqiad.wmnet * 09:06 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92738 and previous config saved to /var/cache/conftool/dbconfig/20260521-090609-fceratto.json * 09:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1037.eqiad.wmnet with reason: Maintenance * 09:02 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1375.eqiad.wmnet * 09:01 btullis@cumin1003: START - Cookbook sre.hosts.provision for host kafka-jumbo1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:55 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6011.drmrs.wmnet * 08:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1023.eqiad.wmnet * 08:47 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 08:47 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1256: Migration of db1256.eqiad.wmnet completed * 08:44 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp601[1-2].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 08:42 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp600[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 08:42 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6004.drmrs.wmnet * 08:37 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es1036: Repooling * 08:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92733 and previous config saved to /var/cache/conftool/dbconfig/20260521-082951-fceratto.json * 08:29 hashar@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.3 refs [[phab:T423912|T423912]] * 08:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92731 and previous config saved to /var/cache/conftool/dbconfig/20260521-081642-fceratto.json * 08:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1036.eqiad.wmnet with reason: Maintenance * 08:02 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1256: Migration of db1256.eqiad.wmnet completed * 08:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6003.drmrs.wmnet * 08:00 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1256.eqiad.wmnet with OS trixie * 07:52 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp600[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:51 marostegui@dns1004: END - running authdns-update * 07:50 marostegui@dns1004: START - running authdns-update * 07:48 marostegui: Failover m3-master [[phab:T426633|T426633]] * 07:47 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1023.eqiad.wmnet * 07:46 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp6010.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:46 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6010.drmrs.wmnet * 07:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster1005.eqiad.wmnet to plain * 07:44 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster1005.eqiad.wmnet to plain * 07:43 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1256.eqiad.wmnet with reason: host reimage * 07:42 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster1005.eqiad.wmnet to drbd * 07:38 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1256.eqiad.wmnet with reason: host reimage * 07:35 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp6010.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:35 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp6002.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:35 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6002.drmrs.wmnet * 07:27 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster1005.eqiad.wmnet to drbd * 07:24 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp6002.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:24 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1256.eqiad.wmnet with OS trixie * 07:22 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1256: Upgrading db1256.eqiad.wmnet * 07:21 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1256: Upgrading db1256.eqiad.wmnet * 07:21 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to plain * 07:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to plain * 07:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbproxy1025.eqiad.wmnet with reason: Rebooting * 07:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to drbd * 06:54 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to drbd * 06:53 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to plain * 06:52 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to plain * 06:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to drbd * 06:42 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists1004.wikimedia.org * 06:40 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org * 06:39 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1003.eqiad.wmnet * 06:34 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org * 06:34 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host lists1004.wikimedia.org * 06:33 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host vrts1003.eqiad.wmnet * 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to drbd * 06:23 arnaudb@cumin1003: END (FAIL) - Cookbook sre.gerrit.reboot-gerrit (exit_code=99) Rebooting Gerrit on gerrit2003 * 06:22 arnaudb@cumin1003: START - Cookbook sre.gerrit.reboot-gerrit Rebooting Gerrit on gerrit2003 * 06:15 marostegui@dns1004: END - running authdns-update * 06:14 marostegui: Failover m2-master [[phab:T426633|T426633]] * 06:13 marostegui@dns1004: START - running authdns-update * 05:39 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1012 from dbctl [[phab:T426930|T426930]]', diff saved to https://phabricator.wikimedia.org/P92728 and previous config saved to /var/cache/conftool/dbconfig/20260521-053858-marostegui.json * 05:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc2 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92727 and previous config saved to /var/cache/conftool/dbconfig/20260521-053000-marostegui.json * 05:29 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1022 to pc2 master [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92726 and previous config saved to /var/cache/conftool/dbconfig/20260521-052905-marostegui.json * 05:21 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc1012.eqiad.wmnet with reason: Cloning * 02:41 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on planet1003.eqiad.wmnet with reason: debug wip * 02:11 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 29s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:29 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1027.eqiad.wmnet * 01:22 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1027.eqiad.wmnet * 00:55 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 3m5rrckq9hu7zgwv6y0yotn5siwcp6q 2426652 2426651 2026-06-14T11:02:58Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2426652 wikitext text/x-wiki == 2026-06-14 == * 11:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 11:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 11:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:06 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-06-13 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 35s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-06-12 == * 19:54 dwisehaupt@dns1004: END - running authdns-update * 19:52 dwisehaupt@dns1004: START - running authdns-update * 18:33 dwisehaupt@dns1006: END - running authdns-update * 18:32 dwisehaupt@dns1006: START - running authdns-update * 16:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:10 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:10 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:59 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 15:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:43 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1301371{{!}}Hotfix for T428620 (T428620)]] (duration: 11m 17s) * 14:36 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 14:35 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1301371{{!}}Hotfix for T428620 (T428620)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:31 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1301371{{!}}Hotfix for T428620 (T428620)]] * 14:29 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:28 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 13:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 12:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 12:22 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:22 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:22 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:22 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 12:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 12:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 12:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 12:04 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:04 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:04 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 12:02 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of prometheus5003.eqsin.wmnet to drbd * 12:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus5003.eqsin.wmnet to drbd * 11:40 moritzm: installing Linux 5.10.257 on Bullseye hosts * 11:36 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 11:35 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:24 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 11:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:56 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 10:56 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 10:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:49 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 10:49 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 10:40 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply * 10:37 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply * 10:36 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply * 10:35 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply * 10:35 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply * 10:35 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply * 10:12 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 10:12 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 10:08 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 09:59 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 09:58 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 09:57 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 06:13 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) * 06:11 jmm@cumin2002: START - Cookbook sre.puppet.disable-merges * 03:07 ryankemper: [[phab:T427951|T427951]] sorry, `[eqiad,codfw].mediawiki.page_html_content_change.rc0` (accidentally a word) * 03:06 ryankemper: [[phab:T427951|T427951]] Deleted all 20 unused dev/test topics on kafka-jumbo (verified empty first); 2 (`[eqiad,codfw]page_html_content_change.rc0`) were immediately auto-recreated empty by a still-running `dse-k8s` enrichment consumer; awaiting owner confirmation before final re-delete * 02:01 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 01m 13s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:00 bblack@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () == 2026-06-11 == * 22:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 22:26 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 22:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 22:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 22:05 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300906{{!}}Restore MediaViewer toggle in Special:Preferences (T428742)]] (duration: 30m 51s) * 21:58 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host releases2003.codfw.wmnet with OS trixie * 21:52 egardner@deploy1003: egardner: Continuing with deployment * 21:51 egardner@deploy1003: egardner: Backport for [[gerrit:1300906{{!}}Restore MediaViewer toggle in Special:Preferences (T428742)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:34 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1300906{{!}}Restore MediaViewer toggle in Special:Preferences (T428742)]] * 21:34 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet with reason: host reimage * 21:29 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300913{{!}}Avoid the escaping from nowiki processing (T398967)]] (duration: 09m 09s) * 21:28 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on releases2003.codfw.wmnet with reason: host reimage * 21:25 arlolra@deploy1003: arlolra: Continuing with deployment * 21:22 arlolra@deploy1003: arlolra: Backport for [[gerrit:1300913{{!}}Avoid the escaping from nowiki processing (T398967)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:20 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1300913{{!}}Avoid the escaping from nowiki processing (T398967)]] * 21:07 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300911{{!}}hCaptcha: Enable for badlogin for all small wikis (T426875)]], [[gerrit:1300905{{!}}RadioRangeBallot: Fix strict mode issue (T428947)]] (duration: 10m 43s) * 21:06 bblack@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text and not P<nowiki>{</nowiki>cp7008*<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 21:01 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 21:00 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300911{{!}}hCaptcha: Enable for badlogin for all small wikis (T426875)]], [[gerrit:1300905{{!}}RadioRangeBallot: Fix strict mode issue (T428947)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:56 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300911{{!}}hCaptcha: Enable for badlogin for all small wikis (T426875)]], [[gerrit:1300905{{!}}RadioRangeBallot: Fix strict mode issue (T428947)]] * 20:51 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300842{{!}}Donor Delight Badge: Unify on "Remove badge" language across treatments (T427313)]], [[gerrit:1300843{{!}}[A11y] Donor Badge: Remove Badge button disappears too quickly (T428646)]], [[gerrit:1300896{{!}}Donor Delight Badge, styles: Amending to final design review feedback (T427313)]] (duration: 34m 10s) * 20:39 jdrewniak@deploy1003: annet, jdrewniak: Continuing with deployment * 20:35 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host releases2003.codfw.wmnet with OS trixie * 20:34 jdrewniak@deploy1003: annet, jdrewniak: Backport for [[gerrit:1300842{{!}}Donor Delight Badge: Unify on "Remove badge" language across treatments (T427313)]], [[gerrit:1300843{{!}}[A11y] Donor Badge: Remove Badge button disappears too quickly (T428646)]], [[gerrit:1300896{{!}}Donor Delight Badge, styles: Amending to final design review feedback (T427313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug * 20:17 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1300842{{!}}Donor Delight Badge: Unify on "Remove badge" language across treatments (T427313)]], [[gerrit:1300843{{!}}[A11y] Donor Badge: Remove Badge button disappears too quickly (T428646)]], [[gerrit:1300896{{!}}Donor Delight Badge, styles: Amending to final design review feedback (T427313)]] * 19:12 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 18:12 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 18:12 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 17:52 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300865{{!}}UploadWizard.config.php: Fix cc-by-4.0-heirs msg issue (T428935 T405146)]] (duration: 08m 15s) * 17:48 reedy@deploy1003: reedy: Continuing with deployment * 17:46 reedy@deploy1003: reedy: Backport for [[gerrit:1300865{{!}}UploadWizard.config.php: Fix cc-by-4.0-heirs msg issue (T428935 T405146)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:44 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1300865{{!}}UploadWizard.config.php: Fix cc-by-4.0-heirs msg issue (T428935 T405146)]] * 17:26 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 17:25 blake@deploy1003: Scap cancelled without rolling back. * 17:25 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 17:24 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 17:24 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 17:24 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 17:24 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 17:23 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 17:23 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 17:23 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 17:23 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 17:23 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 17:23 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 17:20 blake@deploy1003: blake: apache config update ([[phab:T428772|T428772]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:20 blake@deploy1003: Started scap sync-world: apache config update ([[phab:T428772|T428772]]) * 17:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 17:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2212: Migration of db2212.codfw.wmnet completed * 17:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 17:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1235: Migration of db1235.eqiad.wmnet completed * 17:08 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 16:45 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:43 dzahn@dns1005: END - running authdns-update * 16:42 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:41 dzahn@dns1005: START - running authdns-update * 16:41 mutante: releases.wikimedia.org - switching backend from codfw to eqiad - releases1003 is now the source of rsync for uploaded releases files (use releases.discovery.wmnet to not have to think about it) - [[phab:T418299|T418299]] * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts rdb2007.codfw.wmnet * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts rdb1011.eqiad.wmnet * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) * 16:34 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb2009.codfw.wmnet * 16:34 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:34 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2009.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:33 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Migration of db2212.codfw.wmnet completed * 16:27 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2009.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:27 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1235: Migration of db1235.eqiad.wmnet completed * 16:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2212.codfw.wmnet with OS trixie * 16:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1235.eqiad.wmnet with OS trixie * 16:13 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:07 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:06 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 16:05 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 16:05 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 16:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 16:04 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2212.codfw.wmnet with reason: host reimage * 16:01 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 16:01 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:01 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 16:01 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 16:00 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 16:00 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 16:00 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 16:00 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2212.codfw.wmnet with reason: host reimage * 15:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1235.eqiad.wmnet with reason: host reimage * 15:58 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 15:58 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 15:57 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 15:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 15:57 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 15:57 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 15:56 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2009.codfw.wmnet * 15:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:55 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb1011.eqiad.wmnet * 15:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 15:55 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2007.codfw.wmnet * 15:54 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 15:54 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1235.eqiad.wmnet with reason: host reimage * 15:54 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 15:53 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 15:53 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 15:40 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 15:40 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2212.codfw.wmnet with OS trixie * 15:39 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 15:39 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1235.eqiad.wmnet with OS trixie * 15:36 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 15:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1235: Upgrading db1235.eqiad.wmnet * 15:35 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 15:35 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1235: Upgrading db1235.eqiad.wmnet * 15:35 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:32 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 15:32 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:31 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 15:30 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300822{{!}}T428849: temporarily disable noisy warnings in HandleParsoidSectionLinks (T428849 T417530)]] (duration: 11m 29s) * 15:27 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2212: Upgrading db2212.codfw.wmnet * 15:26 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2212: Upgrading db2212.codfw.wmnet * 15:26 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:26 cscott@deploy1003: cscott: Continuing with deployment * 15:26 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1235: Upgrading db1235.eqiad.wmnet * 15:25 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1235: Upgrading db1235.eqiad.wmnet * 15:25 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:21 cscott@deploy1003: cscott: Backport for [[gerrit:1300822{{!}}T428849: temporarily disable noisy warnings in HandleParsoidSectionLinks (T428849 T417530)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:19 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1300822{{!}}T428849: temporarily disable noisy warnings in HandleParsoidSectionLinks (T428849 T417530)]] * 15:18 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 15:17 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 15:13 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 15:13 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 15:13 moritzm: installing libdbi-perl security updates * 14:53 moritzm: installing Bind security updates (just client-side tools/libraries) * 14:51 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry * 14:48 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry * 14:43 moritzm: installing Poppler security updates * 14:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:33 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 14:32 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 14:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1234: Migration of db1234.eqiad.wmnet completed * 14:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin02 and group 01 * 14:24 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin02 and group 01 * 14:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:18 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet * 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet * 14:00 Lucas_WMDE: UTC afternoon backport+config window done * 13:58 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300733{{!}}stream: webrequest.page_view_stats.dev0 (T428725)]] (duration: 08m 12s) * 13:57 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp5024.* * 13:55 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: name=cp5024.* * 13:55 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp5020.* * 13:54 javiermonton@deploy1003: javiermonton: Continuing with deployment * 13:52 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1300733{{!}}stream: webrequest.page_view_stats.dev0 (T428725)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:51 slyngshede@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P<nowiki>{</nowiki>lvs5004*<nowiki>}</nowiki> and A:liberica * 13:50 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1300733{{!}}stream: webrequest.page_view_stats.dev0 (T428725)]] * 13:50 slyngshede@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P<nowiki>{</nowiki>lvs5004*<nowiki>}</nowiki> and A:liberica * 13:50 slyngs: reloading liberica config on lvs5004 * 13:50 moritzm: installing openssl security updates * 13:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:46 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 13:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm * 13:46 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1234: Migration of db1234.eqiad.wmnet completed * 13:46 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 13:45 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 13:45 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 13:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2202.codfw.wmnet with OS trixie * 13:43 alexsanford@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298890{{!}}Add 2FA enforcement demotion config for phase 3 groups (T423120)]] (duration: 07m 19s) * 13:39 alexsanford@deploy1003: alexsanford: Continuing with deployment * 13:38 alexsanford@deploy1003: alexsanford: Backport for [[gerrit:1298890{{!}}Add 2FA enforcement demotion config for phase 3 groups (T423120)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:36 alexsanford@deploy1003: Started scap sync-world: Backport for [[gerrit:1298890{{!}}Add 2FA enforcement demotion config for phase 3 groups (T423120)]] * 13:36 slyngshede@dns1004: END - running authdns-update * 13:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1234.eqiad.wmnet with OS trixie * 13:34 moritzm: installing dovecot security updates * 13:34 slyngshede@dns1004: START - running authdns-update * 13:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300787{{!}}hCaptcha: Enable for MobileFrontend on all group1 wikis (T425940)]] (duration: 06m 59s) * 13:29 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 13:29 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 13:29 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 13:29 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 13:28 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 13:28 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 13:28 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 13:27 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300787{{!}}hCaptcha: Enable for MobileFrontend on all group1 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:26 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2202.codfw.wmnet with reason: host reimage * 13:25 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300787{{!}}hCaptcha: Enable for MobileFrontend on all group1 wikis (T425940)]] * 13:25 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/Android_FAQ 'Wikimedia Apps/FAQ/Android' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:24 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/Android_FAQ 'Wikimedia Apps/FAQ/Android' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:22 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300736{{!}}fix: correct intake-url and payload type for NCS experiment events (T422295)]] (duration: 06m 51s) * 13:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage * 13:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1234.eqiad.wmnet with reason: host reimage * 13:18 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Continuing with deployment * 13:18 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2202.codfw.wmnet with reason: host reimage * 13:18 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Backport for [[gerrit:1300736{{!}}fix: correct intake-url and payload type for NCS experiment events (T422295)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:18 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 13:17 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 13:16 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1300736{{!}}fix: correct intake-url and payload type for NCS experiment events (T422295)]] * 13:15 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage * 13:14 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/Android_FAQ 'Wikimedia Apps/FAQ/Android' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:13 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:13 gkyziridis@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300731{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] (duration: 08m 47s) * 13:13 andrewbogott: sudo -i reprepro --noskipold --component thirdparty/openstack-trixie-flamingo-backports update trixie-wikimedia * 13:12 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1234.eqiad.wmnet with reason: host reimage * 13:12 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 13:12 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/iOS_FAQ 'Wikimedia Apps/FAQ/iOS' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:12 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 13:12 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 13:11 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 13:11 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 13:11 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 13:11 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 13:11 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 13:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 13:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 13:09 gkyziridis@deploy1003: gkyziridis: Continuing with deployment * 13:06 gkyziridis@deploy1003: gkyziridis: Backport for [[gerrit:1300731{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:06 claime: echo 'https://api.wikimedia.org/service/lw/specs/openapi.yaml' {{!}} mwscript-k8s --attach -- purgeList.php * 13:04 gkyziridis@deploy1003: Started scap sync-world: Backport for [[gerrit:1300731{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] * 13:02 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2202.codfw.wmnet with OS trixie * 13:00 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:57 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1234.eqiad.wmnet with OS trixie * 12:55 moritzm: installing Exim security updates on Bullseye * 12:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ganeti5006 * 12:47 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti5006 * 12:46 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti5006 * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti5006.eqsin.wmnet 9.0.132.10.in-addr.arpa 9.0.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 12:46 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache ganeti5006.eqsin.wmnet 9.0.132.10.in-addr.arpa 9.0.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5006 - jmm@cumin2002" * 12:46 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5006 - jmm@cumin2002" * 12:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1234: Upgrading db1234.eqiad.wmnet * 12:44 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1234: Upgrading db1234.eqiad.wmnet * 12:44 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2188: Migration of db2188.codfw.wmnet completed * 12:29 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "UX improvements - oblivian@cumin1003" * 12:29 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: UX improvements - oblivian@cumin1003 * 12:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1232: Migration of db1232.eqiad.wmnet completed * 12:28 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: UX improvements - oblivian@cumin1003 * 12:28 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "UX improvements - oblivian@cumin1003" * 12:27 jmm@cumin2002: START - Cookbook sre.dns.netbox * 12:26 jmm@cumin2002: START - Cookbook sre.hosts.move-vlan for host ganeti5006 * 12:26 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm * 12:21 moritzm: remove ganeti5006 from eqsin cluster for reimage [[phab:T428229|T428229]] * 12:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet * 12:10 moritzm: installing openjdk-21 security updates on Bookworm * 12:03 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300764{{!}}Remove GrowthExperiments extension from closed wikis (T428884)]] (duration: 06m 53s) * 11:59 urbanecm@deploy1003: urbanecm: Continuing with deployment * 11:58 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1300764{{!}}Remove GrowthExperiments extension from closed wikis (T428884)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:56 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1300764{{!}}Remove GrowthExperiments extension from closed wikis (T428884)]] * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb1012.eqiad.wmnet * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb2010.codfw.wmnet * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:48 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2010.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 11:46 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:46 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb2008.codfw.wmnet * 11:46 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:46 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2188: Migration of db2188.codfw.wmnet completed * 11:44 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 11:43 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:43 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2010.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 11:43 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1232: Migration of db1232.eqiad.wmnet completed * 11:38 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:37 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 11:37 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 11:36 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 11:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2188.codfw.wmnet with OS trixie * 11:35 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb1012.eqiad.wmnet * 11:34 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2008.codfw.wmnet * 11:34 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2010.codfw.wmnet * 11:33 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 11:32 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 11:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1232.eqiad.wmnet with OS trixie * 11:27 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2002.codfw.wmnet * 11:25 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300749{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300751{{!}}hCaptcha: Enable for DiscussionTools on all wikis (T426039)]] (duration: 08m 38s) * 11:21 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 11:19 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300749{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300751{{!}}hCaptcha: Enable for DiscussionTools on all wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2188.codfw.wmnet with reason: host reimage * 11:17 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300749{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300751{{!}}hCaptcha: Enable for DiscussionTools on all wikis (T426039)]] * 11:15 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2188.codfw.wmnet with reason: host reimage * 11:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1232.eqiad.wmnet with reason: host reimage * 11:13 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc2002.codfw.wmnet * 11:13 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet * 11:11 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 11:09 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2001.codfw.wmnet * 11:09 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1232.eqiad.wmnet with reason: host reimage * 11:08 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet * 11:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:04 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc2001.codfw.wmnet * 11:04 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host testreduce1002.eqiad.wmnet * 11:04 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:02 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1262.eqiad.wmnet with reason: crash * 11:00 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 11:00 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host testreduce1002.eqiad.wmnet * 10:59 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 10:59 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 10:58 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 10:55 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2188.codfw.wmnet with OS trixie * 10:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2188: Upgrading db2188.codfw.wmnet * 10:52 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2188: Upgrading db2188.codfw.wmnet * 10:52 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:52 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1232.eqiad.wmnet with OS trixie * 10:48 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1232: Upgrading db1232.eqiad.wmnet * 10:48 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1232: Upgrading db1232.eqiad.wmnet * 10:48 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:33 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:32 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:31 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300734{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300727{{!}}hCaptcha: Enable for DiscussionTools on group 1 wikis (T426039)]] (duration: 11m 01s) * 10:26 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:23 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:23 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:22 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300734{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300727{{!}}hCaptcha: Enable for DiscussionTools on group 1 wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:20 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300734{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300727{{!}}hCaptcha: Enable for DiscussionTools on group 1 wikis (T426039)]] * 10:18 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:18 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:10 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 10:10 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 10:09 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2045.codfw.wmnet with OS trixie * 10:09 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repool es2046', diff saved to https://phabricator.wikimedia.org/P94069 and previous config saved to /var/cache/conftool/dbconfig/20260611-100221-marostegui.json * 10:01 marostegui@cumin1003: dbctl commit (dc=all): 'Depool es2046', diff saved to https://phabricator.wikimedia.org/P94068 and previous config saved to /var/cache/conftool/dbconfig/20260611-100145-marostegui.json * 10:01 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:59 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300580{{!}}ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] (duration: 15m 41s) * 09:54 jiji@deploy1003: jiji: Continuing with deployment * 09:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2045.codfw.wmnet with reason: host reimage * 09:45 jiji@deploy1003: jiji: Backport for [[gerrit:1300580{{!}}ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:43 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1300580{{!}}ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] * 09:42 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2045.codfw.wmnet with reason: host reimage * 09:37 elukey: uploaded spicerack_12.8.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia * 09:26 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS trixie * 09:26 marostegui@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host es2045.codfw.wmnet with OS bookworm * 09:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2176: Migration of db2176.codfw.wmnet completed * 09:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1219: Migration of db1219.eqiad.wmnet completed * 09:11 claime: cumin -x 'A:swift-fe' "disable-puppet 'Disabling puppet for ratelimit deploy - cgoubert'" * 08:57 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS bookworm * 08:39 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2176: Migration of db2176.codfw.wmnet completed * 08:34 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94055) * 08:34 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1219: Migration of db1219.eqiad.wmnet completed * 08:33 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94053) * 08:30 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T428823|T428823]] (duration: 01m 18s) * 08:29 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T428823|T428823]] * 08:27 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2176.codfw.wmnet with OS trixie * 08:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Migration to 10.11.17 * 08:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 08:25 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 08:25 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool pc1021: Migration to 10.11.17 * 08:25 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94052) * 08:24 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): Testing upgrade for [[phab:T428823|T428823]] (duration: 01m 17s) * 08:23 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): Testing upgrade for [[phab:T428823|T428823]] * 08:22 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94051) * 08:22 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1219.eqiad.wmnet with OS trixie * 08:17 moritzm: installing PHP 8.2 security updates * 08:15 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 08:14 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 08:11 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 08:11 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 08:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2176.codfw.wmnet with reason: host reimage * 08:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1013.eqiad.wmnet with OS trixie * 08:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5004.eqsin.wmnet to cluster eqsin02 and group 01 * 08:06 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 08:06 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 08:05 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2021.codfw.wmnet,pc1021.eqiad.wmnet with reason: upgrade * 08:05 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1219.eqiad.wmnet with reason: host reimage * 08:05 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5004.eqsin.wmnet to cluster eqsin02 and group 01 * 08:05 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 08:05 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 08:04 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2176.codfw.wmnet with reason: host reimage * 08:04 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 08:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 08:03 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 08:03 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5004.eqsin.wmnet * 07:58 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1219.eqiad.wmnet with reason: host reimage * 07:56 marostegui: install mariadb 10.11.17 on pc1 [[phab:T427345|T427345]] * 07:54 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1013.eqiad.wmnet with reason: host reimage * 07:50 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1013.eqiad.wmnet with reason: host reimage * 07:49 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 07:49 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 07:49 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5004.eqsin.wmnet * 07:47 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 07:47 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 07:46 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2176.codfw.wmnet with OS trixie * 07:43 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1219.eqiad.wmnet with OS trixie * 07:43 moritzm: imported Jenkins 2.541.3 for thirdparty/ci (Bullseye) and thirdparty/jenkins (Bookworm, Trixie) * 07:42 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 07:35 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1013.eqiad.wmnet with OS trixie * 07:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2176: Upgrading db2176.codfw.wmnet * 07:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1219: Upgrading db1219.eqiad.wmnet * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2176: Upgrading db2176.codfw.wmnet * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1219: Upgrading db1219.eqiad.wmnet * 07:31 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:30 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 07:29 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1163: Repooling * 07:19 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 06:51 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS trixie * 06:50 marostegui@cumin1003: dbctl commit (dc=all): 'Repool es2042', diff saved to https://phabricator.wikimedia.org/P94044 and previous config saved to /var/cache/conftool/dbconfig/20260611-065049-marostegui.json * 06:50 marostegui@cumin1003: dbctl commit (dc=all): 'Depool es2042', diff saved to https://phabricator.wikimedia.org/P94043 and previous config saved to /var/cache/conftool/dbconfig/20260611-065027-marostegui.json * 06:44 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1163: Repooling * 06:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1163 [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94041 and previous config saved to /var/cache/conftool/dbconfig/20260611-064319-fceratto.json * 06:42 fceratto@dns1005: END - running authdns-update * 06:40 fceratto@dns1005: START - running authdns-update * 06:33 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:33 fceratto@cumin1003: MariaDB change: Setting sections s1 as read-write for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:33 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:33 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1184 to s1 primary and set section read-write [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94040 and previous config saved to /var/cache/conftool/dbconfig/20260611-063323-fceratto.json * 06:32 fceratto@cumin1003: dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94039 and previous config saved to /var/cache/conftool/dbconfig/20260611-063251-fceratto.json * 06:32 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:32 fceratto@cumin1003: Dbctl change: Setting sections s1 as read-write for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:32 fceratto@cumin1003: MariaDB change: Setting sections s1 as read-write for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:31 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:31 fceratto@cumin1003: dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94037 and previous config saved to /var/cache/conftool/dbconfig/20260611-063100-fceratto.json * 06:30 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:30 fceratto@cumin1003: MariaDB change: Setting sections s1 as read-only for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:30 fceratto@cumin1003: Dbctl change: Setting sections s1 as read-only for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:29 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:29 federico3: Starting s1 eqiad failover from db1163 to db1184 - [[phab:T426083|T426083]] * 06:22 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1184 with weight 0 [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94035 and previous config saved to /var/cache/conftool/dbconfig/20260611-062224-fceratto.json * 06:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 30 hosts with reason: Primary switchover s1 [[phab:T426083|T426083]] * 05:37 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 05:28 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 05:27 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 05:18 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 05:17 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS trixie * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2045: Upgrading es2045.codfw.wmnet * 05:16 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2045: Upgrading es2045.codfw.wmnet * 05:16 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 44s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:23 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2046.* * 01:19 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync * 01:18 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync * 01:18 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1009.eqiad.wmnet with OS trixie * 01:12 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:12 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 01:12 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:12 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:11 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:11 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 01:11 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:10 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:10 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 01:09 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 01:09 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 01:08 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 01:08 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 01:08 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 01:07 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 01:07 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 01:06 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 01:06 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 01:06 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 01:05 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 01:05 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 01:05 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 01:02 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1009.eqiad.wmnet with reason: host reimage * 00:58 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1009.eqiad.wmnet with reason: host reimage * 00:54 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main1009 * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main1009 * 00:41 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main1009 * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main1009.eqiad.wmnet 37.48.64.10.in-addr.arpa 7.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:41 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main1009.eqiad.wmnet 37.48.64.10.in-addr.arpa 7.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1009 - jasmine@cumin2002" * 00:40 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1009 - jasmine@cumin2002" * 00:39 cdanis@cumin1003: dbctl commit (dc=all): 'depool db1262', diff saved to https://phabricator.wikimedia.org/P94032 and previous config saved to /var/cache/conftool/dbconfig/20260611-003950-cdanis.json * 00:36 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 00:34 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5020.* * 00:30 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main1009 * 00:30 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1009.eqiad.wmnet with OS trixie * 00:03 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5024.* == 2026-06-10 == * 23:53 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5024.* * 23:15 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300154{{!}}Disable ShortUrl on bdwikimedia, bhwiki, bnwiki, bnwikisource, eswikibooks, gomwiki (T107188)]] (duration: 11m 37s) * 23:11 krinkle@deploy1003: krinkle: Continuing with deployment * 23:06 krinkle@deploy1003: krinkle: Backport for [[gerrit:1300154{{!}}Disable ShortUrl on bdwikimedia, bhwiki, bnwiki, bnwikisource, eswikibooks, gomwiki (T107188)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:04 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1300154{{!}}Disable ShortUrl on bdwikimedia, bhwiki, bnwiki, bnwikisource, eswikibooks, gomwiki (T107188)]] * 22:57 ladsgroup@dns1004: END - running authdns-update * 22:55 ladsgroup@dns1004: START - running authdns-update * 22:13 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5024.eqsin.wmnet with OS trixie * 22:13 mutante: gerrit - restarting service for logging change * 22:11 dzahn@cumin2002: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 0:10:00 on gerrit.wikimedia.org with reason: service restart * 22:09 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on gerrit2003.wikimedia.org with reason: service restart * 22:06 mutante: gerrit-spare: restarting gerrit * 22:06 mutante: gerrit-replica: restarting gerrit * 21:44 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage * 21:37 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage * 21:22 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300250{{!}}ExecuteTestAndCacheJob: Fix stdClasses serialised wrongly by JobQueue (T428801)]], [[gerrit:1300248{{!}}tests: Fix StandaloneHooksTest ordering, now broken by DB upgrade]] (duration: 08m 23s) * 21:17 jforrester@deploy1003: jforrester: Continuing with deployment * 21:15 jforrester@deploy1003: jforrester: Backport for [[gerrit:1300250{{!}}ExecuteTestAndCacheJob: Fix stdClasses serialised wrongly by JobQueue (T428801)]], [[gerrit:1300248{{!}}tests: Fix StandaloneHooksTest ordering, now broken by DB upgrade]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:13 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1300250{{!}}ExecuteTestAndCacheJob: Fix stdClasses serialised wrongly by JobQueue (T428801)]], [[gerrit:1300248{{!}}tests: Fix StandaloneHooksTest ordering, now broken by DB upgrade]] * 21:03 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5024 * 21:02 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5024 * 21:02 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300247{{!}}Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] (duration: 06m 51s) * 21:00 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp5024 * 21:00 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5024.eqsin.wmnet 35.0.132.10.in-addr.arpa 5.3.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 21:00 brett@cumin2002: START - Cookbook sre.dns.wipe-cache cp5024.eqsin.wmnet 35.0.132.10.in-addr.arpa 5.3.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 21:00 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:00 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5024 - brett@cumin2002" * 20:59 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5024 - brett@cumin2002" * 20:57 catrope@deploy1003: catrope: Continuing with deployment * 20:57 catrope@deploy1003: catrope: Backport for [[gerrit:1300247{{!}}Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:55 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1300247{{!}}Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] * 20:54 brett@cumin2002: START - Cookbook sre.dns.netbox * 20:50 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5024 * 20:49 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5024.eqsin.wmnet with OS trixie * 20:48 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5020.* * 20:44 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300073{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] (duration: 11m 55s) * 20:40 catrope@deploy1003: catrope, gkyziridis: Continuing with deployment * 20:34 catrope@deploy1003: catrope, gkyziridis: Backport for [[gerrit:1300073{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:32 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1300073{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] * 20:30 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5020.eqsin.wmnet with OS trixie * 20:30 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300226{{!}}[arzwiki] Change the wordmark (T427720)]] (duration: 09m 49s) * 20:25 catrope@deploy1003: gergesshamon, catrope: Continuing with deployment * 20:22 catrope@deploy1003: gergesshamon, catrope: Backport for [[gerrit:1300226{{!}}[arzwiki] Change the wordmark (T427720)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1300226{{!}}[arzwiki] Change the wordmark (T427720)]] * 19:59 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5020.eqsin.wmnet with reason: host reimage * 19:53 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5020.eqsin.wmnet with reason: host reimage * 19:30 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 19:27 bblack@cumin1003: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 19:23 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2046.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:19 brett@cumin2002: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2046.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5020 * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5020 * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2044.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:18 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp5020 * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5020.eqsin.wmnet 24.0.132.10.in-addr.arpa 4.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:18 brett@cumin2002: START - Cookbook sre.dns.wipe-cache cp5020.eqsin.wmnet 24.0.132.10.in-addr.arpa 4.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:17 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5020 - brett@cumin2002" * 19:17 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5020 - brett@cumin2002" * 19:14 brett@cumin2002: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2044.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:11 brett@cumin2002: START - Cookbook sre.dns.netbox * 19:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 19:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2174: Migration of db2174.codfw.wmnet completed * 19:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 19:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1218: Migration of db1218.eqiad.wmnet completed * 18:24 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5020 * 18:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5020.eqsin.wmnet with OS trixie * 18:22 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2174: Migration of db2174.codfw.wmnet completed * 18:20 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 18:17 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1218: Migration of db1218.eqiad.wmnet completed * 18:16 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5018.* * 18:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2174.codfw.wmnet with OS trixie * 18:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1218.eqiad.wmnet with OS trixie * 17:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2174.codfw.wmnet with reason: host reimage * 17:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1218.eqiad.wmnet with reason: host reimage * 17:46 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2010.codfw.wmnet with OS trixie * 17:45 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync * 17:44 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync * 17:44 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2174.codfw.wmnet with reason: host reimage * 17:42 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1218.eqiad.wmnet with reason: host reimage * 17:33 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94021) * 17:29 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2010.codfw.wmnet with reason: host reimage * 17:26 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1218.eqiad.wmnet with OS trixie * 17:26 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2174.codfw.wmnet with OS trixie * 17:25 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:24 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 17:24 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1218: Upgrading db1218.eqiad.wmnet * 17:24 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:24 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:24 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1218: Upgrading db1218.eqiad.wmnet * 17:23 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 17:23 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2174: Upgrading db2174.codfw.wmnet * 17:23 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 17:23 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2010.codfw.wmnet with reason: host reimage * 17:23 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:22 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2174: Upgrading db2174.codfw.wmnet * 17:22 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:22 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 17:22 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 17:22 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 17:22 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text and not P<nowiki>{</nowiki>cp7008*<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 17:21 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 17:21 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 17:19 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 17:19 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 17:18 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 17:18 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 17:17 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 17:17 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 17:17 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 17:13 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 17:12 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox) * 17:03 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 17:03 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1206: Migration of db1206.eqiad.wmnet completed * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2010 * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2010 * 17:02 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2010 * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2010.codfw.wmnet 35.48.192.10.in-addr.arpa 5.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:02 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2010.codfw.wmnet 35.48.192.10.in-addr.arpa 5.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2010 - jasmine@cumin2002" * 17:01 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2010 - jasmine@cumin2002" * 16:57 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 16:50 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2010 * 16:50 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2010.codfw.wmnet with OS trixie * 16:41 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 16:39 bblack@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 16:39 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 16:34 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 16:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5018.eqsin.wmnet with OS trixie * 16:22 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 16:20 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 16:17 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1206: Migration of db1206.eqiad.wmnet completed * 16:15 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 16:15 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 16:14 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 16:12 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 16:12 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 16:11 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 16:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1206.eqiad.wmnet with OS trixie * 16:01 blblack: apt: uploaded libvmod-wmfuniq 0.3.0 for trixie * 15:59 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5018.eqsin.wmnet with reason: host reimage * 15:53 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:52 vriley@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:51 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5018.eqsin.wmnet with reason: host reimage * 15:50 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1206.eqiad.wmnet with reason: host reimage * 15:45 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1206.eqiad.wmnet with reason: host reimage * 15:43 sukhe@cumin1003: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool drmrs [reason: no reason specified, no task ID specified] * 15:42 sukhe@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: no reason specified, no task ID specified] * 15:38 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 15:38 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2173: Migration of db2173.codfw.wmnet completed * 15:34 topranks: drain traffic through cr2-drmrs to reset pic0 * 15:33 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94013) * 15:30 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1206.eqiad.wmnet with OS trixie * 15:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1206: Upgrading db1206.eqiad.wmnet * 15:28 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1206: Upgrading db1206.eqiad.wmnet * 15:27 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:25 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:24 vriley@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:24 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1009 * 15:24 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Harroyo-wmf out of all services on: 2436 hosts * 15:23 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1009 * 15:21 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:20 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist translate extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release * 15:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5018 * 15:19 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5018 * 15:18 vriley@cumin1003: START - Cookbook sre.dns.netbox * 15:18 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp5018 * 15:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5018.eqsin.wmnet 18.0.132.10.in-addr.arpa 8.1.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 15:18 brett@cumin2002: START - Cookbook sre.dns.wipe-cache cp5018.eqsin.wmnet 18.0.132.10.in-addr.arpa 8.1.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 15:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:15 brett@cumin2002: START - Cookbook sre.dns.netbox * 15:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 15:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1195: Migration of db1195.eqiad.wmnet completed * 15:12 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:11 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:11 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin1003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:11 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin1003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:08 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300169{{!}}Fix snak value display for rtl languages (T360854)]], [[gerrit:1300168{{!}}Fix snak value display for rtl languages (T360854)]] (duration: 08m 39s) * 15:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 15:01 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1300169{{!}}Fix snak value display for rtl languages (T360854)]], [[gerrit:1300168{{!}}Fix snak value display for rtl languages (T360854)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:59 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:59 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1300169{{!}}Fix snak value display for rtl languages (T360854)]], [[gerrit:1300168{{!}}Fix snak value display for rtl languages (T360854)]] * 14:58 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:55 Lucas_WMDE: lucaswerkmeister-wmde@deploy1003 $ printf 'https://www.mediawiki.org/keys/%s\n' '' 'keys.txt' 'keys.html' {{!}} mwscript-k8s --attach --comment=[[phab:T423267|T423267]] purgeList mediawikiwiki * 14:54 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist translate extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release, now with correct schema * 14:53 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2173: Migration of db2173.codfw.wmnet completed * 14:50 ayounsi@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin2003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:50 ayounsi@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:49 ayounsi@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:48 ayounsi@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:47 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299614{{!}}Add my public key to mediawiki.org/keys (T423267)]] (duration: 08m 33s) * 14:46 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:42 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, matmarex: Continuing with deployment * 14:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2173.codfw.wmnet with OS trixie * 14:40 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, matmarex: Backport for [[gerrit:1299614{{!}}Add my public key to mediawiki.org/keys (T423267)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:40 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:40 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:38 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1299614{{!}}Add my public key to mediawiki.org/keys (T423267)]] * 14:38 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox and (A:dnsbox) * 14:34 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:34 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:33 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:29 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1195: Migration of db1195.eqiad.wmnet completed * 14:28 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:27 vriley@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:26 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 14:26 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 14:24 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist translate extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release, now with dblist translate * 14:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2173.codfw.wmnet with reason: host reimage * 14:23 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 14:22 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 14:22 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply * 14:21 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply * 14:20 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox) * 14:20 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2173.codfw.wmnet with reason: host reimage * 14:20 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:19 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:19 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:18 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:18 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:18 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:18 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1195.eqiad.wmnet with OS trixie * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 14:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 14:15 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 14:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 14:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 14:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 14:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:13 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:13 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 14:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 14:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 14:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 14:09 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 14:08 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:08 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 14:07 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 14:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 14:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 14:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 14:02 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2173.codfw.wmnet with OS trixie * 14:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 14:00 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1195.eqiad.wmnet with reason: host reimage * 14:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 13:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2173: Upgrading db2173.codfw.wmnet * 13:59 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2173: Upgrading db2173.codfw.wmnet * 13:58 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:58 atsuko@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/ttmserver-export.php --wiki=default --ttmserver eqiad-test # [[phab:T425377|T425377]] populating production index on test cluster to estimate time required for the release * 13:56 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1195.eqiad.wmnet with reason: host reimage * 13:54 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Atieno out of all services on: 2436 hosts * 13:42 Lucas_WMDE: UTC afternoon backport+config window done * 13:42 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1195.eqiad.wmnet with OS trixie * 13:36 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297237{{!}}wmf-config: Update private subnets to include additions (T427393)]] (duration: 07m 20s) * 13:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1195: Upgrading db1195.eqiad.wmnet * 13:33 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling restart_daemons on A:hcaptcha-proxy and A:hcaptcha-proxy * 13:33 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum and A:durum * 13:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2170: Migration of db2170.codfw.wmnet completed * 13:33 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1195: Upgrading db1195.eqiad.wmnet * 13:32 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:32 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, brett: Continuing with deployment * 13:32 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough * 13:31 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply * 13:31 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, brett: Backport for [[gerrit:1297237{{!}}wmf-config: Update private subnets to include additions (T427393)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:31 eevans@deploy1003: helmfile [staging] START helmfile.d/services/data-gateway: apply * 13:29 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1297237{{!}}wmf-config: Update private subnets to include additions (T427393)]] * 13:28 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp5018.eqsin.wmnet with reason: host down * 13:28 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy (exit_code=0) rolling restart_daemons on A:tcpproxy and A:tcpproxy * 13:25 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5018.eqsin.wmnet,service=(cdn{{!}}ats-be) * 13:22 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox) * 13:20 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum and A:durum * 13:20 sukhe@cumin1003: START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling restart_daemons on A:hcaptcha-proxy and A:hcaptcha-proxy * 13:19 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299676{{!}}Enable ULS v2 on group0 wikis]] (duration: 17m 00s) * 13:19 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough * 13:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1186: Migration of db1186.eqiad.wmnet completed * 13:18 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-test: apply * 13:18 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-test: apply * 13:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply * 13:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply * 13:15 sbisson@deploy1003: sbisson, abi: Continuing with deployment * 13:10 sukhe@cumin1003: START - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy rolling restart_daemons on A:tcpproxy and A:tcpproxy * 13:05 sbisson@deploy1003: sbisson, abi: Backport for [[gerrit:1299676{{!}}Enable ULS v2 on group0 wikis]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:03 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1014.eqiad.wmnet with OS trixie * 13:02 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299676{{!}}Enable ULS v2 on group0 wikis]] * 12:47 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2170: Migration of db2170.codfw.wmnet completed * 12:46 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5004.eqsin.wmnet with OS bookworm * 12:46 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:46 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:46 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:45 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1014.eqiad.wmnet with reason: host reimage * 12:42 topranks: re-map DSCP AF41 from 'low' to 'normal' priority qos class on network [[phab:T424640|T424640]] * 12:41 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1014.eqiad.wmnet with reason: host reimage * 12:36 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2170.codfw.wmnet with OS trixie * 12:33 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1186: Migration of db1186.eqiad.wmnet completed * 12:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5004.eqsin.wmnet with reason: host reimage * 12:24 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1014 * 12:24 jiji@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host rdb1014 * 12:23 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1186.eqiad.wmnet with OS trixie * 12:21 jiji@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host rdb1014 * 12:21 jiji@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb1014.eqiad.wmnet 42.48.64.10.in-addr.arpa 2.4.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 12:21 jiji@cumin1003: START - Cookbook sre.dns.wipe-cache rdb1014.eqiad.wmnet 42.48.64.10.in-addr.arpa 2.4.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 12:21 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:21 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host rdb1014 - jiji@cumin1003" * 12:21 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host rdb1014 - jiji@cumin1003" * 12:20 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5004.eqsin.wmnet with reason: host reimage * 12:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2170.codfw.wmnet with reason: host reimage * 12:16 jiji@cumin1003: START - Cookbook sre.dns.netbox * 12:13 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1014 * 12:12 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1014.eqiad.wmnet with OS trixie * 12:12 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2170.codfw.wmnet with reason: host reimage * 12:08 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300104{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643{{!}}wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] (duration: 11m 06s) * 12:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1186.eqiad.wmnet with reason: host reimage * 12:03 reedy@deploy1003: reedy: Continuing with deployment * 12:02 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1186.eqiad.wmnet with reason: host reimage * 11:59 reedy@deploy1003: reedy: Backport for [[gerrit:1300104{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643{{!}}wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes c * 11:57 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1300104{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643{{!}}wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] * 11:53 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2170.codfw.wmnet with OS trixie * 11:51 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ganeti5004 * 11:51 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti5004 * 11:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2170: Upgrading db2170.codfw.wmnet * 11:49 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2170: Upgrading db2170.codfw.wmnet * 11:49 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti5004 * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti5004.eqsin.wmnet 40.0.132.10.in-addr.arpa 0.4.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 11:49 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache ganeti5004.eqsin.wmnet 40.0.132.10.in-addr.arpa 0.4.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5004 - jmm@cumin2002" * 11:49 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5004 - jmm@cumin2002" * 11:49 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:48 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1186.eqiad.wmnet with OS trixie * 11:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1186: Upgrading db1186.eqiad.wmnet * 11:45 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1186: Upgrading db1186.eqiad.wmnet * 11:45 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:38 jmm@cumin2002: START - Cookbook sre.dns.netbox * 11:35 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 11:34 jmm@cumin2002: START - Cookbook sre.hosts.move-vlan for host ganeti5004 * 11:34 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 11:34 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5004.eqsin.wmnet with OS bookworm * 11:34 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 11:33 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1151: Security updates * 11:33 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 11:33 root@cumin1003: START - Cookbook sre.mysql.parsercache * 11:33 root@cumin1003: START - Cookbook sre.mysql.pool pool db1151: Security updates * 11:31 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply * 11:30 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply * 11:30 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply * 11:30 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:23 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 11:23 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:23 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 11:23 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 11:16 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 11:15 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 11:15 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 11:15 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:09 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1151: Security updates * 11:09 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 11:09 root@cumin1003: START - Cookbook sre.mysql.parsercache * 11:09 root@cumin1003: START - Cookbook sre.mysql.depool depool db1151: Security updates * 11:08 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300092{{!}}ProductionServices: re-add poolcounter2006 (T426736)]] (duration: 06m 55s) * 11:04 blake@deploy1003: blake: Continuing with deployment * 11:04 blake@deploy1003: blake: Backport for [[gerrit:1300092{{!}}ProductionServices: re-add poolcounter2006 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:03 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:01 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300092{{!}}ProductionServices: re-add poolcounter2006 (T426736)]] * 10:59 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet * 10:57 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 10:57 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 10:57 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 10:56 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 10:56 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply * 10:56 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply * 10:56 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet * 10:56 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300087{{!}}ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] (duration: 06m 42s) * 10:51 blake@deploy1003: blake: Continuing with deployment * 10:51 moritzm: remove ganeti5004 from eqsin cluster for reimage [[phab:T428229|T428229]] * 10:51 blake@deploy1003: blake: Backport for [[gerrit:1300087{{!}}ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:49 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300087{{!}}ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] * 10:47 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet * 10:47 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 10:46 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 10:46 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 10:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 10:43 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet * 10:43 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300082{{!}}ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] (duration: 07m 38s) * 10:41 moritzm: installing nginx security updates * 10:38 blake@deploy1003: blake: Continuing with deployment * 10:38 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1152: Security updates * 10:38 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:38 root@cumin1003: START - Cookbook sre.mysql.parsercache * 10:38 root@cumin1003: START - Cookbook sre.mysql.pool pool db1152: Security updates * 10:38 blake@deploy1003: blake: Backport for [[gerrit:1300082{{!}}ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:37 moritzm: failover Ganeti master in eqsin to ganeti5007 [[phab:T428229|T428229]] * 10:35 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300082{{!}}ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] * 10:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 10:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 10:33 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet * 10:29 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet * 10:29 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300072{{!}}ProductionServices: reboot poolcounter1007 (T426736)]] (duration: 07m 45s) * 10:27 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 10:27 jmm@cumin2002: DONE (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for sretest2009.codfw.wmnet: Renew puppet certificate - jmm@cumin2002 * 10:24 blake@deploy1003: blake: Continuing with deployment * 10:23 blake@deploy1003: blake: Backport for [[gerrit:1300072{{!}}ProductionServices: reboot poolcounter1007 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:22 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 10:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 10:21 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:21 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300072{{!}}ProductionServices: reboot poolcounter1007 (T426736)]] * 10:21 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:21 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:21 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:21 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:20 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet * 10:14 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1152: Security updates * 10:14 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:14 root@cumin1003: START - Cookbook sre.mysql.parsercache * 10:14 root@cumin1003: START - Cookbook sre.mysql.depool depool db1152: Security updates * 10:13 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet * 10:12 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300064{{!}}ProductionServices: reboot poolcounter1006.eqiad (T426736)]] (duration: 07m 46s) * 10:07 blake@deploy1003: blake: Continuing with deployment * 10:06 blake@deploy1003: blake: Backport for [[gerrit:1300064{{!}}ProductionServices: reboot poolcounter1006.eqiad (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:04 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300064{{!}}ProductionServices: reboot poolcounter1006.eqiad (T426736)]] * 09:57 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300058{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] (duration: 09m 32s) * 09:52 kharlan@deploy1003: kharlan: Continuing with deployment * 09:49 kharlan@deploy1003: kharlan: Backport for [[gerrit:1300058{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:47 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1300058{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] * 09:35 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox * 09:34 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 09:32 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 09:32 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 09:26 moritzm: upgrade routinator in eqiad to 0.15.2 [[phab:T428456|T428456]] * 09:23 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 09:23 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 09:22 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 09:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus5003.eqsin.wmnet to plain * 09:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus5003.eqsin.wmnet to plain * 09:15 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:04 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:54 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:20 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:09 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:09 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:07 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 08:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:04 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:01 fceratto@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host db1215.eqiad.wmnet with OS trixie * 07:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:53 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:48 javiermonton@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply * 07:48 javiermonton@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply * 07:44 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1215.eqiad.wmnet with reason: host reimage * 07:41 javiermonton@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply * 07:40 javiermonton@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply * 07:40 moritzm: installing openssl security updates * 07:39 fceratto@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1215.eqiad.wmnet with reason: host reimage * 07:38 javiermonton@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply * 07:37 javiermonton@deploy1003: helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply * 07:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:29 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299556{{!}}ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561{{!}}ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529{{!}}translate: adding separate read/write endpoints (T425377)]] (duration: 14m 03s) * 07:25 atsuko@deploy1003: atsuko: Continuing with deployment * 07:23 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db1215.eqiad.wmnet with OS trixie * 07:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1215.eqiad.wmnet with reason: Reimage * 07:21 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:20 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:20 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:17 atsuko@deploy1003: atsuko: Backport for [[gerrit:1299556{{!}}ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561{{!}}ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529{{!}}translate: adding separate read/write endpoints (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be veri * 07:16 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:15 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1299556{{!}}ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561{{!}}ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529{{!}}translate: adding separate read/write endpoints (T425377)]] * 07:14 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:12 atsukoito: backporting extensions/Translate to wmf/1.47.0-wmf.5 and applying the config * 07:12 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 06:45 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 05:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 05:43 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 05:42 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 05:41 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 47s) * 02:07 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1008.eqiad.wmnet with OS trixie * 02:03 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync * 02:02 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:52 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:51 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 01:51 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:50 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:50 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:49 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1008.eqiad.wmnet with reason: host reimage * 01:49 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 01:49 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:49 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:49 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 01:48 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 01:48 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 01:47 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 01:47 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 01:46 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 01:46 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 01:44 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 01:44 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 01:43 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 01:43 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1008.eqiad.wmnet with reason: host reimage * 01:25 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main1008 * 01:24 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main1008 * 01:24 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main1008 * 01:24 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main1008.eqiad.wmnet 45.32.64.10.in-addr.arpa 5.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 01:23 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main1008.eqiad.wmnet 45.32.64.10.in-addr.arpa 5.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 01:23 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 01:23 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1008 - jasmine@cumin2002" * 01:23 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1008 - jasmine@cumin2002" * 01:19 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 01:12 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main1008 * 01:11 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1008.eqiad.wmnet with OS trixie * 01:00 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2009.codfw.wmnet with OS trixie * 00:54 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync * 00:53 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync * 00:43 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2009.codfw.wmnet with reason: host reimage * 00:40 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:38 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2009.codfw.wmnet with reason: host reimage * 00:38 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 00:38 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:37 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:37 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 00:36 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 00:36 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 00:34 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 00:34 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 00:33 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 00:33 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 00:32 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 00:32 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 00:32 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2009 * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2009 * 00:15 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2009 * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2009.codfw.wmnet 33.48.192.10.in-addr.arpa 3.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:15 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2009.codfw.wmnet 33.48.192.10.in-addr.arpa 3.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2009 - jasmine@cumin2002" * 00:15 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2009 - jasmine@cumin2002" * 00:10 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 00:03 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2009 * 00:03 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2009.codfw.wmnet with OS trixie == 2026-06-09 == * 22:50 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299640{{!}}HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] (duration: 08m 59s) * 22:45 cscott@deploy1003: cscott: Continuing with deployment * 22:43 cscott@deploy1003: cscott: Backport for [[gerrit:1299640{{!}}HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:41 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299640{{!}}HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] * 22:15 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299639{{!}}[Bug] Donor Badge: Remove client prefs for control group (T428501)]] (duration: 20m 57s) * 22:11 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 22:07 mutante: gerrit - apache httpd log file location moved to /srv/gerrit/site_path/review_site/logs/ [[phab:T425667|T425667]] * 22:06 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on gerrit2003.wikimedia.org with reason: debug * 21:56 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1299639{{!}}[Bug] Donor Badge: Remove client prefs for control group (T428501)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:54 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299639{{!}}[Bug] Donor Badge: Remove client prefs for control group (T428501)]] * 21:52 ryankemper: [[phab:T428241|T428241]] removed retired wdqs2009 full-graph journal dump (446G x2, ~892G) from clouddumps100[1-2]:/srv/dumps/xmldatadumps/public/other/wdqs * 21:49 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299602{{!}}Revert "Create VectorComponentPageToolbar component" (T428649)]] (duration: 08m 16s) * 21:48 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 21:45 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 21:43 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1299602{{!}}Revert "Create VectorComponentPageToolbar component" (T428649)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:41 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299602{{!}}Revert "Create VectorComponentPageToolbar component" (T428649)]] * 21:34 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gerrit1003.wikimedia.org with reason: debug * 21:27 maryum: Deployed security fix for [[phab:T428324|T428324]] * 21:24 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 21:15 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart * 21:06 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart * 20:50 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs2002.codfw.wmnet with OS trixie * 20:50 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299588{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] (duration: 11m 13s) * 20:46 cscott@deploy1003: cscott: Continuing with deployment * 20:43 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs2002.codfw.wmnet with OS trixie * 20:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:42 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:41 cscott@deploy1003: cscott: Backport for [[gerrit:1299588{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:39 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299588{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] * 20:38 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:38 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:33 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:33 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:32 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299454{{!}}wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] (duration: 22m 08s) * 20:28 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:28 cscott@deploy1003: cscott, gkyziridis: Continuing with deployment * 20:24 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2004 * 20:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2004 * 20:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2003 * 20:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2003 * 20:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2002 * 20:13 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2002 * 20:13 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2001 * 20:13 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2001 * 20:12 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:12 cscott@deploy1003: cscott, gkyziridis: Backport for [[gerrit:1299454{{!}}wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:10 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299454{{!}}wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] * 20:09 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:04 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:59 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:54 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:53 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:48 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:47 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:47 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:46 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:46 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:45 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:45 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:28 ryankemper@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts wdqs1015.eqiad.wmnet * 19:28 ryankemper@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:28 ryankemper@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002" * 19:27 ryankemper@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002" * 19:20 ryankemper@cumin2002: START - Cookbook sre.dns.netbox * 19:15 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2008.codfw.wmnet with OS trixie * 19:15 ryankemper@cumin2002: START - Cookbook sre.hosts.decommission for hosts wdqs1015.eqiad.wmnet * 19:12 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync * 19:12 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync * 19:00 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:58 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 18:58 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2008.codfw.wmnet with reason: host reimage * 18:58 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 18:58 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 18:57 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 18:57 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 18:56 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 18:56 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 18:55 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 18:55 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 18:55 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 18:54 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 18:54 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:54 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 18:53 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 18:53 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 18:53 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 18:52 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:52 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2003 to codfw - jhancock@cumin2002" * 18:52 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2003 to codfw - jhancock@cumin2002" * 18:52 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 18:52 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 18:51 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2008.codfw.wmnet with reason: host reimage * 18:51 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 18:51 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 18:51 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 18:50 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 18:50 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 18:47 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:47 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:47 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:46 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:46 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:42 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:42 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:31 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 18:29 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2008.codfw.wmnet with OS trixie * 18:26 jasmine@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2008.codfw.wmnet with OS trixie * 17:48 mutante: https://releases.wikimedia.org {{!}} https://releases-jenkins.wikimedia.org - down for maintenance [[phab:T418299|T418299]] * 17:48 cmooney@dns2005: END - running authdns-update * 17:47 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet with reason: reimage * 17:47 cmooney@dns2005: START - running authdns-update * 17:46 sukhe: sudo cumin 'A:hcaptcha-proxy' 'run-puppet-agent': rolling out CR {{Gerrit|1299427}} [[phab:T428539|T428539]] * 17:43 jayme: kafka-main2008 is down due to hardware failure [[phab:T428654|T428654]] * 17:32 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf1002.eqiad.wmnet with OS trixie * 17:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf1002.eqiad.wmnet with reason: host reimage * 17:06 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf1002.eqiad.wmnet with reason: host reimage * 17:05 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2008 * 17:05 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2008 * 17:04 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 17:04 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2008 * 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2008.codfw.wmnet 4.32.192.10.in-addr.arpa 4.0.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:04 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 17:04 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2008.codfw.wmnet 4.32.192.10.in-addr.arpa 4.0.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2008 - jasmine@cumin2002" * 17:04 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5018 * 17:04 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2008 - jasmine@cumin2002" * 17:03 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5018.eqsin.wmnet with OS trixie * 16:58 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 16:58 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 16:57 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 16:57 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 16:57 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 16:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 16:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 16:50 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf1002.eqiad.wmnet with OS trixie * 16:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 16:47 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf1001.eqiad.wmnet with OS trixie * 16:47 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply * 16:47 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply * 16:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 16:41 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 16:41 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 16:35 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2008 * 16:34 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2008.codfw.wmnet with OS trixie * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:31 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply * 16:30 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply * 16:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf1001.eqiad.wmnet with reason: host reimage * 16:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:26 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf1001.eqiad.wmnet with reason: host reimage * 16:23 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply * 16:22 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: apply * 16:20 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:16 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:15 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:13 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:13 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:12 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf1001.eqiad.wmnet with OS trixie * 16:10 jiji@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. * 16:09 jiji@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. * 16:07 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf2002.codfw.wmnet with OS trixie * 16:02 jiji@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 16:02 jiji@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 16:00 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. * 15:59 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 15:59 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. * 15:59 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 15:59 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. * 15:59 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 15:58 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] DONE helmfile.d/services/termbox: apply * 15:58 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] START helmfile.d/services/termbox: apply * 15:57 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 15:57 jiji@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 15:57 lucaswerkmeister-wmde@deploy1003: helmfile [staging] DONE helmfile.d/services/termbox: apply * 15:56 lucaswerkmeister-wmde@deploy1003: helmfile [staging] START helmfile.d/services/termbox: apply * 15:54 jiji@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 15:53 jiji@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 15:51 jiji@deploy1003: Finished scap sync-world: redeploy {{Gerrit|1299468}} (duration: 07m 23s) * 15:49 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf2002.codfw.wmnet with reason: host reimage * 15:47 jiji@deploy1003: jiji: Continuing with deployment * 15:46 jiji@deploy1003: jiji: redeploy {{Gerrit|1299468}} synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:46 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf2002.codfw.wmnet with reason: host reimage * 15:45 jiji@deploy1003: Started scap sync-world: redeploy {{Gerrit|1299468}} * 15:43 brouberol@cumin1003: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-eqiad * 15:34 brennen@deploy1003: Finished deploy [phabricator/deployment@73e57ce]: deploy phab1004 for [[phab:T410849|T410849]] (followup for robots.txt) (duration: 00m 40s) * 15:33 brennen@deploy1003: Started deploy [phabricator/deployment@73e57ce]: deploy phab1004 for [[phab:T410849|T410849]] (followup for robots.txt) * 15:33 brennen@deploy1003: Finished deploy [phabricator/deployment@73e57ce]: deploy phab2002 for [[phab:T410849|T410849]] (followup for robots.txt) (duration: 00m 45s) * 15:32 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299468{{!}}ProductionServices.php: switch filebackend.php to rdb2015:6381 #2 (T418918 T291916)]] (duration: 07m 21s) * 15:32 brennen@deploy1003: Started deploy [phabricator/deployment@73e57ce]: deploy phab2002 for [[phab:T410849|T410849]] (followup for robots.txt) * 15:28 jiji@deploy1003: Rolling back deployment * 15:27 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf2002.codfw.wmnet with OS trixie * 15:27 jiji@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 15:26 jiji@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 15:25 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1299468{{!}}ProductionServices.php: switch filebackend.php to rdb2015:6381 #2 (T418918 T291916)]] * 15:22 urbanecm: Remove `migrateMentorStatusAwayToCommunityConfiguration` from updatelog on all wikis ([[phab:T409170|T409170]]; the script was only ever run as a dry-run) * 15:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 15:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 15:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf2001.codfw.wmnet with OS trixie * 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@d244a3e]: deploy phab1004 for [[phab:T410849|T410849]] (duration: 00m 42s) * 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@d244a3e]: deploy phab1004 for [[phab:T410849|T410849]] * 15:02 brennen@deploy1003: Finished deploy [phabricator/deployment@d244a3e]: deploy phab2002 for [[phab:T410849|T410849]] (duration: 00m 45s) * 15:01 brennen@deploy1003: Started deploy [phabricator/deployment@d244a3e]: deploy phab2002 for [[phab:T410849|T410849]] * 14:58 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf2001.codfw.wmnet with reason: host reimage * 14:52 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf2001.codfw.wmnet with reason: host reimage * 14:52 arnaudb@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phab[2002-2003].codfw.wmnet,phab[1004-1006].eqiad.wmnet with reason: [[phab:T410849|T410849]] * 14:47 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 14:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 14:40 moritzm: upgrade routinator in codfw to 0.15.2 [[phab:T428456|T428456]] * 14:35 brouberol@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad * 14:33 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf2001.codfw.wmnet with OS trixie * 14:26 brouberol@cumin1003: END (ERROR) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=97) rolling reboot on A:cephosd-eqiad * 14:26 brouberol@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad * 14:20 btullis@cumin1003: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-codfw * 14:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host parsoidtest1001.eqiad.wmnet * 14:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2153: Migration of db2153.codfw.wmnet completed * 14:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of rpki2003.codfw.wmnet to drbd * 14:14 moritzm: imported routinator 0.15.2-1bookworm to thirdparty/routinator for bookworm-wikimedia [[phab:T428456|T428456]] * 14:12 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:12 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1184: Migration of db1184.eqiad.wmnet completed * 14:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host parsoidtest1001.eqiad.wmnet * 14:07 Dreamy_Jazz: Afternoon UTC backport window done * 14:07 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 14:06 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299495{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502{{!}}SecurePollLogPager: Cast user IDs to ints before use (T428599)]] (duration: 06m 53s) * 14:06 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 14:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: rack depool * 14:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of rpki2003.codfw.wmnet to drbd * 14:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow2004.codfw.wmnet to drbd * 14:02 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:02 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1299495{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502{{!}}SecurePollLogPager: Cast user IDs to ints before use (T428599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:59 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1299495{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502{{!}}SecurePollLogPager: Cast user IDs to ints before use (T428599)]] * 13:58 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:58 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:56 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 13:56 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 13:55 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:55 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * {{safesubst:SAL entry|1=13:55 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298929{{!}}Simplify fragment processing (T423700)]], [[gerrit:1298926{{!}}Move ::getFragmentsToTransform() to Content<nowiki>{</nowiki>Text,DOM<nowiki>}</nowiki>TransformStage]], [[gerrit:1298927{{!}}OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925{{!}}Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497}} * 13:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:51 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow2004.codfw.wmnet to drbd * 13:50 cscott@deploy1003: cscott: Continuing with deployment * 13:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2045.codfw.wmnet to cluster codfw and group A * 13:48 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2045.codfw.wmnet to cluster codfw and group A * 13:48 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2027.codfw.wmnet to cluster codfw and group A * 13:47 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2027.codfw.wmnet to cluster codfw and group A * 13:46 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 13:45 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 13:44 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * {{safesubst:SAL entry|1=13:42 cscott@deploy1003: cscott: Backport for [[gerrit:1298929{{!}}Simplify fragment processing (T423700)]], [[gerrit:1298926{{!}}Move ::getFragmentsToTransform() to Content<nowiki>{</nowiki>Text,DOM<nowiki>}</nowiki>TransformStage]], [[gerrit:1298927{{!}}OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925{{!}}Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497{{!}}Store indicators}} * 13:41 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * {{safesubst:SAL entry|1=13:40 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1298929{{!}}Simplify fragment processing (T423700)]], [[gerrit:1298926{{!}}Move ::getFragmentsToTransform() to Content<nowiki>{</nowiki>Text,DOM<nowiki>}</nowiki>TransformStage]], [[gerrit:1298927{{!}}OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925{{!}}Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497{{!}}}} * 13:40 btullis@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-codfw * 13:39 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 13:37 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 13:35 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 13:33 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 13:32 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 13:32 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298834{{!}}config: Disable EmailConfirmationBanner on all wikis (T428291)]] (duration: 07m 01s) * 13:30 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2153: Migration of db2153.codfw.wmnet completed * 13:28 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 lucaswerkmeister-wmde@deploy1003: mmartorana, lucaswerkmeister-wmde: Continuing with deployment * 13:27 lucaswerkmeister-wmde@deploy1003: mmartorana, lucaswerkmeister-wmde: Backport for [[gerrit:1298834{{!}}config: Disable EmailConfirmationBanner on all wikis (T428291)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:26 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1184: Migration of db1184.eqiad.wmnet completed * 13:25 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298834{{!}}config: Disable EmailConfirmationBanner on all wikis (T428291)]] * 13:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 13:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 13:23 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:21 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 13:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2153.codfw.wmnet with OS trixie * 13:20 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2241: rack depool * 13:20 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1237: repool after maintenance db1237 * 13:19 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298654{{!}}Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] (duration: 09m 40s) * 13:17 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2006.codfw.wmnet * 13:17 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2006.codfw.wmnet * 13:16 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2251-2253].codfw.wmnet * 13:16 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2251-2253].codfw.wmnet * 13:16 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2005.codfw.wmnet * 13:16 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2005.codfw.wmnet * 13:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1184.eqiad.wmnet with OS trixie * 13:14 lucaswerkmeister-wmde@deploy1003: neriah, lucaswerkmeister-wmde: Continuing with deployment * 13:11 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 * 13:11 lucaswerkmeister-wmde@deploy1003: neriah, lucaswerkmeister-wmde: Backport for [[gerrit:1298654{{!}}Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:09 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298654{{!}}Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] * 13:04 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:04 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:04 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2153.codfw.wmnet with reason: host reimage * 13:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:03 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1015.eqiad.wmnet with OS trixie * 12:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage * 12:58 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2153.codfw.wmnet with reason: host reimage * 12:57 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1016.eqiad.wmnet with OS trixie * 12:57 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 * 12:56 XioNoX: lsw1-a4-codfw> request system reboot - [[phab:T427357|T427357]] * 12:55 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:53 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage * 12:50 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299477{{!}}hCaptcha: Roll out to all wikis for api account creation. (T426050)]] (duration: 07m 21s) * 12:46 kharlan@deploy1003: kharlan, dbrant: Continuing with deployment * 12:46 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 * 12:45 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1015.eqiad.wmnet with reason: host reimage * 12:45 kharlan@deploy1003: kharlan, dbrant: Backport for [[gerrit:1299477{{!}}hCaptcha: Roll out to all wikis for api account creation. (T426050)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:45 topranks: shut sub-interfaces for row A/B legacy vlans on cr1-codfw [[phab:T427357|T427357]] * 12:45 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 * 12:43 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1299477{{!}}hCaptcha: Roll out to all wikis for api account creation. (T426050)]] * 12:42 topranks: increase OSPF cost on ssw1-a1-codfw link to lsw1-a4-codfw to force traffic via alternate spine [[phab:T427357|T427357]] * 12:41 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299478{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]] (duration: 07m 02s) * 12:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1016.eqiad.wmnet with reason: host reimage * 12:40 moritzm: installing wireshark security updates * 12:40 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2153.codfw.wmnet with OS trixie * 12:38 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1184.eqiad.wmnet with OS trixie * 12:37 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 12:36 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1299478{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:34 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2153: Upgrading db2153.codfw.wmnet * 12:34 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1237: repool after maintenance db1237 * 12:34 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1299478{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]] * 12:34 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2153: Upgrading db2153.codfw.wmnet * 12:34 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1184: Upgrading db1184.eqiad.wmnet * 12:33 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1184: Upgrading db1184.eqiad.wmnet * 12:33 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:32 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1237.eqiad.wmnet with OS trixie * 12:32 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1015.eqiad.wmnet with reason: host reimage * 12:32 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1016.eqiad.wmnet with reason: host reimage * 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 12:27 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2005.codfw.wmnet * 12:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2046: repool after maintenance * 12:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2006.codfw.wmnet * 12:23 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298829{{!}}wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] (duration: 16m 04s) * 12:23 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2006.codfw.wmnet * 12:22 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2251-2253].codfw.wmnet * 12:22 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2005.codfw.wmnet * 12:20 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2251-2253].codfw.wmnet * 12:20 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 12:20 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: rack depool * 12:20 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 12:20 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2241: rack depool * 12:19 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1016 * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1016 * 12:19 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1015 * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1015 * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1016.eqiad.wmnet with OS trixie * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1015.eqiad.wmnet with OS trixie * 12:17 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 * 12:17 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 24 hosts with reason: Rack A4 depool * 12:16 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Continuing with deployment * 12:15 topranks: drain traffic on ssw1-a1-codfw - add gshut community in evpn underlay - [[phab:T427357|T427357]] * 12:14 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 * 12:13 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Backport for [[gerrit:1298829{{!}}wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:10 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1237.eqiad.wmnet with reason: host reimage * 12:07 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1298829{{!}}wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] * 12:05 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1237.eqiad.wmnet with reason: host reimage * 12:00 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Dmaza out of all services on: 2435 hosts * 11:51 atsuko@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 11:51 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie * 11:49 atsuko@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 11:48 atsuko@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 11:47 atsuko@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 11:45 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 11:44 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 11:43 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:43 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2046: repool after maintenance * 11:38 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 11:36 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:36 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2046.codfw.wmnet with OS trixie * 11:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2185.codfw.wmnet with reason: Reimage * 11:31 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging HMonroy out of all services on: 2435 hosts * 11:28 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging KSiebert out of all services on: 2435 hosts * 11:26 slyngs: CAS-SSO upgrade to version 7.3.7.2 * 11:26 slyngshede@dns1004: END - running authdns-update * 11:24 slyngshede@dns1004: START - running authdns-update * 11:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2046.codfw.wmnet with reason: host reimage * 11:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1043: repool after upgrade * 11:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2046.codfw.wmnet with reason: host reimage * 10:55 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2046.codfw.wmnet with OS trixie * 10:53 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2046: Upgrading es2046.codfw.wmnet * 10:53 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2046: Upgrading es2046.codfw.wmnet * 10:52 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply * 10:52 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:52 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply * 10:52 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply * 10:52 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:52 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply * 10:51 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:32 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1043: repool after upgrade * 10:31 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:28 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1160: Repooling * 10:26 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1043.eqiad.wmnet with OS trixie * 10:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:17 elukey: complete rollout of apache2 upgrades * 10:16 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:15 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:13 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:12 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:12 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:08 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1043.eqiad.wmnet with reason: host reimage * 10:04 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:04 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1043.eqiad.wmnet with reason: host reimage * 10:04 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:04 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:04 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:04 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:04 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:57 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1160: Repooling * 09:51 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 09:51 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 09:50 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 09:50 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 09:49 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1043.eqiad.wmnet with OS trixie * 09:48 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool es1043: Upgrading es1043.eqiad.wmnet * 09:48 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 09:47 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:45 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 09:41 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 09:36 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=5 --verbose --last-checked="20260603"` (after stopping previous scan run) * 09:34 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=5 --verbose` (after stopping previous scan run) * 09:27 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 09:26 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 09:17 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 09:17 fceratto@cumin1003: MariaDB change: Setting sections s5 as read-write * 09:17 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 09:14 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1043: Upgrading es1043.eqiad.wmnet * 09:14 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:12 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1042 to es4 eqiad primary [[phab:T428386|T428386]]', diff saved to https://phabricator.wikimedia.org/P93943 and previous config saved to /var/cache/conftool/dbconfig/20260609-091215-marostegui.json * 09:11 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1043 to es4 eqiad primary [[phab:T428386|T428386]]', diff saved to https://phabricator.wikimedia.org/P93942 and previous config saved to /var/cache/conftool/dbconfig/20260609-091147-marostegui.json * 09:03 jiji@cumin1003: conftool action : set/pooled=yes; selector: service=docker-registry,name=registry2005.codfw.wmnet * 08:59 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:59 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:57 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1237.eqiad.wmnet with OS trixie * 08:55 jiji@cumin1003: conftool action : set/pooled=no; selector: service=docker-registry,name=registry2005.codfw.wmnet * 08:55 jiji@cumin1003: conftool action : set/pooled=yes; selector: service=docker-registry,name=registry2004.codfw.wmnet * 08:50 jiji@cumin1003: conftool action : set/pooled=no; selector: service=docker-registry,name=registry2004.codfw.wmnet * 08:22 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=docker-registry,name=codfw * 08:22 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=docker-registry,name=eqiad * 08:08 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=docker-registry,name=eqiad * 08:08 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=docker-registry,name=codfw * 07:59 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:59 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix typoes - ayounsi@cumin1003" * 07:59 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix typoes - ayounsi@cumin1003" * 07:52 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 07:47 brouberol@dns1004: END - running authdns-update * 07:46 brouberol@dns1004: START - running authdns-update * 07:44 brouberol@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/kafka-ui: apply * 07:43 brouberol@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/kafka-ui: apply * 07:43 brouberol@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/kafka-ui: apply * 07:42 brouberol@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/kafka-ui: apply * 07:41 brouberol@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/kafka-ui: apply * 07:39 brouberol@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/kafka-ui: apply * 07:38 brouberol@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:37 brouberol@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 07:37 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie * 07:36 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.major-upgrade (exit_code=97) * 07:36 brouberol@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:36 brouberol@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:26 fceratto@dns1004: END - running authdns-update * 07:24 fceratto@dns1004: START - running authdns-update * 07:22 marostegui@dns1004: END - running authdns-update * 07:21 marostegui@dns1004: START - running authdns-update * 07:19 elukey@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:19 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Fix dse-k8s-wdqs2002 duplicate ipv6 address - elukey@cumin1003" * 07:19 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Fix dse-k8s-wdqs2002 duplicate ipv6 address - elukey@cumin1003" * 07:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1160.eqiad.wmnet with reason: Maintenance * 07:12 elukey@cumin1003: START - Cookbook sre.dns.netbox * 07:11 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1160: Repooling * 07:11 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1160: Repooling * 07:11 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1160: Repooling * 07:11 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1160: Repooling * 07:00 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:00 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1237.eqiad.wmnet with OS trixie * 06:24 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1160 [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93940 and previous config saved to /var/cache/conftool/dbconfig/20260609-062412-fceratto.json * 06:17 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 06:16 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 06:16 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 06:16 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 06:15 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 06:15 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 06:15 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 06:14 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 06:12 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1244 to s4 primary and set section read-write [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93939 and previous config saved to /var/cache/conftool/dbconfig/20260609-061222-fceratto.json * 06:11 fceratto@cumin1003: dbctl commit (dc=all): 'Set s4 eqiad as read-only for maintenance - [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93938 and previous config saved to /var/cache/conftool/dbconfig/20260609-061131-fceratto.json * 06:10 federico3: Starting s4 eqiad failover from db1160 to db1244 - [[phab:T426086|T426086]] * 06:01 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1244 with weight 0 [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93937 and previous config saved to /var/cache/conftool/dbconfig/20260609-060121-fceratto.json * 06:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 40 hosts with reason: Primary switchover s4 [[phab:T426086|T426086]] * 05:40 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie * 05:37 marostegui@dns1004: START - running authdns-update * 05:27 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1237: Upgrading db1237.eqiad.wmnet * 05:27 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1237: Upgrading db1237.eqiad.wmnet * 05:27 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:24 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1237 [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93935 and previous config saved to /var/cache/conftool/dbconfig/20260609-052420-marostegui.json * 05:23 marostegui@dns1004: START - running authdns-update * 05:23 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93934 and previous config saved to /var/cache/conftool/dbconfig/20260609-052311-marostegui.json * 05:22 marostegui@cumin1003: dbctl commit (dc=all): 'Set x1 eqiad as read-only for maintenance - [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93933 and previous config saved to /var/cache/conftool/dbconfig/20260609-052253-marostegui.json * 05:22 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T428158|T428158]] * 05:19 marostegui@cumin1003: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93932 and previous config saved to /var/cache/conftool/dbconfig/20260609-051859-marostegui.json * 05:18 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: Primary switchover x1 [[phab:T428158|T428158]] * 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.3 (duration: 02m 43s) * 03:40 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] (duration: 37m 16s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 38s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-06-08 == * 22:00 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298915{{!}}CommonSettings: Set $wgScoreSafeMode = false (T428484)]] (duration: 07m 42s) * 21:56 reedy@deploy1003: reedy: Continuing with deployment * 21:54 reedy@deploy1003: reedy: Backport for [[gerrit:1298915{{!}}CommonSettings: Set $wgScoreSafeMode = false (T428484)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:53 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1298915{{!}}CommonSettings: Set $wgScoreSafeMode = false (T428484)]] * 21:12 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298891{{!}}OOUIHTMLForm: Avoid treating form header as a clickable label (T428359)]] (duration: 08m 10s) * 21:07 mlitn@deploy1003: mlitn, neriah: Continuing with deployment * 21:05 mlitn@deploy1003: mlitn, neriah: Backport for [[gerrit:1298891{{!}}OOUIHTMLForm: Avoid treating form header as a clickable label (T428359)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:03 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1298891{{!}}OOUIHTMLForm: Avoid treating form header as a clickable label (T428359)]] * 20:43 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297162{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias]], [[gerrit:1298841{{!}}Squashed diff to master]] (duration: 07m 05s) * 20:39 mlitn@deploy1003: mlitn: Continuing with deployment * 20:38 mlitn@deploy1003: mlitn: Backport for [[gerrit:1297162{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias]], [[gerrit:1298841{{!}}Squashed diff to master]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:36 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1297162{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias]], [[gerrit:1298841{{!}}Squashed diff to master]] * 20:29 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298390{{!}}English Wikibooks: update FlaggedRevs configuration (T428329)]], [[gerrit:1298328{{!}}English Wikiversity: Add new user group "autopatrolled" (T428269)]] (duration: 08m 58s) * 20:25 mlitn@deploy1003: mlitn, vadymts1: Continuing with deployment * 20:22 mlitn@deploy1003: mlitn, vadymts1: Backport for [[gerrit:1298390{{!}}English Wikibooks: update FlaggedRevs configuration (T428329)]], [[gerrit:1298328{{!}}English Wikiversity: Add new user group "autopatrolled" (T428269)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1298390{{!}}English Wikibooks: update FlaggedRevs configuration (T428329)]], [[gerrit:1298328{{!}}English Wikiversity: Add new user group "autopatrolled" (T428269)]] * 20:03 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298879{{!}}SimpleCaptcha: Re-render captcha when edit form is redisplayed (T428437)]] (duration: 37m 43s) * 19:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:31 kharlan@deploy1003: kharlan: Continuing with deployment * 19:30 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:30 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:29 kharlan@deploy1003: kharlan: Backport for [[gerrit:1298879{{!}}SimpleCaptcha: Re-render captcha when edit form is redisplayed (T428437)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 19:28 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:27 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:25 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1298879{{!}}SimpleCaptcha: Re-render captcha when edit form is redisplayed (T428437)]] * 19:24 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab (duration: 01m 32s) * 19:23 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:22 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab * 19:20 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab (duration: 01m 40s) * 19:19 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab * 19:16 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:14 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:06 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:59 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2004 * 18:52 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2004 * 18:52 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2003 * 18:52 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2003 * 18:51 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:51 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2004 to codfw - jhancock@cumin2002" * 18:51 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2004 to codfw - jhancock@cumin2002" * 18:44 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:42 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:42 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2030 to codfw - jhancock@cumin2002" * 18:42 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2030 to codfw - jhancock@cumin2002" * 18:37 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:33 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2002 * 18:32 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2002 * 18:31 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:31 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2002 to codfw - jhancock@cumin2002" * 18:31 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2002 to codfw - jhancock@cumin2002" * 18:25 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:22 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2001 * 18:22 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2001 * 18:21 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:21 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: updating dse-k8s-wdqs2001 to codfw - jhancock@cumin2002" * 18:21 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: updating dse-k8s-wdqs2001 to codfw - jhancock@cumin2002" * 18:17 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:02 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T427286|T427286]] (duration: 00m 12s) * 18:02 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T427286|T427286]] * 17:37 jnuche@deploy1003: Installation of scap version "4.268.0" completed for 2 hosts * 17:35 jnuche@deploy1003: Installing scap version "4.268.0" for 2 host(s) * 17:21 claime: restarting varnish-frontend service on cp6012 * 17:21 claime: restarting varnish-frontend service on cp6011 * 17:21 claime: restarted varnish-frontend service on cp6009 * 17:13 taavi: bounce sirenbot to get it to re-join a channel * 17:05 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 17:05 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:58 urbanecm@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 16:57 urbanecm@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 16:55 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 16:53 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 16:53 urbanecm@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 16:52 urbanecm@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 16:30 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 16:29 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 16:29 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 16:28 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 16:28 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 16:28 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 16:28 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 16:27 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 16:27 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 16:26 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 16:26 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 16:25 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 16:18 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 16:17 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 16:17 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 16:16 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 16:16 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 16:16 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 16:16 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 16:15 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 16:14 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 16:14 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 16:14 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 16:14 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 16:13 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 16:13 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 16:13 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 16:12 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 16:12 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 16:10 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 16:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 16:10 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 16:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 16:10 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 16:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 16:09 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 16:08 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 16:08 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 16:07 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 16:06 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 15:57 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2042: repool after upgrade * 15:45 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db[2183-2184].codfw.wmnet * 15:45 jynus@cumin2002: START - Cookbook sre.hosts.remove-downtime for db[2183-2184].codfw.wmnet * 15:18 jynus: dbmaint on backup1-codfw@codfw ([[phab:T428467|T428467]]) * 15:12 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2042: repool after upgrade * 15:12 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 15:09 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 15:09 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 15:09 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 15:07 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2042.codfw.wmnet with OS trixie * 15:04 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 15:04 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 15:03 jynus@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db[2183-2184].codfw.wmnet with reason: Switchover db * 15:03 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 15:03 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 15:02 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 15:01 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply * 15:00 eevans@deploy1003: helmfile [staging] START helmfile.d/services/data-gateway: apply * 14:59 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:55 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:55 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:54 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:50 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:50 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:50 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:49 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:49 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2042.codfw.wmnet with reason: host reimage * 14:42 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2042.codfw.wmnet with reason: host reimage * 14:32 Lucas_WMDE: UTC afternoon backport+config window done * 14:32 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298709{{!}}Add translatable messages for WikiProject names (T427804)]], [[gerrit:1298710{{!}}Use translatable messages for WikiProject links (T427804)]], [[gerrit:1297644{{!}}WikiProject links - remove 'text' config (T427804)]] (duration: 31m 57s) * 14:27 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 14:26 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2042.codfw.wmnet with OS trixie * 14:26 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 14:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2042: Upgrading es2042.codfw.wmnet * 14:25 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2042: Upgrading es2042.codfw.wmnet * 14:25 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:24 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2043 to es4 codfw primary [[phab:T428386|T428386]]', diff saved to https://phabricator.wikimedia.org/P93926 and previous config saved to /var/cache/conftool/dbconfig/20260608-142423-marostegui.json * 14:23 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 14:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1041: repool after maintenance * 14:19 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Continuing with deployment * 14:18 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Backport for [[gerrit:1298709{{!}}Add translatable messages for WikiProject names (T427804)]], [[gerrit:1298710{{!}}Use translatable messages for WikiProject links (T427804)]], [[gerrit:1297644{{!}}WikiProject links - remove 'text' config (T427804)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:11 cgoubert@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=liftwing-openapi-server.* * 14:10 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp6013.* * 14:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:05 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 14:05 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 13:54 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 13:52 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 13:50 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 13:50 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 13:50 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296550{{!}}hCaptcha: Don't show AbuseFilter CAPTCHA for wbsetclaim API (T427608)]] (duration: 08m 31s) * 13:48 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 13:46 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 13:43 cgoubert@dns1004: END - running authdns-update * 13:43 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296550{{!}}hCaptcha: Don't show AbuseFilter CAPTCHA for wbsetclaim API (T427608)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:41 cgoubert@dns1004: START - running authdns-update * 13:41 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296550{{!}}hCaptcha: Don't show AbuseFilter CAPTCHA for wbsetclaim API (T427608)]] * 13:39 urbanecm@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * {{safesubst:SAL entry|1=13:38 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298758{{!}}feat(V2): toggle experiment features based on custom url override (T424646)]], [[gerrit:1298762{{!}}specialCreateAccount: use GECreateAccountExperimentV2 instead of hook (T424646)]], [[gerrit:1298764{{!}}fix: correctly read experiments param on Special:UserLogin]], [[gerrit:1298765{{!}}signup.js: use JS var instead of TestKitchen to show exp}} * 13:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1041: repool after maintenance * 13:38 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 13:38 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:37 urbanecm@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 13:36 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 13:35 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1041.eqiad.wmnet with OS trixie * 13:34 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 13:34 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2041: repool after upgrade * 13:34 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Continuing with deployment * 13:34 urbanecm@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 13:32 urbanecm@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * {{safesubst:SAL entry|1=13:30 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Backport for [[gerrit:1298758{{!}}feat(V2): toggle experiment features based on custom url override (T424646)]], [[gerrit:1298762{{!}}specialCreateAccount: use GECreateAccountExperimentV2 instead of hook (T424646)]], [[gerrit:1298764{{!}}fix: correctly read experiments param on Special:UserLogin]], [[gerrit:1298765{{!}}signup.js: use JS var instead of TestKitchen to show}} * {{safesubst:SAL entry|1=13:29 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298758{{!}}feat(V2): toggle experiment features based on custom url override (T424646)]], [[gerrit:1298762{{!}}specialCreateAccount: use GECreateAccountExperimentV2 instead of hook (T424646)]], [[gerrit:1298764{{!}}fix: correctly read experiments param on Special:UserLogin]], [[gerrit:1298765{{!}}signup.js: use JS var instead of TestKitchen to show expe}} * 13:21 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298418{{!}}NewUserMessage: Add $wgNewUserMessageOnAutoCreateFirstEdit (T426206)]], [[gerrit:1298717{{!}}Replace NewUserMessageOnAutoCreateFirstEdit with wgNewUserMessageOnFirstEdit (T426206)]], [[gerrit:1298734{{!}}Enable wgNewUserMessageOnFirstEdit on incubatorwiki (T426206)]] (duration: 11m 06s) * 13:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1041.eqiad.wmnet with reason: host reimage * 13:17 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Continuing with deployment * 13:12 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 13:12 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Backport for [[gerrit:1298418{{!}}NewUserMessage: Add $wgNewUserMessageOnAutoCreateFirstEdit (T426206)]], [[gerrit:1298717{{!}}Replace NewUserMessageOnAutoCreateFirstEdit with wgNewUserMessageOnFirstEdit (T426206)]], [[gerrit:1298734{{!}}Enable wgNewUserMessageOnFirstEdit on incubatorwiki (T426206)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki * 13:12 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 13:12 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1041.eqiad.wmnet with reason: host reimage * 13:11 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 13:11 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 13:10 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298418{{!}}NewUserMessage: Add $wgNewUserMessageOnAutoCreateFirstEdit (T426206)]], [[gerrit:1298717{{!}}Replace NewUserMessageOnAutoCreateFirstEdit with wgNewUserMessageOnFirstEdit (T426206)]], [[gerrit:1298734{{!}}Enable wgNewUserMessageOnFirstEdit on incubatorwiki (T426206)]] * 12:57 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298767{{!}}Follow-up: Allow CaptchaConsequence to be skipped via hook (T427608)]] (duration: 06m 20s) * 12:57 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1041.eqiad.wmnet with OS trixie * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:56 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1041: Upgrading es1041.eqiad.wmnet * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:55 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1041: Upgrading es1041.eqiad.wmnet * 12:55 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:54 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:53 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 12:53 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1298767{{!}}Follow-up: Allow CaptchaConsequence to be skipped via hook (T427608)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:51 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:51 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1298767{{!}}Follow-up: Allow CaptchaConsequence to be skipped via hook (T427608)]] * 12:49 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:49 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2041: repool after upgrade * 12:49 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:47 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:46 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 12:44 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 12:43 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 12:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 12:41 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 12:40 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2063.codfw.wmnet with OS bullseye * 12:32 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2062.codfw.wmnet with OS bullseye * 12:27 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2041.codfw.wmnet with OS trixie * 12:21 joal@deploy1003: Finished deploy [analytics/refinery@d67c584] (thin): Regular analytics weekly train THIN [analytics/refinery@d67c584f] (duration: 02m 00s) * 12:19 joal@deploy1003: Started deploy [analytics/refinery@d67c584] (thin): Regular analytics weekly train THIN [analytics/refinery@d67c584f] * 12:19 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2063.codfw.wmnet with reason: host reimage * 12:18 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 12:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 12:16 joal@deploy1003: Finished deploy [analytics/refinery@d67c584]: Regular analytics weekly train [analytics/refinery@d67c584f] (duration: 07m 52s) * 12:15 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2063.codfw.wmnet with reason: host reimage * 12:13 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2062.codfw.wmnet with reason: host reimage * 12:09 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2041.codfw.wmnet with reason: host reimage * 12:08 joal@deploy1003: Started deploy [analytics/refinery@d67c584]: Regular analytics weekly train [analytics/refinery@d67c584f] * 12:08 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2062.codfw.wmnet with reason: host reimage * 12:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add eqiad e8 public vlans - ayounsi@cumin1003" * 12:06 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add eqiad e8 public vlans - ayounsi@cumin1003" * 12:03 joal@deploy1003: Finished deploy [analytics/refinery@d67c584] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@d67c584f] (duration: 02m 00s) * 12:03 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2041.codfw.wmnet with reason: host reimage * 12:01 joal@deploy1003: Started deploy [analytics/refinery@d67c584] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@d67c584f] * 12:01 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 12:00 ayounsi@cumin1003: END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) * 12:00 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 12:00 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 12:00 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:57 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2063 * 11:57 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2063 * 11:57 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2063 * 11:57 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2063.codfw.wmnet 52.16.192.10.in-addr.arpa 2.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:56 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2063.codfw.wmnet 52.16.192.10.in-addr.arpa 2.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:56 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:56 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2063 - mvernon@cumin2002" * 11:56 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2063 - mvernon@cumin2002" * 11:51 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:51 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2063 * 11:50 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2063.codfw.wmnet with OS bullseye * 11:50 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2062 * 11:50 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2062 * 11:49 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2062 * 11:49 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2062.codfw.wmnet 123.0.192.10.in-addr.arpa 3.2.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:49 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2062.codfw.wmnet 123.0.192.10.in-addr.arpa 3.2.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:49 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:49 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2062 - mvernon@cumin2002" * 11:49 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2062 - mvernon@cumin2002" * 11:47 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2041.codfw.wmnet with OS trixie * 11:45 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2041: Upgrading es2041.codfw.wmnet * 11:45 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2041: Upgrading es2041.codfw.wmnet * 11:44 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:44 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.major-upgrade (exit_code=97) * 11:44 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:44 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1042: repool after maintenance * 11:43 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:43 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2062 * 11:42 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2062.codfw.wmnet with OS bullseye * 11:30 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298728{{!}}SpecialMediaSearch: Prefer thumb steps over thumb limits (T424032)]] (duration: 17m 39s) * 11:25 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 11:18 Raine: progressively switching shellbox to bookworm (start) * 11:15 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 11:14 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 11:14 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1298728{{!}}SpecialMediaSearch: Prefer thumb steps over thumb limits (T424032)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:13 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 11:12 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 11:12 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1298728{{!}}SpecialMediaSearch: Prefer thumb steps over thumb limits (T424032)]] * 11:02 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be2062 * 11:02 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be2063 * 10:58 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1042: repool after maintenance * 10:58 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:56 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1042.eqiad.wmnet with OS trixie * 10:47 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298721{{!}}GuessedThumbnailInfo: Also allow showing webp originals (T428202)]] (duration: 16m 41s) * 10:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1042.eqiad.wmnet with reason: host reimage * 10:39 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 10:39 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 10:38 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 10:36 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2160.codfw.wmnet * 10:36 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2160.codfw.wmnet * 10:35 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2043: repool after upgrade * 10:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2160.codfw.wmnet with reason: Reboot * 10:34 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1298721{{!}}GuessedThumbnailInfo: Also allow showing webp originals (T428202)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:34 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1042.eqiad.wmnet with reason: host reimage * 10:30 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1298721{{!}}GuessedThumbnailInfo: Also allow showing webp originals (T428202)]] * 10:18 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1042.eqiad.wmnet with OS trixie * 10:18 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:18 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:18 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:18 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:15 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1042: Upgrading es1042.eqiad.wmnet * 10:14 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:14 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:14 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:14 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:13 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1042: Upgrading es1042.eqiad.wmnet * 10:13 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:12 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be2063 * 10:09 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be2062 * 10:07 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:07 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:07 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:06 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 09:52 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply * 09:52 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply * 09:50 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply * 09:49 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply * 09:49 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2043: repool after upgrade * 09:49 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2043.codfw.wmnet with OS trixie * 09:44 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 09:44 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 09:42 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: sync * 09:42 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: sync * 09:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2043.codfw.wmnet with reason: host reimage * 09:27 jelto@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org * 09:23 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2043.codfw.wmnet with reason: host reimage * 09:17 jelto@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org * 09:15 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: sync * 09:15 ozge@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: sync * 09:07 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2043.codfw.wmnet with OS trixie * 09:06 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2043: Upgrading es2043.codfw.wmnet * 09:06 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2043: Upgrading es2043.codfw.wmnet * 09:05 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:41 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1217.eqiad.wmnet with OS trixie * 08:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1217.eqiad.wmnet with reason: host reimage * 08:15 taavi@cumin1003: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database urwikisource ([[phab:T415977|T415977]]) * 08:14 taavi@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database urwikisource ([[phab:T415977|T415977]]) * 08:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1217.eqiad.wmnet with reason: host reimage * 08:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2052: repool after upgrade * 08:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1051: repool after maintenance * 08:03 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Managing sanitization for wikis urwikisource in section s5 * 07:55 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1217.eqiad.wmnet with OS trixie * 07:53 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1217.eqiad.wmnet with reason: reimage * 07:53 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis urwikisource in section s5 * 07:52 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Checking sanitization for wikis urwikisource in section s5 * 07:50 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis urwikisource in section s5 * 07:50 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.sanitize-wiki (exit_code=97) Managing sanitization for wikis urwikisource in section s5 * 07:50 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis urwikisource in section s5 * 07:44 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297681{{!}}Global rollout - Sub-ref deployments to Group 0, Group 1 and frwiki (T425662)]] (duration: 32m 51s) * 07:32 wmde-fisch@deploy1003: wmde-fisch, lilients: Continuing with deployment * 07:29 wmde-fisch@deploy1003: wmde-fisch, lilients: Backport for [[gerrit:1297681{{!}}Global rollout - Sub-ref deployments to Group 0, Group 1 and frwiki (T425662)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:21 elukey: upgrade sudo package on an-* hosts for [[phab:T428384|T428384]] * 07:18 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2052: repool after upgrade * 07:18 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1051: repool after maintenance * 07:17 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:17 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:12 taavi@cumin1003: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database urwikisource ([[phab:T415977|T415977]]) * 07:12 elukey: upgrade exim4 packages on seaborgium for security upgrades * 07:11 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1297681{{!}}Global rollout - Sub-ref deployments to Group 0, Group 1 and frwiki (T425662)]] * 06:36 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1051.eqiad.wmnet with OS trixie * 06:20 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1051.eqiad.wmnet with reason: host reimage * 06:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1051.eqiad.wmnet with reason: host reimage * 06:15 taavi@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database urwikisource ([[phab:T415977|T415977]]) * 05:58 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1051.eqiad.wmnet with OS trixie * 05:54 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2052.codfw.wmnet with OS trixie * 05:44 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool es1051: Upgrading es1051.eqiad.wmnet * 05:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2052.codfw.wmnet with reason: host reimage * 05:35 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2052.codfw.wmnet with reason: host reimage * 05:35 marostegui@dns1004: END - running authdns-update * 05:34 marostegui@dns1004: START - running authdns-update * 05:33 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1051: Upgrading es1051.eqiad.wmnet * 05:33 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:31 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1054 to es3 eqiad primary [[phab:T428050|T428050]]', diff saved to https://phabricator.wikimedia.org/P93895 and previous config saved to /var/cache/conftool/dbconfig/20260608-053156-marostegui.json * 05:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2052.codfw.wmnet with OS trixie * 05:18 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2052: Upgrading es2052.codfw.wmnet * 05:18 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2052: Upgrading es2052.codfw.wmnet * 05:18 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade == 2026-06-07 == * 16:32 elukey: `elukey@cumin1003:~$ sudo cumin 'cp6* and not cp6014* and not cp6010*' "varnish-frontend-restart" -b 1` * 16:29 elukey: restart varnish-frontend on cp6014 == 2026-06-06 == * 09:07 ammarpad@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=hewiki --logwiki=metawiki W.Mechelke Tungsten_Mechelke # [[phab:T428182|T428182]] == 2026-06-05 == * 22:16 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 21:01 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=10 --verbose` (after stopping the other commons scan) * 20:56 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=30 --verbose` (after stopping the other commons scan) * 20:20 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290093{{!}}Enable wmgUseUrlShortenerLegacy on test2wiki (T107188)]] (duration: 10m 02s) * 20:16 krinkle@deploy1003: krinkle: Continuing with deployment * 20:12 krinkle@deploy1003: krinkle: Backport for [[gerrit:1290093{{!}}Enable wmgUseUrlShortenerLegacy on test2wiki (T107188)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:10 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1290093{{!}}Enable wmgUseUrlShortenerLegacy on test2wiki (T107188)]] * 16:45 jgreen@dns1004: END - running authdns-update * 16:44 jgreen@dns1004: START - running authdns-update * 16:17 dzahn@dns1005: END - running authdns-update * 16:17 mutante: DNS - adding new project language "mag" - Magahi - a language spoken in India and Nepal by about 12 million native speakers ([[phab:T428266|T428266]]) * 16:16 dzahn@dns1005: START - running authdns-update * 14:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:38 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:37 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 12:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:30 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:30 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 12:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2202.codfw.wmnet with reason: Reboot * 12:28 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:28 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 12:08 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:07 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 12:07 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:06 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 11:29 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 11:28 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:55 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:54 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:31 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1054: repool after upgrade * 08:08 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 08:07 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 08:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 08:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/kafka-ui: apply * 07:39 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1054: repool after upgrade * 07:38 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 07:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 07:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/kafka-ui: apply * 07:07 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 06:01 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1054.eqiad.wmnet with OS trixie * 05:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1054.eqiad.wmnet with reason: host reimage * 05:37 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1054.eqiad.wmnet with reason: host reimage * 05:22 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1054.eqiad.wmnet with OS trixie * 05:21 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1054: Upgrading es1054.eqiad.wmnet * 05:21 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1054: Upgrading es1054.eqiad.wmnet * 05:20 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 01:55 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1010.eqiad.wmnet with OS trixie * 01:39 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: host reimage * 01:32 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: host reimage * 01:16 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1010.eqiad.wmnet with OS trixie * 00:56 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1007.eqiad.wmnet with OS trixie * 00:40 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1007.eqiad.wmnet with reason: host reimage * 00:33 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1007.eqiad.wmnet with reason: host reimage * 00:17 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1007.eqiad.wmnet with OS trixie * 00:02 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297268{{!}}Redirect unknown wikinews languages to portal (T427126)]] (duration: 07m 02s) == 2026-06-04 == * 23:57 ladsgroup@deploy1003: ladsgroup, pppery: Continuing with deployment * 23:57 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1006.eqiad.wmnet with OS trixie * 23:57 ladsgroup@deploy1003: ladsgroup, pppery: Backport for [[gerrit:1297268{{!}}Redirect unknown wikinews languages to portal (T427126)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:55 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1297268{{!}}Redirect unknown wikinews languages to portal (T427126)]] * 23:40 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1006.eqiad.wmnet with reason: host reimage * 23:36 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1006.eqiad.wmnet with reason: host reimage * 23:20 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1006.eqiad.wmnet with OS trixie * 21:28 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host releases1003.eqiad.wmnet with OS trixie * 21:04 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases1003.eqiad.wmnet with reason: host reimage * 20:58 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on releases1003.eqiad.wmnet with reason: host reimage * 20:50 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5030.* * 20:42 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host releases1003.eqiad.wmnet with OS trixie * 20:27 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1100.eqiad.wmnet,service=(cdn{{!}}ats-be) * 20:26 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6013.drmrs.wmnet,service=(cdn{{!}}ats-be) * 20:20 brett@dns1006: END - running authdns-update * 20:19 brett@dns1006: START - running authdns-update * 20:18 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5030.eqsin.wmnet with OS trixie * 20:10 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296015{{!}}Deploy PRV to 6 wikis (T427851)]] (duration: 07m 39s) * 20:08 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist group2.dblist extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` * 20:06 arlolra@deploy1003: arlolra: Continuing with deployment * 20:04 arlolra@deploy1003: arlolra: Backport for [[gerrit:1296015{{!}}Deploy PRV to 6 wikis (T427851)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:02 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1296015{{!}}Deploy PRV to 6 wikis (T427851)]] * 19:49 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5030.eqsin.wmnet with reason: host reimage * 19:43 cmooney@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5030.eqsin.wmnet with reason: host reimage * 19:15 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5030 * 19:15 cmooney@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5030 * 19:14 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cp5030 * 19:14 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5030.eqsin.wmnet 27.0.132.10.in-addr.arpa 7.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:14 cmooney@cumin1003: START - Cookbook sre.dns.wipe-cache cp5030.eqsin.wmnet 27.0.132.10.in-addr.arpa 7.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:14 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:14 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5030 - cmooney@cumin1003" * 19:13 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5030 - cmooney@cumin1003" * 19:09 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 19:08 cmooney@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5030 * 19:08 cmooney@cumin1003: START - Cookbook sre.hosts.reimage for host cp5030.eqsin.wmnet with OS trixie * 18:51 cmooney@dns2005: END - running authdns-update * 18:50 cmooney@dns2005: START - running authdns-update * 18:43 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:42 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove IPs that had been used for eqsin cr links - cmooney@cumin1003" * 18:40 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove IPs that had been used for eqsin cr links - cmooney@cumin1003" * 18:37 sukhe: sukhe@cp6013:~$ sudo traffic_server -C clear_cache * 18:36 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 18:08 dancy@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.5 refs [[phab:T423914|T423914]] * 17:17 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297751{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297752{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] (duration: 06m 40s) * 17:13 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 17:13 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297751{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297752{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:11 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297751{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297752{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] * 16:55 topranks: shift traffic off cr1-esams et-1/0/1 link to asw1-by27-esams [[phab:T427056|T427056]] * 16:45 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297741{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297742{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] (duration: 13m 58s) * 16:41 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 16:33 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297741{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297742{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:31 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297741{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297742{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] * 16:17 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 16:03 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297740{{!}}hCaptcha: Move ConfirmEditCaptchaClass hook inside hCaptcha block (T428183)]] (duration: 10m 21s) * 16:03 elukey: uploaded spicerack_12.7.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia * 15:59 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 15:55 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297740{{!}}hCaptcha: Move ConfirmEditCaptchaClass hook inside hCaptcha block (T428183)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:53 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297740{{!}}hCaptcha: Move ConfirmEditCaptchaClass hook inside hCaptcha block (T428183)]] * 15:44 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5030.* * 15:41 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2007.codfw.wmnet with OS trixie * 15:39 ladsgroup@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0) * 15:28 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views * 15:24 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297730{{!}}ptwiki: Disable Article Guidance experiment (T426871)]] (duration: 07m 26s) * 15:24 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2007.codfw.wmnet with reason: host reimage * 15:20 sbisson@deploy1003: sbisson: Continuing with deployment * 15:19 sbisson@deploy1003: sbisson: Backport for [[gerrit:1297730{{!}}ptwiki: Disable Article Guidance experiment (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:19 jayme@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2007.codfw.wmnet with reason: host reimage * 15:17 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1297730{{!}}ptwiki: Disable Article Guidance experiment (T426871)]] * 15:13 ladsgroup@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0) * 15:06 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297724{{!}}Revert "Start reading from new file tables on commons"]] (duration: 07m 00s) * 15:05 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views * 15:02 zabe@deploy1003: zabe: Continuing with deployment * 15:01 zabe@deploy1003: zabe: Backport for [[gerrit:1297724{{!}}Revert "Start reading from new file tables on commons"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:59 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1297724{{!}}Revert "Start reading from new file tables on commons"]] * 14:57 zabe@deploy1003: Finished scap sync-world: [[phab:T416548|T416548]] (duration: 05m 10s) * 14:56 jayme@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-main2007.codfw.wmnet with OS trixie * 14:52 zabe@deploy1003: Started scap sync-world: [[phab:T416548|T416548]] * 14:50 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:49 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:43 zabe@deploy1003: sync-world aborted: Backport for [[gerrit:1270513{{!}}Start reading from new file tables on commons (T416548)]] (duration: 03m 58s) * 14:43 zabe@deploy1003: zabe: Continuing with deployment * 14:41 zabe@deploy1003: zabe: Backport for [[gerrit:1270513{{!}}Start reading from new file tables on commons (T416548)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:40 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f1-codfw * 14:40 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device lsw1-f1-codfw * 14:39 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1270513{{!}}Start reading from new file tables on commons (T416548)]] * 14:36 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297711{{!}}hCaptcha: Enable for MobileFrontend in some Group 2 wikis (T425940)]] (duration: 08m 20s) * 14:32 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:30 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297711{{!}}hCaptcha: Enable for MobileFrontend in some Group 2 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:29 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1057: repool after upgrade * 14:28 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297711{{!}}hCaptcha: Enable for MobileFrontend in some Group 2 wikis (T425940)]] * 14:20 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 14:16 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:15 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply * 14:15 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:15 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply * 14:13 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297704{{!}}Use the globalblock-local-status right over globalblock-whitelist (T277942)]], [[gerrit:1296620{{!}}core-Permissions: Stop assigning unused globalblock-whitelist right (T277942)]] (duration: 06m 46s) * 14:10 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 14:08 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:08 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297704{{!}}Use the globalblock-local-status right over globalblock-whitelist (T277942)]], [[gerrit:1296620{{!}}core-Permissions: Stop assigning unused globalblock-whitelist right (T277942)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:07 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:06 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply * 14:06 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297704{{!}}Use the globalblock-local-status right over globalblock-whitelist (T277942)]], [[gerrit:1296620{{!}}core-Permissions: Stop assigning unused globalblock-whitelist right (T277942)]] * 14:06 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:06 tappof: bump space for prometheus k8s-aux in eqiad * 14:05 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply * 14:05 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:04 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply * 13:56 _joe_: transferred requestctl api tokens for all ops to the db ([[phab:T428119|T428119]]) * 13:56 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2050 to es3 codfw primary [[phab:T428050|T428050]]', diff saved to https://phabricator.wikimedia.org/P93878 and previous config saved to /var/cache/conftool/dbconfig/20260604-135631-marostegui.json * 13:56 Dreamy_Jazz: Afternoon UTC backport window done * 13:54 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297700{{!}}Revert "hCaptcha: Provide always challenge sitekey for account creation"]] (duration: 13m 38s) * 13:51 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 13:50 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 13:47 sukhe: sukhe@cp6011:~$ sudo -i varnish-frontend-restart * 13:44 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1057: repool after upgrade * 13:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:43 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297700{{!}}Revert "hCaptcha: Provide always challenge sitekey for account creation"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:41 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1057.eqiad.wmnet with OS trixie * 13:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297700{{!}}Revert "hCaptcha: Provide always challenge sitekey for account creation"]] * 13:38 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297692{{!}}hCaptcha: Provide always challenge sitekey for account creation (T421041)]] (duration: 05m 27s) * 13:38 dreamyjazz@deploy1003: dreamyjazz: Rolling back deployment * 13:36 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: down * 13:35 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297692{{!}}hCaptcha: Provide always challenge sitekey for account creation (T421041)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:33 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297692{{!}}hCaptcha: Provide always challenge sitekey for account creation (T421041)]] * 13:31 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295978{{!}}Update config for WikiProjects linking prototype (T427804)]] (duration: 17m 13s) * 13:26 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Continuing with deployment * 13:25 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1057.eqiad.wmnet with reason: host reimage * 13:17 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1057.eqiad.wmnet with reason: host reimage * 13:16 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Backport for [[gerrit:1295978{{!}}Update config for WikiProjects linking prototype (T427804)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:14 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1295978{{!}}Update config for WikiProjects linking prototype (T427804)]] * 13:13 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:13 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1220: Migration of db1220.eqiad.wmnet completed * 13:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: down * 13:12 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1224', diff saved to https://phabricator.wikimedia.org/P93875 and previous config saved to /var/cache/conftool/dbconfig/20260604-131219-marostegui.json * 13:00 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1057.eqiad.wmnet with OS trixie * 13:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1057: Upgrading es1057.eqiad.wmnet * 12:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1057: Upgrading es1057.eqiad.wmnet * 12:59 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:56 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296557{{!}}wmf-config: Skip CAPTCHA for action=mcrundo (T427612)]] (duration: 08m 30s) * 12:52 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Continuing with deployment * 12:50 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Backport for [[gerrit:1296557{{!}}wmf-config: Skip CAPTCHA for action=mcrundo (T427612)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2050: repool after upgrade * 12:48 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296557{{!}}wmf-config: Skip CAPTCHA for action=mcrundo (T427612)]] * 12:37 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 12:37 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 12:28 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1220: Migration of db1220.eqiad.wmnet completed * 12:20 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1220.eqiad.wmnet with OS trixie * 12:04 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2050: repool after upgrade * 12:04 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 12:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1220.eqiad.wmnet with reason: host reimage * 11:59 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1220.eqiad.wmnet with reason: host reimage * 11:42 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1220.eqiad.wmnet with OS trixie * 11:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2050.codfw.wmnet with OS trixie * 11:40 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1220: Upgrading db1220.eqiad.wmnet * 11:37 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1220: Upgrading db1220.eqiad.wmnet * 11:36 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1179: Migration of db1179.eqiad.wmnet completed * 11:23 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2050.codfw.wmnet with reason: host reimage * 11:16 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2050.codfw.wmnet with reason: host reimage * 11:00 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2050.codfw.wmnet with OS trixie * 11:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2050: Upgrading es2050.codfw.wmnet * 10:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2050: Upgrading es2050.codfw.wmnet * 10:59 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:59 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2057: repool after upgrade * 10:58 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:55 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 10:46 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1179: Migration of db1179.eqiad.wmnet completed * 10:38 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1179.eqiad.wmnet with OS trixie * 10:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1179.eqiad.wmnet with reason: host reimage * 10:16 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply * 10:15 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply * 10:15 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply * 10:15 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/kartotherian: apply * 10:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1179.eqiad.wmnet with reason: host reimage * 10:13 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2057: repool after upgrade * 10:13 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2057.codfw.wmnet with OS trixie * 09:59 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1179.eqiad.wmnet with OS trixie * 09:58 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1179: Upgrading db1179.eqiad.wmnet * 09:58 jynus: redoing m2 backups after grant change [[phab:T411111|T411111]] * 09:57 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1179: Upgrading db1179.eqiad.wmnet * 09:56 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:54 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2057.codfw.wmnet with reason: host reimage * 09:53 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 09:49 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2057.codfw.wmnet with reason: host reimage * 09:39 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:39 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Migration of db1224.eqiad.wmnet completed * 09:38 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 09:37 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 09:36 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 09:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/kafka-ui: apply * 09:33 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2057.codfw.wmnet with OS trixie * 09:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2057: Upgrading es2057.codfw.wmnet * 09:32 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2057: Upgrading es2057.codfw.wmnet * 09:31 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:26 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=30 --sleep=60 --verbose` * 09:25 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist "group0.dblist + group1.dblist - mediamoderation-continuous-scan.dblist" extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` * 08:54 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Introduce pluggable authentication - oblivian@cumin1003" * 08:54 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Introduce pluggable authentication - oblivian@cumin1003 * 08:53 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Migration of db1224.eqiad.wmnet completed * 08:53 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Introduce pluggable authentication - oblivian@cumin1003 * 08:53 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Introduce pluggable authentication - oblivian@cumin1003" * 08:29 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 08:29 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 08:24 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 08:24 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 08:21 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 08:21 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1224.eqiad.wmnet with OS trixie * 08:21 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 08:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1224.eqiad.wmnet with reason: host reimage * 08:02 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2249.codfw.wmnet with reason: upgrade * 08:00 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1224.eqiad.wmnet with reason: host reimage * 07:53 marostegui: Install mariadb 10.11.17 on db2249 [[phab:T427345|T427345]] * 07:43 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1224.eqiad.wmnet with OS trixie * 07:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1224: Upgrading db1224.eqiad.wmnet * 07:41 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1224: Upgrading db1224.eqiad.wmnet * 07:41 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:39 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:39 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1255: Migration of db1255.eqiad.wmnet completed * 07:34 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297536{{!}}hCaptcha risk scores: VE plugin to collect risk scores for block notices (T426943)]], [[gerrit:1297200{{!}}hCaptcha: Render a fresh mobile widget for each captcha attempt (T425929)]], [[gerrit:1297173{{!}}hCaptcha: Enable risk-score collection for users blocked by IP blocks (T424629)]] (duration: 08m 56s) * 07:29 kharlan@deploy1003: kharlan, harroyo-wmf: Continuing with deployment * 07:27 kharlan@deploy1003: kharlan, harroyo-wmf: Backport for [[gerrit:1297536{{!}}hCaptcha risk scores: VE plugin to collect risk scores for block notices (T426943)]], [[gerrit:1297200{{!}}hCaptcha: Render a fresh mobile widget for each captcha attempt (T425929)]], [[gerrit:1297173{{!}}hCaptcha: Enable risk-score collection for users blocked by IP blocks (T424629)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwd * 07:25 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297536{{!}}hCaptcha risk scores: VE plugin to collect risk scores for block notices (T426943)]], [[gerrit:1297200{{!}}hCaptcha: Render a fresh mobile widget for each captcha attempt (T425929)]], [[gerrit:1297173{{!}}hCaptcha: Enable risk-score collection for users blocked by IP blocks (T424629)]] * 07:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2191: Migration of db2191.codfw.wmnet completed * 07:12 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297550{{!}}Revert "EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion"]] (duration: 06m 45s) * 07:08 kharlan@deploy1003: kharlan: Continuing with deployment * 07:08 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297550{{!}}Revert "EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:06 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297550{{!}}Revert "EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion"]] * 07:04 otto@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297260{{!}}EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] (duration: 399m 30s) * 07:03 otto@deploy1003: otto: Rolling back deployment * 06:53 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1255: Migration of db1255.eqiad.wmnet completed * 06:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1255.eqiad.wmnet with OS trixie * 06:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2191: Migration of db2191.codfw.wmnet completed * 06:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1255.eqiad.wmnet with reason: host reimage * 06:32 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2191.codfw.wmnet with OS trixie * 06:31 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1255.eqiad.wmnet with reason: host reimage * 06:16 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1255.eqiad.wmnet with OS trixie * 06:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2191.codfw.wmnet with reason: host reimage * 06:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1255: Upgrading db1255.eqiad.wmnet * 06:12 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1255: Upgrading db1255.eqiad.wmnet * 06:12 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2191.codfw.wmnet with reason: host reimage * 06:04 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db1255 [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93836 and previous config saved to /var/cache/conftool/dbconfig/20260604-060428-cwilliams.json * 06:03 cwilliams@dns1004: END - running authdns-update * 06:02 cwilliams@dns1004: START - running authdns-update * 05:54 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db1258 to x3 primary and set section read-write [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93835 and previous config saved to /var/cache/conftool/dbconfig/20260604-055429-cwilliams.json * 05:53 cwilliams@cumin1003: dbctl commit (dc=all): 'Set x3 eqiad as read-only for maintenance - [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93834 and previous config saved to /var/cache/conftool/dbconfig/20260604-055346-cwilliams.json * 05:53 cezmunsta: Starting x3 eqiad failover from db1255 to db1258 - [[phab:T427895|T427895]] * 05:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2191.codfw.wmnet with OS trixie * 05:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2191: Upgrading db2191.codfw.wmnet * 05:50 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2191: Upgrading db2191.codfw.wmnet * 05:50 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db1258 with weight 0 [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93833 and previous config saved to /var/cache/conftool/dbconfig/20260604-055021-cwilliams.json * 05:50 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:50 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Primary switchover x3 [[phab:T427895|T427895]] * 05:48 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 05:46 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2191 [[phab:T428120|T428120]]', diff saved to https://phabricator.wikimedia.org/P93832 and previous config saved to /var/cache/conftool/dbconfig/20260604-054614-marostegui.json * 05:45 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db2215 to x1 primary [[phab:T428120|T428120]]', diff saved to https://phabricator.wikimedia.org/P93831 and previous config saved to /var/cache/conftool/dbconfig/20260604-054528-marostegui.json * 05:44 marostegui: Starting x1 codfw failover from db2191 to db2215 - [[phab:T428120|T428120]] * 05:27 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: Primary switchover x1 [[phab:T428120|T428120]] * 05:27 marostegui@cumin1003: dbctl commit (dc=all): 'Set db2215 with weight 0 [[phab:T428120|T428120]]', diff saved to https://phabricator.wikimedia.org/P93830 and previous config saved to /var/cache/conftool/dbconfig/20260604-052722-marostegui.json * 05:19 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 03:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93829 and previous config saved to /var/cache/conftool/dbconfig/20260604-034546-fceratto.json * 03:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P93828 and previous config saved to /var/cache/conftool/dbconfig/20260604-033538-fceratto.json * 03:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P93827 and previous config saved to /var/cache/conftool/dbconfig/20260604-032531-fceratto.json * 03:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93826 and previous config saved to /var/cache/conftool/dbconfig/20260604-031523-fceratto.json * 03:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1263 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93825 and previous config saved to /var/cache/conftool/dbconfig/20260604-030710-fceratto.json * 03:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1263.eqiad.wmnet with reason: Maintenance * 03:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93824 and previous config saved to /var/cache/conftool/dbconfig/20260604-030642-fceratto.json * 02:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P93823 and previous config saved to /var/cache/conftool/dbconfig/20260604-025634-fceratto.json * 02:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P93822 and previous config saved to /var/cache/conftool/dbconfig/20260604-024627-fceratto.json * 02:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93821 and previous config saved to /var/cache/conftool/dbconfig/20260604-023619-fceratto.json * 02:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1262 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93820 and previous config saved to /var/cache/conftool/dbconfig/20260604-022809-fceratto.json * 02:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Maintenance * 02:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93819 and previous config saved to /var/cache/conftool/dbconfig/20260604-022742-fceratto.json * 02:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P93818 and previous config saved to /var/cache/conftool/dbconfig/20260604-021734-fceratto.json * 02:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P93817 and previous config saved to /var/cache/conftool/dbconfig/20260604-020726-fceratto.json * 01:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93816 and previous config saved to /var/cache/conftool/dbconfig/20260604-015718-fceratto.json * 01:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1261 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93815 and previous config saved to /var/cache/conftool/dbconfig/20260604-014909-fceratto.json * 01:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1261.eqiad.wmnet with reason: Maintenance * 01:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93814 and previous config saved to /var/cache/conftool/dbconfig/20260604-014841-fceratto.json * 01:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P93813 and previous config saved to /var/cache/conftool/dbconfig/20260604-013833-fceratto.json * 01:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P93812 and previous config saved to /var/cache/conftool/dbconfig/20260604-012826-fceratto.json * 01:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93811 and previous config saved to /var/cache/conftool/dbconfig/20260604-011818-fceratto.json * 01:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1260 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93810 and previous config saved to /var/cache/conftool/dbconfig/20260604-011005-fceratto.json * 01:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1260.eqiad.wmnet with reason: Maintenance * 01:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93809 and previous config saved to /var/cache/conftool/dbconfig/20260604-010937-fceratto.json * 00:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P93808 and previous config saved to /var/cache/conftool/dbconfig/20260604-005929-fceratto.json * 00:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P93807 and previous config saved to /var/cache/conftool/dbconfig/20260604-004922-fceratto.json * 00:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93806 and previous config saved to /var/cache/conftool/dbconfig/20260604-003914-fceratto.json * 00:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1252 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93805 and previous config saved to /var/cache/conftool/dbconfig/20260604-002851-fceratto.json * 00:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1252.eqiad.wmnet with reason: Maintenance * 00:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93804 and previous config saved to /var/cache/conftool/dbconfig/20260604-002821-fceratto.json * 00:26 otto@deploy1003: otto: Backport for [[gerrit:1297260{{!}}EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 00:24 otto@deploy1003: Started scap sync-world: Backport for [[gerrit:1297260{{!}}EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] * 00:18 Amir1: mwscript-k8s --follow --dblist=all -- extensions/timeline/maintenance/DeleteOldTimelineFiles.php --date {{Gerrit|20210101000000}} * 00:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P93803 and previous config saved to /var/cache/conftool/dbconfig/20260604-001813-fceratto.json * 00:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P93802 and previous config saved to /var/cache/conftool/dbconfig/20260604-000805-fceratto.json == 2026-06-03 == * 23:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93801 and previous config saved to /var/cache/conftool/dbconfig/20260603-235758-fceratto.json * 23:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93800 and previous config saved to /var/cache/conftool/dbconfig/20260603-234935-fceratto.json * 23:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1248.eqiad.wmnet with reason: Maintenance * 23:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93799 and previous config saved to /var/cache/conftool/dbconfig/20260603-234907-fceratto.json * 23:42 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296561{{!}}Add a maintenance script to delete old files]], [[gerrit:1296560{{!}}Add a maintenance script to delete old files]] (duration: 07m 09s) * 23:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P93798 and previous config saved to /var/cache/conftool/dbconfig/20260603-233859-fceratto.json * 23:37 ladsgroup@deploy1003: ladsgroup, reedy: Continuing with deployment * 23:36 ladsgroup@deploy1003: ladsgroup, reedy: Backport for [[gerrit:1296561{{!}}Add a maintenance script to delete old files]], [[gerrit:1296560{{!}}Add a maintenance script to delete old files]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:34 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1296561{{!}}Add a maintenance script to delete old files]], [[gerrit:1296560{{!}}Add a maintenance script to delete old files]] * 23:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P93797 and previous config saved to /var/cache/conftool/dbconfig/20260603-232852-fceratto.json * 23:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 23:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 23:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93796 and previous config saved to /var/cache/conftool/dbconfig/20260603-231844-fceratto.json * 23:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93795 and previous config saved to /var/cache/conftool/dbconfig/20260603-231031-fceratto.json * 23:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1247.eqiad.wmnet with reason: Maintenance * 23:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93794 and previous config saved to /var/cache/conftool/dbconfig/20260603-231001-fceratto.json * 22:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P93793 and previous config saved to /var/cache/conftool/dbconfig/20260603-225953-fceratto.json * 22:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P93792 and previous config saved to /var/cache/conftool/dbconfig/20260603-224945-fceratto.json * 22:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93791 and previous config saved to /var/cache/conftool/dbconfig/20260603-223937-fceratto.json * 22:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1244 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93790 and previous config saved to /var/cache/conftool/dbconfig/20260603-223116-fceratto.json * 22:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1244.eqiad.wmnet with reason: Maintenance * 22:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93789 and previous config saved to /var/cache/conftool/dbconfig/20260603-223048-fceratto.json * 22:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P93788 and previous config saved to /var/cache/conftool/dbconfig/20260603-222041-fceratto.json * 22:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P93787 and previous config saved to /var/cache/conftool/dbconfig/20260603-221034-fceratto.json * 22:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93786 and previous config saved to /var/cache/conftool/dbconfig/20260603-220026-fceratto.json * 21:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1243 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93785 and previous config saved to /var/cache/conftool/dbconfig/20260603-215110-fceratto.json * 21:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1243.eqiad.wmnet with reason: Maintenance * 21:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93784 and previous config saved to /var/cache/conftool/dbconfig/20260603-215053-fceratto.json * 21:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P93783 and previous config saved to /var/cache/conftool/dbconfig/20260603-214046-fceratto.json * 21:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P93782 and previous config saved to /var/cache/conftool/dbconfig/20260603-213038-fceratto.json * 21:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93781 and previous config saved to /var/cache/conftool/dbconfig/20260603-212030-fceratto.json * 21:12 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1242 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93779 and previous config saved to /var/cache/conftool/dbconfig/20260603-211206-fceratto.json * 21:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1242.eqiad.wmnet with reason: Maintenance * 21:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93778 and previous config saved to /var/cache/conftool/dbconfig/20260603-211138-fceratto.json * 21:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P93774 and previous config saved to /var/cache/conftool/dbconfig/20260603-210130-fceratto.json * 20:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P93773 and previous config saved to /var/cache/conftool/dbconfig/20260603-205122-fceratto.json * 20:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93772 and previous config saved to /var/cache/conftool/dbconfig/20260603-204115-fceratto.json * 20:33 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297228{{!}}Attribution research don't use testKitchen compatibility layer (T417050)]] (duration: 06m 41s) * 20:32 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1241 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93771 and previous config saved to /var/cache/conftool/dbconfig/20260603-203254-fceratto.json * 20:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1241.eqiad.wmnet with reason: Maintenance * 20:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93770 and previous config saved to /var/cache/conftool/dbconfig/20260603-203227-fceratto.json * 20:29 cjming@deploy1003: cjming: Continuing with deployment * 20:29 cjming@deploy1003: cjming: Backport for [[gerrit:1297228{{!}}Attribution research don't use testKitchen compatibility layer (T417050)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:26 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1297228{{!}}Attribution research don't use testKitchen compatibility layer (T417050)]] * 20:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P93769 and previous config saved to /var/cache/conftool/dbconfig/20260603-202219-fceratto.json * 20:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P93766 and previous config saved to /var/cache/conftool/dbconfig/20260603-201211-fceratto.json * 20:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93765 and previous config saved to /var/cache/conftool/dbconfig/20260603-200203-fceratto.json * 19:59 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 19:59 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 19:59 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 19:59 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 19:53 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93764 and previous config saved to /var/cache/conftool/dbconfig/20260603-195341-fceratto.json * 19:53 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1238.eqiad.wmnet with reason: Maintenance * 19:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93763 and previous config saved to /var/cache/conftool/dbconfig/20260603-195313-fceratto.json * 19:47 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5032.* * 19:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P93762 and previous config saved to /var/cache/conftool/dbconfig/20260603-194306-fceratto.json * 19:39 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5032.* * 19:37 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5032.* * 19:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P93761 and previous config saved to /var/cache/conftool/dbconfig/20260603-193258-fceratto.json * 19:26 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 19:25 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 19:25 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 19:25 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 19:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93760 and previous config saved to /var/cache/conftool/dbconfig/20260603-192250-fceratto.json * 19:22 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 19:22 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 19:14 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93759 and previous config saved to /var/cache/conftool/dbconfig/20260603-191437-fceratto.json * 19:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1024-1025].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 19:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1221.eqiad.wmnet with reason: Maintenance * 19:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93758 and previous config saved to /var/cache/conftool/dbconfig/20260603-191348-fceratto.json * 19:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P93757 and previous config saved to /var/cache/conftool/dbconfig/20260603-190340-fceratto.json * 18:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P93756 and previous config saved to /var/cache/conftool/dbconfig/20260603-185331-fceratto.json * 18:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93755 and previous config saved to /var/cache/conftool/dbconfig/20260603-184324-fceratto.json * 18:34 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1199 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93754 and previous config saved to /var/cache/conftool/dbconfig/20260603-183455-fceratto.json * 18:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance * 18:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93753 and previous config saved to /var/cache/conftool/dbconfig/20260603-183427-fceratto.json * 18:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P93752 and previous config saved to /var/cache/conftool/dbconfig/20260603-182420-fceratto.json * 18:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P93751 and previous config saved to /var/cache/conftool/dbconfig/20260603-181412-fceratto.json * 18:10 dancy@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.5 refs [[phab:T423914|T423914]] * 18:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93750 and previous config saved to /var/cache/conftool/dbconfig/20260603-180404-fceratto.json * 17:57 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5032.* * 17:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93749 and previous config saved to /var/cache/conftool/dbconfig/20260603-175544-fceratto.json * 17:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance * 17:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93748 and previous config saved to /var/cache/conftool/dbconfig/20260603-175342-fceratto.json * 17:52 hashar: contint1003: sudo puppet agent --disable "Prevent Jenkins from coming back" * 17:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P93747 and previous config saved to /var/cache/conftool/dbconfig/20260603-174334-fceratto.json * 17:38 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:37 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 17:37 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:36 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:36 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:35 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:35 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:35 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:34 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:34 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:33 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P93746 and previous config saved to /var/cache/conftool/dbconfig/20260603-173327-fceratto.json * 17:33 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:32 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:29 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5032.* * 17:26 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host sretest2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 17:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93745 and previous config saved to /var/cache/conftool/dbconfig/20260603-172319-fceratto.json * 17:18 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: Stopping before sync operations * 17:17 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: Started scap sync-world: No-deploy scap run to verify scap config change * 17:17 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:15 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:15 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1253 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93744 and previous config saved to /var/cache/conftool/dbconfig/20260603-171521-fceratto.json * 17:15 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:15 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance * 17:14 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93743 and previous config saved to /var/cache/conftool/dbconfig/20260603-171452-fceratto.json * 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:13 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:12 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:10 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:09 ayounsi@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2012.wikimedia.org with OS trixie * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P93742 and previous config saved to /var/cache/conftool/dbconfig/20260603-170444-fceratto.json * 17:04 swfrench@deploy1003: Stopping before sync operations * 17:03 swfrench@deploy1003: Started scap sync-world: No-deploy scap run to verify clean state before config change * 16:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P93741 and previous config saved to /var/cache/conftool/dbconfig/20260603-165436-fceratto.json * 16:53 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:53 hashar: Restarting CI Jenkins one last time # [[phab:T418521|T418521]] * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:44 btullis@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295922{{!}}Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] (duration: 07m 16s) * 16:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93740 and previous config saved to /var/cache/conftool/dbconfig/20260603-164428-fceratto.json * 16:43 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:43 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:42 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:41 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:40 btullis@deploy1003: btullis: Continuing with deployment * 16:39 btullis@deploy1003: btullis: Backport for [[gerrit:1295922{{!}}Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93739 and previous config saved to /var/cache/conftool/dbconfig/20260603-163726-fceratto.json * 16:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance * 16:37 btullis@deploy1003: Started scap sync-world: Backport for [[gerrit:1295922{{!}}Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] * 16:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93738 and previous config saved to /var/cache/conftool/dbconfig/20260603-163658-fceratto.json * 16:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P93737 and previous config saved to /var/cache/conftool/dbconfig/20260603-162650-fceratto.json * 16:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P93736 and previous config saved to /var/cache/conftool/dbconfig/20260603-161643-fceratto.json * 16:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93735 and previous config saved to /var/cache/conftool/dbconfig/20260603-160635-fceratto.json * 16:04 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:59 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93734 and previous config saved to /var/cache/conftool/dbconfig/20260603-155928-fceratto.json * 15:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance * 15:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93733 and previous config saved to /var/cache/conftool/dbconfig/20260603-155859-fceratto.json * 15:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 15:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P93732 and previous config saved to /var/cache/conftool/dbconfig/20260603-154852-fceratto.json * 15:46 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:46 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2012.wikimedia.org with OS trixie * 15:40 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:40 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 15:40 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 15:40 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 15:39 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 15:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P93731 and previous config saved to /var/cache/conftool/dbconfig/20260603-153844-fceratto.json * 15:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93729 and previous config saved to /var/cache/conftool/dbconfig/20260603-152836-fceratto.json * 15:25 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host sretest2012 * 15:25 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2012 * 15:25 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host sretest2012 * 15:25 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2012 * 15:24 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:23 mutante: disabling jenkins on CI servers for maintenance * 15:23 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host sretest2012 * 15:23 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2012 * 15:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 15:21 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1202 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93728 and previous config saved to /var/cache/conftool/dbconfig/20260603-152129-fceratto.json * 15:21 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance * 15:21 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:21 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding sretest2012 to codfw - jhancock@cumin2002" * 15:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 15:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93727 and previous config saved to /var/cache/conftool/dbconfig/20260603-152102-fceratto.json * 15:20 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding sretest2012 to codfw - jhancock@cumin2002" * 15:18 brouberol@dns1004: END - running authdns-update * 15:18 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:16 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 15:16 brouberol@dns1004: START - running authdns-update * 15:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P93726 and previous config saved to /var/cache/conftool/dbconfig/20260603-151055-fceratto.json * 15:01 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P93725 and previous config saved to /var/cache/conftool/dbconfig/20260603-150047-fceratto.json * 14:57 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 14:52 cmooney@cumin1003: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox * 14:51 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93723 and previous config saved to /var/cache/conftool/dbconfig/20260603-145039-fceratto.json * 14:48 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297137{{!}}Revert "MultimediaViewer: enable image carousel as a beta feature on Wikipedias"]] (duration: 06m 46s) * 14:47 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 14:46 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:46 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:43 mlitn@deploy1003: mlitn: Continuing with deployment * 14:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93722 and previous config saved to /var/cache/conftool/dbconfig/20260603-144334-fceratto.json * 14:43 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:43 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance * 14:43 mlitn@deploy1003: mlitn: Backport for [[gerrit:1297137{{!}}Revert "MultimediaViewer: enable image carousel as a beta feature on Wikipedias"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93721 and previous config saved to /var/cache/conftool/dbconfig/20260603-144306-fceratto.json * 14:41 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:41 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:41 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1297137{{!}}Revert "MultimediaViewer: enable image carousel as a beta feature on Wikipedias"]] * 14:39 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:39 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:39 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:39 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:38 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:35 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 14:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 14:34 sgimeno@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297130{{!}}editor: make redesigned anon warning the default experience (T424595)]] (duration: 10m 45s) * 14:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P93719 and previous config saved to /var/cache/conftool/dbconfig/20260603-143259-fceratto.json * 14:30 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:28 sgimeno@deploy1003: sgimeno: Continuing with deployment * 14:25 sgimeno@deploy1003: sgimeno: Backport for [[gerrit:1297130{{!}}editor: make redesigned anon warning the default experience (T424595)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:24 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:24 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:23 sgimeno@deploy1003: Started scap sync-world: Backport for [[gerrit:1297130{{!}}editor: make redesigned anon warning the default experience (T424595)]] * 14:23 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P93717 and previous config saved to /var/cache/conftool/dbconfig/20260603-142251-fceratto.json * 14:22 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:22 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:21 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:21 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:21 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:20 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:20 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:20 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:20 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:19 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:19 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:16 vriley@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:16 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:16 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:13 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:12 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93716 and previous config saved to /var/cache/conftool/dbconfig/20260603-141242-fceratto.json * 14:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:11 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:11 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:10 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mc2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:10 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host mc2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:10 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:09 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:08 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:07 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:05 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296631{{!}}translate: adding separate read/write endpoints (T425377)]] (duration: 13m 06s) * 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1191 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93715 and previous config saved to /var/cache/conftool/dbconfig/20260603-140537-fceratto.json * 14:05 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance * 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93714 and previous config saved to /var/cache/conftool/dbconfig/20260603-140507-fceratto.json * 14:01 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:58 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:58 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:58 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 13:56 dcausse@deploy1003: atsuko, dcausse: Rolling back deployment * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T426633|T426633]])', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20260603-133440-fceratto.json * 13:29 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:29 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2186: Migration of db2186.codfw.wmnet completed * 13:28 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295910{{!}}hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]] (duration: 07m 36s) * 13:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1174 ([[phab:T426633|T426633]])', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20260603-132638-fceratto.json * 13:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance * 13:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93710 and previous config saved to /var/cache/conftool/dbconfig/20260603-132605-fceratto.json * 13:25 sukhe: sudo cumin 'A:lvs or A:liberica' 'disable-puppet "merging CR 1282764"' * 13:23 kharlan@deploy1003: kharlan: Continuing with deployment * 13:22 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295910{{!}}hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:20 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295910{{!}}hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]] * 13:18 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296649{{!}}hCaptcha: Roll out to all except enwiki for mobile apps. (T426048)]] (duration: 07m 46s) * 13:16 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20260603-131556-fceratto.json * 13:15 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:13 kharlan@deploy1003: dbrant, kharlan: Continuing with deployment * 13:12 kharlan@deploy1003: dbrant, kharlan: Backport for [[gerrit:1296649{{!}}hCaptcha: Roll out to all except enwiki for mobile apps. (T426048)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296649{{!}}hCaptcha: Roll out to all except enwiki for mobile apps. (T426048)]] * 13:09 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:09 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add codfw d3 and e5 public vlans - ayounsi@cumin1003" * 13:09 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add codfw d3 and e5 public vlans - ayounsi@cumin1003" * 13:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P93708 and previous config saved to /var/cache/conftool/dbconfig/20260603-130548-fceratto.json * 13:05 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 12:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93706 and previous config saved to /var/cache/conftool/dbconfig/20260603-125540-fceratto.json * 12:51 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297110{{!}}ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] (duration: 07m 44s) * 12:49 jgreen@dns1004: END - running authdns-update * 12:47 jgreen@dns1004: START - running authdns-update * 12:46 jiji@deploy1003: jiji: Continuing with deployment * 12:46 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93705 and previous config saved to /var/cache/conftool/dbconfig/20260603-124624-fceratto.json * 12:46 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance * 12:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93704 and previous config saved to /var/cache/conftool/dbconfig/20260603-124556-fceratto.json * 12:45 jiji@deploy1003: jiji: Backport for [[gerrit:1297110{{!}}ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:43 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2186: Migration of db2186.codfw.wmnet completed * 12:43 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1297110{{!}}ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] * 12:41 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1067.eqiad.wmnet with OS bullseye * 12:38 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1292364{{!}}Update hCaptcha checks to retrieve API parameters from $_REQUEST (T427105)]] (duration: 11m 15s) * 12:36 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2186.codfw.wmnet with OS trixie * 12:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P93702 and previous config saved to /var/cache/conftool/dbconfig/20260603-123548-fceratto.json * 12:34 dreamyjazz@deploy1003: somerandomdeveloper, dreamyjazz: Continuing with deployment * 12:31 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1066.eqiad.wmnet with OS bullseye * 12:29 dreamyjazz@deploy1003: somerandomdeveloper, dreamyjazz: Backport for [[gerrit:1292364{{!}}Update hCaptcha checks to retrieve API parameters from $_REQUEST (T427105)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:27 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1292364{{!}}Update hCaptcha checks to retrieve API parameters from $_REQUEST (T427105)]] * 12:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P93701 and previous config saved to /var/cache/conftool/dbconfig/20260603-122541-fceratto.json * 12:22 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1067.eqiad.wmnet with reason: host reimage * 12:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2186.codfw.wmnet with reason: host reimage * 12:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93700 and previous config saved to /var/cache/conftool/dbconfig/20260603-121533-fceratto.json * 12:13 mvernon@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ms-be1066.eqiad.wmnet with reason: host reimage * 12:13 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2186.codfw.wmnet with reason: host reimage * 12:11 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1067.eqiad.wmnet with reason: host reimage * 12:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93699 and previous config saved to /var/cache/conftool/dbconfig/20260603-120732-fceratto.json * 12:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 12:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance * 12:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93698 and previous config saved to /var/cache/conftool/dbconfig/20260603-120634-fceratto.json * 12:03 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1066.eqiad.wmnet with reason: host reimage * 11:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P93697 and previous config saved to /var/cache/conftool/dbconfig/20260603-115626-fceratto.json * 11:54 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2186.codfw.wmnet with OS trixie * 11:54 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be1067 * 11:54 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1067 * 11:52 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1067 * 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be1067.eqiad.wmnet 96.48.64.10.in-addr.arpa 6.9.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:52 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be1067.eqiad.wmnet 96.48.64.10.in-addr.arpa 6.9.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1067 - mvernon@cumin2002" * 11:52 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1067 - mvernon@cumin2002" * 11:48 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2186: Upgrading db2186.codfw.wmnet * 11:48 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2186: Upgrading db2186.codfw.wmnet * 11:48 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:47 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:46 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be1067 * 11:46 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be1067.eqiad.wmnet with OS bullseye * 11:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P93695 and previous config saved to /var/cache/conftool/dbconfig/20260603-114618-fceratto.json * 11:46 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be1066 * 11:46 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1066 * 11:45 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1066 * 11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be1066.eqiad.wmnet 117.32.64.10.in-addr.arpa 7.1.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:45 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be1066.eqiad.wmnet 117.32.64.10.in-addr.arpa 7.1.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1066 - mvernon@cumin2002" * 11:45 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1066 - mvernon@cumin2002" * 11:43 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply * 11:41 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:40 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be1066 * 11:40 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be1066.eqiad.wmnet with OS bullseye * 11:39 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be1067 * 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93693 and previous config saved to /var/cache/conftool/dbconfig/20260603-113611-fceratto.json * 11:33 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:33 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2196: Migration of db2196.codfw.wmnet completed * 11:29 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93691 and previous config saved to /var/cache/conftool/dbconfig/20260603-112909-fceratto.json * 11:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance * 11:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance * 11:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93690 and previous config saved to /var/cache/conftool/dbconfig/20260603-112838-fceratto.json * 11:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:20 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:20 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:20 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P93689 and previous config saved to /var/cache/conftool/dbconfig/20260603-111831-fceratto.json * 11:14 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:09 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply * 11:09 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply * 11:08 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P93687 and previous config saved to /var/cache/conftool/dbconfig/20260603-110823-fceratto.json * 11:07 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be1066 * 11:07 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply * 11:06 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply * 11:05 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply * 11:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:01 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:01 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:00 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289895{{!}}Update UserInfoCard to be enabled by default for certain user groups (T426021)]] (duration: 07m 37s) * 11:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:59 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply * 10:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:59 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply * 10:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:58 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply * 10:58 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93685 and previous config saved to /var/cache/conftool/dbconfig/20260603-105815-fceratto.json * 10:58 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:56 mszwarc@deploy1003: mszwarc: Continuing with deployment * 10:55 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1289895{{!}}Update UserInfoCard to be enabled by default for certain user groups (T426021)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:54 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply * 10:54 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply * 10:53 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply * 10:53 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1289895{{!}}Update UserInfoCard to be enabled by default for certain user groups (T426021)]] * 10:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:50 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1198 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93684 and previous config saved to /var/cache/conftool/dbconfig/20260603-105006-fceratto.json * 10:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance * 10:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93683 and previous config saved to /var/cache/conftool/dbconfig/20260603-104939-fceratto.json * 10:45 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:45 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:44 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2196: Migration of db2196.codfw.wmnet completed * 10:44 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:41 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 10:40 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 10:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P93681 and previous config saved to /var/cache/conftool/dbconfig/20260603-103931-fceratto.json * 10:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1053: repool after upgrade * 10:37 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2196.codfw.wmnet with OS trixie * 10:36 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297090{{!}}hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] (duration: 12m 03s) * 10:32 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P93679 and previous config saved to /var/cache/conftool/dbconfig/20260603-102924-fceratto.json * 10:26 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297090{{!}}hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:24 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297090{{!}}hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] * 10:22 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be1067 * 10:21 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be1066 * 10:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2196.codfw.wmnet with reason: host reimage * 10:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93677 and previous config saved to /var/cache/conftool/dbconfig/20260603-101916-fceratto.json * 10:15 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2013.codfw.wmnet * 10:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2196.codfw.wmnet with reason: host reimage * 10:11 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93676 and previous config saved to /var/cache/conftool/dbconfig/20260603-101105-fceratto.json * 10:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1189.eqiad.wmnet with reason: Maintenance * 10:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93675 and previous config saved to /var/cache/conftool/dbconfig/20260603-101037-fceratto.json * 10:10 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2013.codfw.wmnet * 10:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P93673 and previous config saved to /var/cache/conftool/dbconfig/20260603-100029-fceratto.json * 09:59 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2196.codfw.wmnet with OS trixie * 09:57 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2196: Upgrading db2196.codfw.wmnet * 09:57 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2196: Upgrading db2196.codfw.wmnet * 09:57 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:52 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1053: repool after upgrade * 09:52 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:52 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:52 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:52 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:51 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:51 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:51 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P93670 and previous config saved to /var/cache/conftool/dbconfig/20260603-095022-fceratto.json * 09:49 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:49 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:48 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1053.eqiad.wmnet with OS trixie * 09:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:43 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2013.codfw.wmnet * 09:41 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on es1053.eqiad.wmnet with reason: host reimage * 09:41 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1053.eqiad.wmnet with reason: host reimage * 09:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93669 and previous config saved to /var/cache/conftool/dbconfig/20260603-094014-fceratto.json * 09:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2215: Migration of db2215.codfw.wmnet completed * 09:38 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2013.codfw.wmnet * 09:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93667 and previous config saved to /var/cache/conftool/dbconfig/20260603-093146-fceratto.json * 09:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance * 09:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93666 and previous config saved to /var/cache/conftool/dbconfig/20260603-093119-fceratto.json * 09:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1211: Migration of db1211.eqiad.wmnet completed * 09:27 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297069{{!}}hCaptcha: Collect risk score for blocked account creations (T427784)]] (duration: 07m 26s) * 09:25 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1053.eqiad.wmnet with OS trixie * 09:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add public1-b3-codfw gateway IPs - ayounsi@cumin1003" * 09:24 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add public1-b3-codfw gateway IPs - ayounsi@cumin1003" * 09:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1053: Upgrading es1053.eqiad.wmnet * 09:23 kharlan@deploy1003: kharlan: Continuing with deployment * 09:22 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1053: Upgrading es1053.eqiad.wmnet * 09:22 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:21 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297069{{!}}hCaptcha: Collect risk score for blocked account creations (T427784)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:21 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply * 09:21 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2054: repool after upgrade * 09:21 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply * 09:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P93661 and previous config saved to /var/cache/conftool/dbconfig/20260603-092111-fceratto.json * 09:20 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 09:20 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297069{{!}}hCaptcha: Collect risk score for blocked account creations (T427784)]] * 09:14 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297065{{!}}Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] (duration: 07m 06s) * 09:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P93659 and previous config saved to /var/cache/conftool/dbconfig/20260603-091104-fceratto.json * 09:10 kharlan@deploy1003: kharlan: Continuing with deployment * 09:09 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297065{{!}}Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:07 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297065{{!}}Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] * 09:06 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 09:06 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297064{{!}}Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 10m 54s) * 09:05 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 09:04 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 09:01 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003 - [[phab:T422043|T422043]]" * 09:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93656 and previous config saved to /var/cache/conftool/dbconfig/20260603-090056-fceratto.json * 09:00 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003 - [[phab:T422043|T422043]]" * 09:00 ayounsi@cumin1003: END (ERROR) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=97) generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003" * 09:00 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003" * 08:59 kharlan@deploy1003: kharlan: Continuing with deployment * 08:59 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297064{{!}}Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:55 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297064{{!}}Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] * 08:53 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296635{{!}}Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 11m 43s) * 08:52 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2215: Migration of db2215.codfw.wmnet completed * 08:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet * 08:52 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet * 08:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb[1022-1023].eqiad.wmnet * 08:51 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb[1022-1023].eqiad.wmnet * 08:50 kharlan@deploy1003: kharlan: Rolling back deployment * 08:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93652 and previous config saved to /var/cache/conftool/dbconfig/20260603-084846-fceratto.json * 08:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance * 08:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93651 and previous config saved to /var/cache/conftool/dbconfig/20260603-084819-fceratto.json * 08:47 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296635{{!}}Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2215.codfw.wmnet with OS trixie * 08:45 jiji@cumin1003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check docker-registry: maintenance * 08:45 jiji@cumin1003: START - Cookbook sre.discovery.service-route check docker-registry: maintenance * 08:43 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1211: Migration of db1211.eqiad.wmnet completed * 08:41 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296635{{!}}Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] * 08:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1211.eqiad.wmnet with OS trixie * 08:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93649 and previous config saved to /var/cache/conftool/dbconfig/20260603-083811-fceratto.json * 08:37 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296632{{!}}Image Browsing: add accessible labels to carousel elements (T407793)]] (duration: 32m 11s) * 08:36 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2054: repool after upgrade * 08:35 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool es2054.codfw.wmnet: After reimage * 08:35 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2054.codfw.wmnet: After reimage * 08:35 jiji@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:34 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 08:34 jiji@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:33 jiji@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:33 jiji@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:31 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:31 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:31 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2054.codfw.wmnet with OS trixie * 08:30 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:29 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 08:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2215.codfw.wmnet with reason: host reimage * 08:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93647 and previous config saved to /var/cache/conftool/dbconfig/20260603-082804-fceratto.json * 08:25 mszwarc@deploy1003: mlitn, mszwarc: Continuing with deployment * 08:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1211.eqiad.wmnet with reason: host reimage * 08:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1049: repool after upgrade * 08:22 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2215.codfw.wmnet with reason: host reimage * 08:22 mszwarc@deploy1003: mlitn, mszwarc: Backport for [[gerrit:1296632{{!}}Image Browsing: add accessible labels to carousel elements (T407793)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:18 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1211.eqiad.wmnet with reason: host reimage * 08:18 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 08:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93645 and previous config saved to /var/cache/conftool/dbconfig/20260603-081756-fceratto.json * 08:17 jiji@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 08:17 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 08:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 08:14 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2054.codfw.wmnet with reason: host reimage * 08:08 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2054.codfw.wmnet with reason: host reimage * 08:05 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296632{{!}}Image Browsing: add accessible labels to carousel elements (T407793)]] * {{safesubst:SAL entry|1=08:04 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296580{{!}}Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703{{!}}jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713{{!}}conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627{{!}}Add missing lazy img to carousel (T427821)]], [[gerrit:1295968{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T426799)]}} * 08:03 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93643 and previous config saved to /var/cache/conftool/dbconfig/20260603-080346-fceratto.json * 08:03 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1211.eqiad.wmnet with OS trixie * 08:03 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance * 08:03 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2215.codfw.wmnet with OS trixie * 08:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1211: Upgrading db1211.eqiad.wmnet * 08:02 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2215: Upgrading db2215.codfw.wmnet * 08:01 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:01 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1211: Upgrading db1211.eqiad.wmnet * 08:01 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2215: Upgrading db2215.codfw.wmnet * 08:01 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:01 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:01 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1157: Repooling * 08:01 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1157: Repooling * 08:00 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 07:57 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1022-1023].eqiad.wmnet with reason: Reimaging upstream server * 07:57 mszwarc@deploy1003: anzx, mlitn, mfossati, mszwarc: Continuing with deployment * 07:56 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Reimaging upstream server * {{safesubst:SAL entry|1=07:54 mszwarc@deploy1003: anzx, mlitn, mfossati, mszwarc: Backport for [[gerrit:1296580{{!}}Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703{{!}}jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713{{!}}conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627{{!}}Add missing lazy img to carousel (T427821)]], [[gerrit:1295968{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T42}} * 07:52 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2231: repool after maintenance * 07:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2054.codfw.wmnet with OS trixie * 07:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2054: Upgrading es2054.codfw.wmnet * 07:50 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2054: Upgrading es2054.codfw.wmnet * 07:50 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:50 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296580{{!}}Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703{{!}}jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713{{!}}conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627{{!}}Add missing lazy img to carousel (T427821)]], [[gerrit:1295968{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T426799)]] * 07:48 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296516{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] (duration: 32m 13s) * 07:44 marostegui@dns1004: END - running authdns-update * 07:43 marostegui@dns1004: START - running authdns-update * 07:42 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1056 to es2 eqiad primary [[phab:T427875|T427875]]', diff saved to https://phabricator.wikimedia.org/P93637 and previous config saved to /var/cache/conftool/dbconfig/20260603-074250-marostegui.json * 07:37 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1049: repool after upgrade * 07:37 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:35 mszwarc@deploy1003: mszwarc, stran: Continuing with deployment * 07:35 mszwarc@deploy1003: mszwarc, stran: Backport for [[gerrit:1296516{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:32 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1049.eqiad.wmnet with OS trixie * 07:16 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296516{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] * 07:14 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1049.eqiad.wmnet with reason: host reimage * 07:07 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1049.eqiad.wmnet with reason: host reimage * 07:07 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2231: repool after maintenance * 07:04 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 06:57 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2231.codfw.wmnet with OS trixie * 06:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1049.eqiad.wmnet with OS trixie * 06:46 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1049: Upgrading es1049.eqiad.wmnet * 06:46 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2056 to es2 codfw primary [[phab:T427875|T427875]]', diff saved to https://phabricator.wikimedia.org/P93632 and previous config saved to /var/cache/conftool/dbconfig/20260603-064623-marostegui.json * 06:45 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1049: Upgrading es1049.eqiad.wmnet * 06:45 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:44 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1056: repool after upgrade * 06:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2231.codfw.wmnet with reason: host reimage * 06:36 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2231.codfw.wmnet with reason: host reimage * 06:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2231.codfw.wmnet with OS trixie * 06:09 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2231: Upgrading db2231.codfw.wmnet * 06:09 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2231: Upgrading db2231.codfw.wmnet * 06:09 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:59 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1056: repool after upgrade * 05:59 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 05:55 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1056.eqiad.wmnet with OS trixie * 05:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1056.eqiad.wmnet with reason: host reimage * 05:33 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1056.eqiad.wmnet with reason: host reimage * 05:18 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1056.eqiad.wmnet with OS trixie * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1056: Upgrading es1056.eqiad.wmnet * 05:17 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1056: Upgrading es1056.eqiad.wmnet * 05:16 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade == 2026-06-02 == * 22:21 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296689{{!}}hCaptcha: Correct inaccurate comment]] (duration: 06m 27s) * 22:18 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 22:18 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 22:17 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 22:17 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296689{{!}}hCaptcha: Correct inaccurate comment]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:15 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296689{{!}}hCaptcha: Correct inaccurate comment]] * 22:13 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296551{{!}}hCaptcha: Enable for badlogin on group0 wikis (T426875)]] (duration: 08m 31s) * 22:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 22:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 22:09 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 22:07 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296551{{!}}hCaptcha: Enable for badlogin on group0 wikis (T426875)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:05 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296551{{!}}hCaptcha: Enable for badlogin on group0 wikis (T426875)]] * 20:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93621 and previous config saved to /var/cache/conftool/dbconfig/20260602-203945-fceratto.json * 20:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93620 and previous config saved to /var/cache/conftool/dbconfig/20260602-202937-fceratto.json * 20:27 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1054.eqiad.wmnet * 20:27 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:27 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1054.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:26 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1054.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:20 jiji@cumin1003: START - Cookbook sre.dns.netbox * 20:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93619 and previous config saved to /var/cache/conftool/dbconfig/20260602-201929-fceratto.json * 20:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93618 and previous config saved to /var/cache/conftool/dbconfig/20260602-200922-fceratto.json * 20:03 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1054.eqiad.wmnet * 19:48 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1053.eqiad.wmnet * 19:48 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:48 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1053.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 19:37 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1053.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 19:09 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93617 and previous config saved to /var/cache/conftool/dbconfig/20260602-190907-fceratto.json * 19:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance * 19:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93616 and previous config saved to /var/cache/conftool/dbconfig/20260602-190811-fceratto.json * 19:05 dancy@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.5 refs [[phab:T423914|T423914]] * 18:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P93615 and previous config saved to /var/cache/conftool/dbconfig/20260602-185804-fceratto.json * 18:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P93614 and previous config saved to /var/cache/conftool/dbconfig/20260602-184757-fceratto.json * 18:38 jiji@cumin1003: START - Cookbook sre.dns.netbox * 18:38 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:38 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93612 and previous config saved to /var/cache/conftool/dbconfig/20260602-183749-fceratto.json * 18:37 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:37 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:33 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1053.eqiad.wmnet * 18:30 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1259 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93611 and previous config saved to /var/cache/conftool/dbconfig/20260602-183023-fceratto.json * 18:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance * 18:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93610 and previous config saved to /var/cache/conftool/dbconfig/20260602-182956-fceratto.json * 18:27 mutante: gerrit delete unused plugin projects: barricade, WikimediaBlocks and WikimediaWebSessions * 18:26 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1052.eqiad.wmnet * 18:26 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:26 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1052.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 18:25 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1052.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 18:25 dancy: Train is blocked at testwikis on https://phabricator.wikimedia.org/T427935 * 18:21 Daimona: Running query from [[phab:T427962|T427962]]#11978299 in x1.wikishared * 18:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P93609 and previous config saved to /var/cache/conftool/dbconfig/20260602-181949-fceratto.json * 18:16 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296615{{!}}feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614{{!}}feat(cleanMentorList): Add a feature flag (T427386)]] (duration: 34m 09s) * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 18:12 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 18:12 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 18:12 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 18:10 jiji@cumin1003: START - Cookbook sre.dns.netbox * 18:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P93608 and previous config saved to /var/cache/conftool/dbconfig/20260602-180941-fceratto.json * 18:08 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 18:07 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 18:06 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 18:06 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 18:05 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:05 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:05 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 18:05 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 18:04 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 18:02 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 18:02 swfrench-wmf: reverting shellbox to 2026-05-20-192555 due to errors in shellbox-syntaxhighlight * 18:02 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 18:01 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 18:01 urbanecm@deploy1003: urbanecm: Continuing with deployment * 18:01 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1296615{{!}}feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614{{!}}feat(cleanMentorList): Add a feature flag (T427386)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:00 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1052.eqiad.wmnet * 17:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93607 and previous config saved to /var/cache/conftool/dbconfig/20260602-175933-fceratto.json * 17:58 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:57 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:56 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1051.eqiad.wmnet * 17:56 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:56 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1051.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:55 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1051.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:53 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93605 and previous config saved to /var/cache/conftool/dbconfig/20260602-175227-fceratto.json * 17:52 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance * 17:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93604 and previous config saved to /var/cache/conftool/dbconfig/20260602-175157-fceratto.json * 17:51 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:51 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:50 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:50 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:50 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:49 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:49 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:48 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:48 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:47 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:44 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:42 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:42 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:42 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P93603 and previous config saved to /var/cache/conftool/dbconfig/20260602-174150-fceratto.json * 17:41 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1296615{{!}}feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614{{!}}feat(cleanMentorList): Add a feature flag (T427386)]] * 17:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P93602 and previous config saved to /var/cache/conftool/dbconfig/20260602-173143-fceratto.json * 17:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93601 and previous config saved to /var/cache/conftool/dbconfig/20260602-172135-fceratto.json * 17:14 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93600 and previous config saved to /var/cache/conftool/dbconfig/20260602-171422-fceratto.json * 17:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance * 17:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93599 and previous config saved to /var/cache/conftool/dbconfig/20260602-171354-fceratto.json * 17:04 jiji@cumin1003: START - Cookbook sre.dns.netbox * 17:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P93598 and previous config saved to /var/cache/conftool/dbconfig/20260602-170344-fceratto.json * 16:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P93597 and previous config saved to /var/cache/conftool/dbconfig/20260602-165336-fceratto.json * 16:49 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1051.eqiad.wmnet * 16:48 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1050.eqiad.wmnet * 16:48 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:48 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1050.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:47 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1050.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93596 and previous config saved to /var/cache/conftool/dbconfig/20260602-164328-fceratto.json * 16:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93595 and previous config saved to /var/cache/conftool/dbconfig/20260602-163622-fceratto.json * 16:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance * 16:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93594 and previous config saved to /var/cache/conftool/dbconfig/20260602-163550-fceratto.json * 16:34 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:34 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1072.eqiad.wmnet with OS trixie * 16:30 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:29 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:27 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2006.codfw.wmnet with OS trixie * 16:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P93593 and previous config saved to /var/cache/conftool/dbconfig/20260602-162542-fceratto.json * 16:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P93591 and previous config saved to /var/cache/conftool/dbconfig/20260602-161534-fceratto.json * 16:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1072.eqiad.wmnet with reason: host reimage * 16:10 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1071.eqiad.wmnet with OS trixie * 16:10 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296624{{!}}Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 06m 40s) * 16:09 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2006.codfw.wmnet with reason: host reimage * 16:05 kharlan@deploy1003: kharlan: Continuing with deployment * 16:05 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1072.eqiad.wmnet with reason: host reimage * 16:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93590 and previous config saved to /var/cache/conftool/dbconfig/20260602-160527-fceratto.json * 16:05 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2006.codfw.wmnet with reason: host reimage * 16:05 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296624{{!}}Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:03 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296624{{!}}Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] * 15:59 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295909{{!}}hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] (duration: 09m 48s) * 15:59 kharlan@deploy1003: kharlan: Rolling back deployment * 15:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93589 and previous config saved to /var/cache/conftool/dbconfig/20260602-155817-fceratto.json * 15:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance * 15:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93588 and previous config saved to /var/cache/conftool/dbconfig/20260602-155749-fceratto.json * 15:54 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1071.eqiad.wmnet with reason: host reimage * 15:53 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1072.eqiad.wmnet with OS trixie * 15:51 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1070.eqiad.wmnet with OS trixie * 15:51 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295909{{!}}hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:50 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1071.eqiad.wmnet with reason: host reimage * 15:49 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295909{{!}}hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] * 15:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P93587 and previous config saved to /var/cache/conftool/dbconfig/20260602-154742-fceratto.json * 15:47 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296558{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] (duration: 07m 24s) * 15:43 kharlan@deploy1003: kharlan: Continuing with deployment * 15:42 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296558{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:40 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296558{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] * 15:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P93586 and previous config saved to /var/cache/conftool/dbconfig/20260602-153734-fceratto.json * 15:37 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1071.eqiad.wmnet with OS trixie * 15:36 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1069.eqiad.wmnet with OS trixie * 15:35 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1070.eqiad.wmnet with reason: host reimage * 15:32 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:32 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:31 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1070.eqiad.wmnet with reason: host reimage * 15:30 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:29 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93585 and previous config saved to /var/cache/conftool/dbconfig/20260602-152726-fceratto.json * 15:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2158: Repooling * {{safesubst:SAL entry|1=15:22 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295502{{!}}Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106{{!}}Remove unused 'writeapi' right]], [[gerrit:1296566{{!}}Clean up bot password configuration]], [[gerrit:1296563{{!}}Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574{{!}}cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582{{!}}U}} * 15:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1069.eqiad.wmnet with reason: host reimage * 15:20 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93583 and previous config saved to /var/cache/conftool/dbconfig/20260602-152026-fceratto.json * 15:20 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance * 15:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93582 and previous config saved to /var/cache/conftool/dbconfig/20260602-151958-fceratto.json * 15:19 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:19 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:18 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1070.eqiad.wmnet with OS trixie * 15:18 dreamyjazz@deploy1003: matmarex, anzx, dreamyjazz: Continuing with deployment * 15:18 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 15:17 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:17 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:15 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1069.eqiad.wmnet with reason: host reimage * {{safesubst:SAL entry|1=15:15 dreamyjazz@deploy1003: matmarex, anzx, dreamyjazz: Backport for [[gerrit:1295502{{!}}Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106{{!}}Remove unused 'writeapi' right]], [[gerrit:1296566{{!}}Clean up bot password configuration]], [[gerrit:1296563{{!}}Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574{{!}}cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582}} * 15:14 jiji@cumin1003: START - Cookbook sre.dns.netbox * {{safesubst:SAL entry|1=15:13 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1295502{{!}}Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106{{!}}Remove unused 'writeapi' right]], [[gerrit:1296566{{!}}Clean up bot password configuration]], [[gerrit:1296563{{!}}Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574{{!}}cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582{{!}}Us}} * 15:12 jayme@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-main2006.codfw.wmnet with OS trixie * 15:12 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1068.eqiad.wmnet with OS trixie * 15:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P93580 and previous config saved to /var/cache/conftool/dbconfig/20260602-150951-fceratto.json * 15:09 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296514{{!}}[Growth] Set wgGEMentorshipCleanupEnabled to false on all wikis (T427386)]] (duration: 06m 22s) * 15:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1167: Repooling after Icing wait-for-green timeout * 15:06 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1050.eqiad.wmnet * 15:06 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1049.eqiad.wmnet * 15:06 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:06 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1049.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:05 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1049.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:02 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1296514{{!}}[Growth] Set wgGEMentorshipCleanupEnabled to false on all wikis (T427386)]] * 15:02 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1069.eqiad.wmnet with OS trixie * 15:01 jiji@cumin1003: START - Cookbook sre.dns.netbox * 14:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P93578 and previous config saved to /var/cache/conftool/dbconfig/20260602-145943-fceratto.json * 14:54 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage * 14:52 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:52 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:52 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1049.eqiad.wmnet * 14:51 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1067.eqiad.wmnet with OS trixie * 14:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:50 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage * 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93575 and previous config saved to /var/cache/conftool/dbconfig/20260602-144935-fceratto.json * 14:42 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for pc2021.codfw.wmnet * 14:42 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for pc2021.codfw.wmnet * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2250.codfw.wmnet * 14:41 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2250.codfw.wmnet * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2158.codfw.wmnet * 14:41 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2158.codfw.wmnet * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc2021: Repooling * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 14:41 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache * 14:41 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc2021: Repooling * 14:41 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93573 and previous config saved to /var/cache/conftool/dbconfig/20260602-144110-fceratto.json * 14:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance * 14:41 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2158: Repooling * 14:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93571 and previous config saved to /var/cache/conftool/dbconfig/20260602-144043-fceratto.json * 14:38 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:38 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:38 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:37 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1048.eqiad.wmnet * 14:37 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:37 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1048.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 14:37 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1068.eqiad.wmnet with OS trixie * 14:36 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1066.eqiad.wmnet with OS trixie * 14:34 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage * 14:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P93569 and previous config saved to /var/cache/conftool/dbconfig/20260602-143035-fceratto.json * 14:30 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage * 14:25 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1048.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 14:21 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1167: Repooling after Icing wait-for-green timeout * 14:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage * 14:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P93566 and previous config saved to /var/cache/conftool/dbconfig/20260602-142027-fceratto.json * 14:17 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1067.eqiad.wmnet with OS trixie * 14:17 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 14:17 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1167.eqiad.wmnet * 14:17 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1167.eqiad.wmnet * 14:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1065.eqiad.wmnet with OS trixie * 14:15 jayme@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2006.codfw.wmnet with OS trixie * 14:14 jiji@cumin1003: START - Cookbook sre.dns.netbox * 14:13 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage * 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93564 and previous config saved to /var/cache/conftool/dbconfig/20260602-141019-fceratto.json * 14:09 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments userOptions.php --delete --nowarn growthexperiments-homepage-variant # [[phab:T417621|T417621]] * 14:09 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1048.eqiad.wmnet * 14:08 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments userOptions.php --delete growthexperiments-homepage-variant # [[phab:T417621|T417621]] * 14:05 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 14:01 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93563 and previous config saved to /var/cache/conftool/dbconfig/20260602-140140-fceratto.json * 14:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 14:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1156.eqiad.wmnet with reason: Maintenance * 14:01 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1066.eqiad.wmnet with OS trixie * 14:00 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage * 14:00 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 14:00 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 14:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93562 and previous config saved to /var/cache/conftool/dbconfig/20260602-140022-fceratto.json * 14:00 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1064.eqiad.wmnet with OS trixie * 13:56 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage * 13:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1167.eqiad.wmnet with OS trixie * 13:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P93561 and previous config saved to /var/cache/conftool/dbconfig/20260602-135015-fceratto.json * 13:47 topranks: revert all config to normal on cr1-codfw and ssw1-a1-codfw * 13:43 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1065.eqiad.wmnet with OS trixie * 13:42 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1064.eqiad.wmnet with reason: host reimage * 13:40 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1063.eqiad.wmnet with OS trixie * 13:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P93560 and previous config saved to /var/cache/conftool/dbconfig/20260602-134007-fceratto.json * 13:38 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1167.eqiad.wmnet with reason: host reimage * 13:35 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs1002.eqiad.wmnet with OS trixie * 13:35 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs1003.eqiad.wmnet with OS trixie * 13:34 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:34 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:32 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1064.eqiad.wmnet with reason: host reimage * 13:31 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1167.eqiad.wmnet with reason: host reimage * 13:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93559 and previous config saved to /var/cache/conftool/dbconfig/20260602-132959-fceratto.json * 13:27 slyngshede@dns1004: END - running authdns-update * 13:25 slyngshede@dns1004: START - running authdns-update * 13:24 topranks: increase OSPF cost on ssw1-a1-codfw et-0/0/4 towards lsw1-a5-codfw [[phab:T427301|T427301]] * 13:23 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1063.eqiad.wmnet with reason: host reimage * 13:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93558 and previous config saved to /var/cache/conftool/dbconfig/20260602-132314-fceratto.json * 13:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1210.eqiad.wmnet with reason: Maintenance * 13:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93557 and previous config saved to /var/cache/conftool/dbconfig/20260602-132246-fceratto.json * 13:20 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1064.eqiad.wmnet with OS trixie * 13:19 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 13:19 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1062.eqiad.wmnet with OS trixie * 13:18 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1063.eqiad.wmnet with reason: host reimage * 13:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2049: repool after upgrade * 13:17 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 13:16 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1167.eqiad.wmnet with OS trixie * 13:15 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 13:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1167: Upgrading db1167.eqiad.wmnet * 13:13 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1167: Upgrading db1167.eqiad.wmnet * 13:13 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:12 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 13:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P93554 and previous config saved to /var/cache/conftool/dbconfig/20260602-131238-fceratto.json * 13:12 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 13:12 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 13:11 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 13:07 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1003.eqiad.wmnet with OS trixie * 13:07 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1002.eqiad.wmnet with OS trixie * 13:06 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1063.eqiad.wmnet with OS trixie * 13:04 jayme@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-main2006.codfw.wmnet with OS trixie * 13:04 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:04 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:03 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1022-1023].eqiad.wmnet with reason: Reimaging upstream servers * 13:03 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1001.eqiad.wmnet with OS trixie * 13:03 topranks: increase OSPF cost on ssw1-a1-codfw et-0/0/2 towards lsw1-a3-codfw [[phab:T427301|T427301]] * 13:03 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1062.eqiad.wmnet with reason: host reimage * 13:02 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Reimaging upstream servers * 13:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P93553 and previous config saved to /var/cache/conftool/dbconfig/20260602-130230-fceratto.json * 12:59 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1062.eqiad.wmnet with reason: host reimage * 12:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:57 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:57 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2161: Migration of db2161.codfw.wmnet completed * 12:54 topranks: shutdown sub-interfaces on cr1-codfw et-1/1/5 for row A/B vlans [[phab:T427301|T427301]] * 12:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 12:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93550 and previous config saved to /var/cache/conftool/dbconfig/20260602-125223-fceratto.json * 12:50 topranks: enable bgp graceful-shutdown in overlay on ssw1-a1-codfw [[phab:T427301|T427301]] * 12:49 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc1061.eqiad.wmnet with OS trixie * 12:48 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt * 12:48 ayounsi@cumin1003: START - Cookbook sre.hosts.remove-downtime for lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt * 12:47 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1062.eqiad.wmnet with OS trixie * 12:45 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93548 and previous config saved to /var/cache/conftool/dbconfig/20260602-124541-fceratto.json * 12:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance * 12:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93547 and previous config saved to /var/cache/conftool/dbconfig/20260602-124512-fceratto.json * 12:43 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc1060.eqiad.wmnet with OS trixie * 12:42 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 12:42 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mc1061.eqiad.wmnet with reason: host reimage * 12:42 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1061.eqiad.wmnet with reason: host reimage * 12:41 topranks: enable bgp graceful-shutdown in underlay on ssw1-a1-codfw [[phab:T427301|T427301]] * 12:35 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mc1060.eqiad.wmnet with reason: host reimage * 12:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P93545 and previous config saved to /var/cache/conftool/dbconfig/20260602-123505-fceratto.json * 12:33 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 12:33 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1060.eqiad.wmnet with reason: host reimage * 12:31 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2049: repool after upgrade * 12:31 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 12:29 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1061.eqiad.wmnet with OS trixie * 12:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2049.codfw.wmnet with OS trixie * 12:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P93542 and previous config saved to /var/cache/conftool/dbconfig/20260602-122459-fceratto.json * 12:24 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1059.eqiad.wmnet with OS trixie * 12:21 XioNoX: reboot lsw1-a3-codfw for software upgrade - [[phab:T427301|T427301]] * 12:20 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1060.eqiad.wmnet with OS trixie * 12:20 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 12:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1058.eqiad.wmnet with OS trixie * 12:17 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 12:16 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296532{{!}}hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533{{!}}hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] (duration: 09m 02s) * 12:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93539 and previous config saved to /var/cache/conftool/dbconfig/20260602-121451-fceratto.json * 12:11 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 12:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2049.codfw.wmnet with reason: host reimage * 12:11 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt with reason: Switch maintenance * 12:10 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2161: Migration of db2161.codfw.wmnet completed * 12:09 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Switch maintenance * 12:09 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296532{{!}}hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533{{!}}hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:08 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1200 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93537 and previous config saved to /var/cache/conftool/dbconfig/20260602-120755-fceratto.json * 12:07 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1059.eqiad.wmnet with reason: host reimage * 12:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance * 12:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93536 and previous config saved to /var/cache/conftool/dbconfig/20260602-120728-fceratto.json * 12:07 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 12:07 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296532{{!}}hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533{{!}}hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] * 12:05 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2049.codfw.wmnet with reason: host reimage * 12:04 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1058.eqiad.wmnet with reason: host reimage * 12:02 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1059.eqiad.wmnet with reason: host reimage * 12:01 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2161.codfw.wmnet with OS trixie * 12:00 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1058.eqiad.wmnet with reason: host reimage * 11:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P93535 and previous config saved to /var/cache/conftool/dbconfig/20260602-115721-fceratto.json * 11:55 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 11:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:55 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 11:53 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 11:53 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 11:53 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:50 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1059.eqiad.wmnet with OS trixie * 11:49 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1057.eqiad.wmnet with OS trixie * 11:49 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2049.codfw.wmnet with OS trixie * 11:48 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2049: Upgrading es2049.codfw.wmnet * 11:48 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2049: Upgrading es2049.codfw.wmnet * 11:47 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:47 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1058.eqiad.wmnet with OS trixie * 11:47 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2056: repool after upgrade * 11:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P93532 and previous config saved to /var/cache/conftool/dbconfig/20260602-114713-fceratto.json * 11:45 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1056.eqiad.wmnet with OS trixie * 11:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2161.codfw.wmnet with reason: host reimage * 11:40 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2161.codfw.wmnet with reason: host reimage * 11:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93531 and previous config saved to /var/cache/conftool/dbconfig/20260602-113705-fceratto.json * 11:33 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1057.eqiad.wmnet with reason: host reimage * 11:30 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1185 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93529 and previous config saved to /var/cache/conftool/dbconfig/20260602-113019-fceratto.json * 11:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance * 11:29 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1056.eqiad.wmnet with reason: host reimage * 11:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1161: Repooling * 11:26 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1161: Repooling * 11:23 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2161.codfw.wmnet with OS trixie * 11:22 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1057.eqiad.wmnet with reason: host reimage * 11:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2161: Upgrading db2161.codfw.wmnet * 11:21 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2161: Upgrading db2161.codfw.wmnet * 11:21 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1056.eqiad.wmnet with reason: host reimage * 11:21 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P93527 and previous config saved to /var/cache/conftool/dbconfig/20260602-111954-fceratto.json * 11:15 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2161 [[phab:T427892|T427892]]', diff saved to https://phabricator.wikimedia.org/P93525 and previous config saved to /var/cache/conftool/dbconfig/20260602-111511-cwilliams.json * 11:12 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2165 to s8 primary [[phab:T427892|T427892]]', diff saved to https://phabricator.wikimedia.org/P93524 and previous config saved to /var/cache/conftool/dbconfig/20260602-111200-cwilliams.json * 11:10 cezmunsta: Starting s8 codfw failover from db2161 to db2165 - [[phab:T427892|T427892]] * 11:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P93523 and previous config saved to /var/cache/conftool/dbconfig/20260602-110947-fceratto.json * 11:09 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1057.eqiad.wmnet with OS trixie * 11:09 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1056.eqiad.wmnet with OS trixie * 11:04 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2165 with weight 0 [[phab:T427892|T427892]]', diff saved to https://phabricator.wikimedia.org/P93522 and previous config saved to /var/cache/conftool/dbconfig/20260602-110420-cwilliams.json * 11:03 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s8 [[phab:T427892|T427892]] * 11:02 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2056: repool after upgrade * 11:01 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93520 and previous config saved to /var/cache/conftool/dbconfig/20260602-105939-fceratto.json * 10:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93519 and previous config saved to /var/cache/conftool/dbconfig/20260602-105239-fceratto.json * 10:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 10:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance * 10:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93518 and previous config saved to /var/cache/conftool/dbconfig/20260602-105202-fceratto.json * 10:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2056.codfw.wmnet with OS trixie * 10:42 moritzm: installing busybox security updates * 10:42 claime: Enabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 10:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P93517 and previous config saved to /var/cache/conftool/dbconfig/20260602-104154-fceratto.json * 10:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P93516 and previous config saved to /var/cache/conftool/dbconfig/20260602-103146-fceratto.json * 10:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2056.codfw.wmnet with reason: host reimage * 10:27 claime: Disabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 10:25 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2056.codfw.wmnet with reason: host reimage * 10:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93515 and previous config saved to /var/cache/conftool/dbconfig/20260602-102139-fceratto.json * 10:09 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2056.codfw.wmnet with OS trixie * 10:08 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2056: Upgrading es2056.codfw.wmnet * 10:08 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2056: Upgrading es2056.codfw.wmnet * 10:08 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:06 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply * 10:06 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply * 09:56 claime: Enabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 09:46 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on cumin2003.codfw.wmnet with reason: in setup * 09:45 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1187: Pooling * 09:37 claime: Running puppet on cp6010 and cp6011 - [[phab:T422937|T422937]] * 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow2004.codfw.wmnet to plain * 09:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93511 and previous config saved to /var/cache/conftool/dbconfig/20260602-093716-fceratto.json * 09:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance * 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow2004.codfw.wmnet to plain * 09:34 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of rpki2003.codfw.wmnet to plain * 09:34 claime: Disabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 09:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of rpki2003.codfw.wmnet to plain * 09:32 moritzm: temporarily remove ganeti2045 from the codfw cluster [[phab:T427357|T427357]] * 09:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1055.eqiad.wmnet with OS trixie * 09:15 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1187: Pooling * 09:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1055.eqiad.wmnet with reason: host reimage * 09:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1187 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93508 and previous config saved to /var/cache/conftool/dbconfig/20260602-091126-fceratto.json * 09:09 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1055.eqiad.wmnet with reason: host reimage * 09:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1187 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93506 and previous config saved to /var/cache/conftool/dbconfig/20260602-090432-fceratto.json * 09:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1187.eqiad.wmnet with reason: Maintenance * 08:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2250.codfw.wmnet with reason: rack A3 maintenance * 08:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:56 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1055.eqiad.wmnet with OS trixie * 08:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:54 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:54 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:53 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 08:52 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 08:51 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 08:50 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 08:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 08:41 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:39 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:37 urbanecm: Reset user email of Barras@votewiki to the one of Barras@SUL * 08:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance * 08:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93505 and previous config saved to /var/cache/conftool/dbconfig/20260602-083033-fceratto.json * 08:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:29 slyngs: IDP, new configuration in preparation for webauthn * 08:20 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P93504 and previous config saved to /var/cache/conftool/dbconfig/20260602-082026-fceratto.json * 08:19 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 08:18 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 08:18 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:17 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296488{{!}}Revert "translate: adding separate read/write endpoints" (T425377)]] (duration: 03m 33s) * 08:16 atsuko@deploy1003: atsuko: Rolling back deployment * 08:16 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2053: repool after upgrade * 08:15 atsuko@deploy1003: atsuko: Backport for [[gerrit:1296488{{!}}Revert "translate: adding separate read/write endpoints" (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:13 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1296488{{!}}Revert "translate: adding separate read/write endpoints" (T425377)]] * 08:11 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:10 marostegui: Install mariadb 10.11.17 on es2053 [[phab:T427345|T427345]] * 08:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P93502 and previous config saved to /var/cache/conftool/dbconfig/20260602-081018-fceratto.json * 08:09 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:09 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: Depool for rack maintenance * 08:03 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296262{{!}}translate: fixing missed variable in credentials formatting closure (T425377)]] (duration: 14m 47s) * 08:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93499 and previous config saved to /var/cache/conftool/dbconfig/20260602-080011-fceratto.json * 07:59 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 07:59 atsuko@deploy1003: atsuko: Rolling back deployment * 07:58 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 07:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1181 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93498 and previous config saved to /var/cache/conftool/dbconfig/20260602-075759-fceratto.json * 07:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1181.eqiad.wmnet with reason: Maintenance * 07:57 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling * 07:50 atsuko@deploy1003: atsuko: Backport for [[gerrit:1296262{{!}}translate: fixing missed variable in credentials formatting closure (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:49 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1296262{{!}}translate: fixing missed variable in credentials formatting closure (T425377)]] * 07:48 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1181: Pooling * 07:47 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1181: Pooling * 07:44 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1181: Reboot * 07:43 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1181: Reboot * 07:42 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1181.eqiad.wmnet with reason: Reboot * 07:41 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 07:41 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:41 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1181: Migration of db1181.eqiad.wmnet completed * 07:40 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294949{{!}}translate: adding separate read/write endpoints (T425377)]] (duration: 21m 01s) * 07:39 atsuko@deploy1003: atsuko: Rolling back deployment * 07:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93490 and previous config saved to /var/cache/conftool/dbconfig/20260602-073904-fceratto.json * 07:32 XioNoX: pfw1-eqiad# delete protocols bgp group Production family inet6 - [[phab:T423384|T423384]] * 07:30 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2053: repool after upgrade * 07:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2158.codfw.wmnet with reason: rack A3 maintenance * 07:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93487 and previous config saved to /var/cache/conftool/dbconfig/20260602-072856-fceratto.json * 07:28 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2158: rack A3 maintenance * 07:28 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2158: rack A3 maintenance * 07:27 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on pc2021.codfw.wmnet with reason: rack A3 maintenance * 07:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc2021: rack A3 maintenance * 07:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 07:25 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache * 07:25 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool pc2021: rack A3 maintenance * 07:23 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2241: Depool for rack maintenance * 07:23 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2241.codfw.wmnet * 07:23 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2241.codfw.wmnet * 07:21 atsuko@deploy1003: atsuko: Backport for [[gerrit:1294949{{!}}translate: adding separate read/write endpoints (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:20 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2053.codfw.wmnet with OS trixie * 07:19 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1294949{{!}}translate: adding separate read/write endpoints (T425377)]] * 07:15 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2241.codfw.wmnet with reason: Depool for rack maintenance * 07:14 marostegui: Install mariadb 10.11.17 on db2186 [[phab:T427345|T427345]] * 07:12 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: Depool for rack maintenance * 07:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2186.codfw.wmnet with reason: upgrade * 07:12 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2241: Depool for rack maintenance * 07:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2053.codfw.wmnet with reason: host reimage * 06:59 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2053.codfw.wmnet with reason: host reimage * 06:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93478 and previous config saved to /var/cache/conftool/dbconfig/20260602-065533-fceratto.json * 06:55 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1181: Migration of db1181.eqiad.wmnet completed * 06:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 06:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1181.eqiad.wmnet with OS trixie * 06:43 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2053.codfw.wmnet with OS trixie * 06:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2053: Upgrading es2053.codfw.wmnet * 06:41 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2053: Upgrading es2053.codfw.wmnet * 06:41 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:37 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 06:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 06:36 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 06:36 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1052: repool after upgrade * 06:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1181.eqiad.wmnet with reason: host reimage * 06:24 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1181.eqiad.wmnet with reason: host reimage * 06:22 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 06:21 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 06:16 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 06:15 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 06:08 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1181.eqiad.wmnet with OS trixie * 06:05 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1181: Upgrading db1181.eqiad.wmnet * 06:05 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1181: Upgrading db1181.eqiad.wmnet * 06:04 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:02 marostegui@dns1004: END - running authdns-update * 06:01 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1181 [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93473 and previous config saved to /var/cache/conftool/dbconfig/20260602-060157-marostegui.json * 06:01 marostegui@dns1004: START - running authdns-update * 06:00 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db1236 to s7 primary and set section read-write [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93472 and previous config saved to /var/cache/conftool/dbconfig/20260602-060041-marostegui.json * 06:00 marostegui@cumin1003: dbctl commit (dc=all): 'Set s7 eqiad as read-only for maintenance - [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93471 and previous config saved to /var/cache/conftool/dbconfig/20260602-060018-marostegui.json * 06:00 marostegui: Starting s7 eqiad failover from db1181 to db1236 - [[phab:T426088|T426088]] * 05:51 marostegui@cumin1003: dbctl commit (dc=all): 'Set db1236 with weight 0 [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93470 and previous config saved to /var/cache/conftool/dbconfig/20260602-055153-marostegui.json * 05:51 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 [[phab:T426088|T426088]] * 05:50 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1052: repool after upgrade * 05:50 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 05:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1052.eqiad.wmnet with OS trixie * 05:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1052.eqiad.wmnet with reason: host reimage * 05:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:22 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1052.eqiad.wmnet with reason: host reimage * 05:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:07 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1052.eqiad.wmnet with OS trixie * 05:06 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1052: Upgrading es1052.eqiad.wmnet * 05:06 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1052: Upgrading es1052.eqiad.wmnet * 05:05 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 04:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 04:49 ryankemper: [[phab:T425007|T425007]] (k8s) created 4 wdqs namespaces on `dse-k8s-codfw`'s `admin_ng` ns: `wdqs-[internal,external]` & `wdqs-[internal,external]-next`; certs issued * 04:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 04:40 ryankemper@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 04:36 ryankemper@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 04:05 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.2 (duration: 05m 33s) == 2026-06-01 == * 23:27 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295963{{!}}Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962{{!}}Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] (duration: 07m 17s) * 23:23 jdlrobson@deploy1003: mfossati, jdlrobson: Continuing with deployment * 23:22 jdlrobson@deploy1003: mfossati, jdlrobson: Backport for [[gerrit:1295963{{!}}Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962{{!}}Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:20 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1295963{{!}}Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962{{!}}Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] * 23:15 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296022{{!}}Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028{{!}}styles: Limit selector to badge client pref (T427407)]] (duration: 09m 33s) * 23:11 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 23:07 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1296022{{!}}Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028{{!}}styles: Limit selector to badge client pref (T427407)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:06 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1296022{{!}}Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028{{!}}styles: Limit selector to badge client pref (T427407)]] * 23:04 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6015.* * 22:36 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296024{{!}}Add maintenance script to scrape SVG render files]] (duration: 06m 22s) * 22:32 reedy@deploy1003: reedy: Continuing with deployment * 22:31 reedy@deploy1003: reedy: Backport for [[gerrit:1296024{{!}}Add maintenance script to scrape SVG render files]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:30 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1296024{{!}}Add maintenance script to scrape SVG render files]] * 22:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 22:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 22:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 21:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 21:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 21:51 sbassett: Deployed updated mitigation for [[phab:T326691|T326691]] * 21:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 21:35 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 21:35 maryum: Deployed security fix for [[phab:T427611|T427611]] * 21:35 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 21:33 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 21:32 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 21:27 maryum: Deployed security fix for [[phab:T427235|T427235]] * 21:13 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296002{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009{{!}}Redirect Special:AccountRecovery to the shared domain (T427692)]] (duration: 09m 20s) * 21:09 catrope@deploy1003: catrope, arlolra: Continuing with deployment * 21:09 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 21:09 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 21:08 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 21:07 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 21:07 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 21:06 catrope@deploy1003: catrope, arlolra: Backport for [[gerrit:1296002{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009{{!}}Redirect Special:AccountRecovery to the shared domain (T427692)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:04 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1296002{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009{{!}}Redirect Special:AccountRecovery to the shared domain (T427692)]] * 20:53 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 20:37 ryankemper@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wdqs1015.eqiad.wmnet with reason: [[phab:T427852|T427852]] hw failure * 20:26 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1285412{{!}}Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531{{!}}Enable AbuseFilter block action on nlwiki (T427384)]] (duration: 07m 48s) * 20:22 catrope@deploy1003: sfaci, xxblackburnxx, catrope: Continuing with deployment * 20:20 catrope@deploy1003: sfaci, xxblackburnxx, catrope: Backport for [[gerrit:1285412{{!}}Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531{{!}}Enable AbuseFilter block action on nlwiki (T427384)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:18 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1285412{{!}}Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531{{!}}Enable AbuseFilter block action on nlwiki (T427384)]] * 20:12 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295504{{!}}passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] (duration: 07m 37s) * 20:08 catrope@deploy1003: catrope: Continuing with deployment * 20:07 catrope@deploy1003: catrope: Backport for [[gerrit:1295504{{!}}passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:05 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1295504{{!}}passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] * 19:48 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 19:47 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 19:47 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 19:46 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 19:46 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 19:45 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 19:01 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: sync * 19:00 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: sync * 18:24 otto@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295950{{!}}mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] (duration: 06m 42s) * 18:20 otto@deploy1003: otto: Continuing with deployment * 18:19 otto@deploy1003: otto: Backport for [[gerrit:1295950{{!}}mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:17 otto@deploy1003: Started scap sync-world: Backport for [[gerrit:1295950{{!}}mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] * 18:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 18:05 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 18:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd2001.codfw.wmnet to plain * 18:02 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 18:02 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd2001.codfw.wmnet to plain * 18:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2003.codfw.wmnet to plain * 18:01 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply * 18:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2003.codfw.wmnet to plain * 17:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 17:58 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 17:53 jasmine@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2006.codfw.wmnet with OS trixie * 17:42 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295976{{!}}nlwiki: change to Wikipedia 25 logo (T424519)]] (duration: 07m 29s) * 17:37 samtar@deploy1003: chlod, samtar: Continuing with deployment * 17:36 samtar@deploy1003: chlod, samtar: Backport for [[gerrit:1295976{{!}}nlwiki: change to Wikipedia 25 logo (T424519)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:34 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1295976{{!}}nlwiki: change to Wikipedia 25 logo (T424519)]] * 17:20 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1236: Update * 17:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd2001.codfw.wmnet to drbd * 17:04 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling * 17:04 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 17:04 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1180: Pooling * 17:03 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 17:03 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling * 17:03 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 16:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd2001.codfw.wmnet to drbd * 16:58 Amir1: drop flaggedrevs tables on wikinews wikis ([[phab:T423577|T423577]]) * 16:57 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 16:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93462 and previous config saved to /var/cache/conftool/dbconfig/20260601-165717-fceratto.json * 16:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93460 and previous config saved to /var/cache/conftool/dbconfig/20260601-164709-fceratto.json * 16:42 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling * 16:37 ryankemper@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,cluster=wdqs-main,service=wdqs-main,name=wdqs1015.eqiad.wmnet * 16:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93458 and previous config saved to /var/cache/conftool/dbconfig/20260601-163701-fceratto.json * 16:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:35 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1236.eqiad.wmnet * 16:35 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1236.eqiad.wmnet * 16:35 ryankemper@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,cluster=wdqs,service=wdqs-main,name=wdqs1015.eqiad.wmnet * 16:34 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update * 16:34 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1236: Update * 16:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1236.eqiad.wmnet with reason: Kernel update [[phab:T426633|T426633]] * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:30 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1236.eqiad.wmnet * 16:30 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1236.eqiad.wmnet * 16:30 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update * 16:29 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1236: Update * 16:29 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update * 16:29 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2003.codfw.wmnet to drbd * 16:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93455 and previous config saved to /var/cache/conftool/dbconfig/20260601-162653-fceratto.json * 16:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 16:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1209: Migration of db1209.eqiad.wmnet completed * 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1236.eqiad.wmnet with reason: Kernel update [[phab:T426633|T426633]] * 16:09 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1236: Update * 16:09 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1236: Update * 16:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:06 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 16:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2003.codfw.wmnet to drbd * 16:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 16:03 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 16:02 moritzm: temporarily remove ganeti2027 from the codfw cluster [[phab:T427357|T427357]] * 15:56 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:56 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.depool (exit_code=97) depool db1224: Pooling * 15:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2005.codfw.wmnet with OS bullseye * 15:53 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1224: Pooling * 15:51 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 15:49 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling * 15:49 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:48 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet * 15:45 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling * 15:44 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2005.codfw.wmnet with reason: host reimage * 15:40 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:40 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling * 15:40 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet * 15:40 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet * 15:40 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet * 15:40 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet * 15:39 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 15:39 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 15:39 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1209: Migration of db1209.eqiad.wmnet completed * 15:39 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:38 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:38 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling * 15:37 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2005.codfw.wmnet with reason: host reimage * 15:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 15:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1209.eqiad.wmnet with OS trixie * 15:28 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295802{{!}}hCaptcha: Raise SiteVerify error threshold to 100]] (duration: 06m 15s) * 15:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93446 and previous config saved to /var/cache/conftool/dbconfig/20260601-152638-fceratto.json * 15:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 15:26 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:25 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet * 15:25 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet * 15:25 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling * 15:25 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:24 kharlan@deploy1003: kharlan: Continuing with deployment * 15:24 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295802{{!}}hCaptcha: Raise SiteVerify error threshold to 100]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:22 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host testvm2005.codfw.wmnet with OS bullseye * 15:22 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295802{{!}}hCaptcha: Raise SiteVerify error threshold to 100]] * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:20 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295946{{!}}hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] (duration: 08m 24s) * 15:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:16 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 15:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1209.eqiad.wmnet with reason: host reimage * 15:14 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1295946{{!}}hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:13 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:12 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1295946{{!}}hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] * 15:10 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1209.eqiad.wmnet with reason: host reimage * 15:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93445 and previous config saved to /var/cache/conftool/dbconfig/20260601-151024-fceratto.json * 15:08 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-reboot (exit_code=0) rolling reboot on A:sessionstore * 15:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93443 and previous config saved to /var/cache/conftool/dbconfig/20260601-150017-fceratto.json * 14:55 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1209.eqiad.wmnet with OS trixie * 14:52 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 14:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1209: Upgrading db1209.eqiad.wmnet * 14:52 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 14:52 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1209: Upgrading db1209.eqiad.wmnet * 14:52 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 14:51 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:51 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 14:50 atsuko@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 14:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93441 and previous config saved to /var/cache/conftool/dbconfig/20260601-145010-fceratto.json * 14:49 atsuko@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 14:49 atsuko@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 14:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:42 atsuko@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 14:41 atsuko@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93440 and previous config saved to /var/cache/conftool/dbconfig/20260601-144002-fceratto.json * 14:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:30 ladsgroup@deploy1003: Synchronized portals: Deploy portals ([[phab:T421797|T421797]]) (duration: 02m 43s) * 14:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:27 ladsgroup@deploy1003: Synchronized portals/wikipedia.org/assets: Deploy portals ([[phab:T421797|T421797]]) (duration: 06m 10s) * 14:25 sukhe@dns1004: END - running authdns-update * 14:23 sukhe@dns1004: START - running authdns-update * 14:22 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 14:21 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:16 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:12 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:12 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:11 Lucas_WMDE: UTC afternoon backport+config window done * 14:10 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295918{{!}}Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] (duration: 11m 06s) * 14:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:05 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, codenamenoreste: Continuing with deployment * 14:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, codenamenoreste: Backport for [[gerrit:1295918{{!}}Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:01 eevans@cumin1003: START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:sessionstore * 13:58 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1295918{{!}}Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] * 13:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:52 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1265.eqiad.wmnet with OS trixie * 13:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93439 and previous config saved to /var/cache/conftool/dbconfig/20260601-133947-fceratto.json * 13:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 13:37 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage * 13:35 atsukoito: restarted pybal.service on lvs2013 * 13:31 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage * 13:31 atsukoito: restarted pybal.service on lvs2014 * 13:24 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs-test2001.codfw.wmnet * 13:24 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:22 atsukoito: restarted pybal.service on lvs1019 * 13:22 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in eqiad/ml-serve-eqiad: maintenance * 13:21 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in eqiad/ml-serve-eqiad: maintenance * 13:20 atsukoito: restarted pybal.service on lvs1020 * 13:20 Msz2001: UTC afternoon backpot+config window done * 13:20 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295875{{!}}Add SetGlobalPreference maintenance script (T427476)]] (duration: 06m 22s) * 13:19 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs-test2001.codfw.wmnet * 13:18 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1265.eqiad.wmnet with OS trixie * 13:18 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:16 mszwarc@deploy1003: mszwarc: Continuing with deployment * 13:15 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1295875{{!}}Add SetGlobalPreference maintenance script (T427476)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:14 atsukoito: sudo cumin 'A:lvs-low-traffic-eqiad' 'systemctl restart pybal.service' * 13:14 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1295875{{!}}Add SetGlobalPreference maintenance script (T427476)]] * 13:12 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295536{{!}}swwiki: Enable the Visual Editor on the project namespace (T427117)]] (duration: 10m 06s) * 13:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93438 and previous config saved to /var/cache/conftool/dbconfig/20260601-130949-fceratto.json * 13:08 mszwarc@deploy1003: codenamenoreste, mszwarc: Continuing with deployment * 13:07 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 13:06 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . * 13:05 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 13:04 mszwarc@deploy1003: codenamenoreste, mszwarc: Backport for [[gerrit:1295536{{!}}swwiki: Enable the Visual Editor on the project namespace (T427117)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:04 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 13:04 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 13:03 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 13:02 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1295536{{!}}swwiki: Enable the Visual Editor on the project namespace (T427117)]] * 12:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93437 and previous config saved to /var/cache/conftool/dbconfig/20260601-125941-fceratto.json * 12:56 dpogorzelski@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=inference,name=eqiad * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'readability' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'logo-detection' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'edit-check' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . * 12:52 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:50 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:49 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93436 and previous config saved to /var/cache/conftool/dbconfig/20260601-124934-fceratto.json * 12:48 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:47 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:46 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:44 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:43 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:42 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:41 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93435 and previous config saved to /var/cache/conftool/dbconfig/20260601-123926-fceratto.json * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:29 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:28 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 12:28 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:27 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:27 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2005.codfw.wmnet to plain * 12:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2005.codfw.wmnet to plain * 12:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet * 12:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 12:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2005.codfw.wmnet to drbd * 12:20 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:17 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:15 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in eqiad/ml-serve-eqiad: maintenance * 12:15 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in eqiad/ml-serve-eqiad: maintenance * 12:11 dpogorzelski@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=inference,name=eqiad * 12:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2005.codfw.wmnet to drbd * 12:05 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet * 12:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 12:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2027.codfw.wmnet * 12:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 11:59 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in eqiad/ml-serve-eqiad: maintenance * 11:59 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in eqiad/ml-serve-eqiad: maintenance * 11:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93434 and previous config saved to /var/cache/conftool/dbconfig/20260601-113911-fceratto.json * 11:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 11:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93433 and previous config saved to /var/cache/conftool/dbconfig/20260601-113843-fceratto.json * 11:37 moritzm: installing Exim security updates * 11:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:32 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:32 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:32 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93432 and previous config saved to /var/cache/conftool/dbconfig/20260601-112835-fceratto.json * 11:25 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 11:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:22 moritzm: installing imagemagick security updates * 11:22 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:22 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:22 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 11:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93430 and previous config saved to /var/cache/conftool/dbconfig/20260601-111827-fceratto.json * 11:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:14 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 11:12 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 11:10 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93429 and previous config saved to /var/cache/conftool/dbconfig/20260601-110820-fceratto.json * 11:04 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 11:01 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1055: repool after upgrade * 11:01 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93427 and previous config saved to /var/cache/conftool/dbconfig/20260601-110121-fceratto.json * 11:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance * 10:54 marostegui@dns1004: END - running authdns-update * 10:52 marostegui@dns1004: START - running authdns-update * 10:48 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1050 to es1 eqiad primary [[phab:T427032|T427032]]', diff saved to https://phabricator.wikimedia.org/P93425 and previous config saved to /var/cache/conftool/dbconfig/20260601-104837-marostegui.json * 10:47 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2055 to es1 codfw primary [[phab:T427032|T427032]]', diff saved to https://phabricator.wikimedia.org/P93424 and previous config saved to /var/cache/conftool/dbconfig/20260601-104739-marostegui.json * 10:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1177: Migration of db1177.eqiad.wmnet completed * 10:40 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet * 10:34 kamila@cumin1003: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet * 10:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93421 and previous config saved to /var/cache/conftool/dbconfig/20260601-103316-fceratto.json * 10:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93418 and previous config saved to /var/cache/conftool/dbconfig/20260601-102308-fceratto.json * 10:16 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1055: repool after upgrade * 10:15 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1055.eqiad.wmnet with OS trixie * 10:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93415 and previous config saved to /var/cache/conftool/dbconfig/20260601-101300-fceratto.json * 10:09 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 10:07 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 10:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93414 and previous config saved to /var/cache/conftool/dbconfig/20260601-100252-fceratto.json * 10:00 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1177: Migration of db1177.eqiad.wmnet completed * 09:58 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1055.eqiad.wmnet with reason: host reimage * 09:56 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 09:54 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 09:53 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1055.eqiad.wmnet with reason: host reimage * 09:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1177.eqiad.wmnet with OS trixie * 09:51 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 09:50 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 09:39 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1055.eqiad.wmnet with OS trixie * 09:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1055: Upgrading es1055.eqiad.wmnet * 09:38 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1055: Upgrading es1055.eqiad.wmnet * 09:37 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1177.eqiad.wmnet with reason: host reimage * 09:31 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1177.eqiad.wmnet with reason: host reimage * 09:17 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1177.eqiad.wmnet with OS trixie * 09:15 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 09:14 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 09:13 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 09:12 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 09:12 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1177: Upgrading db1177.eqiad.wmnet * 09:11 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1177: Upgrading db1177.eqiad.wmnet * 09:11 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93410 and previous config saved to /var/cache/conftool/dbconfig/20260601-090237-fceratto.json * 09:02 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93409 and previous config saved to /var/cache/conftool/dbconfig/20260601-090209-fceratto.json * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P93408 and previous config saved to /var/cache/conftool/dbconfig/20260601-085202-fceratto.json * 08:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P93407 and previous config saved to /var/cache/conftool/dbconfig/20260601-084154-fceratto.json * 08:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93406 and previous config saved to /var/cache/conftool/dbconfig/20260601-083146-fceratto.json * 08:24 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93405 and previous config saved to /var/cache/conftool/dbconfig/20260601-082442-fceratto.json * 08:24 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1168.eqiad.wmnet with reason: Maintenance * 07:58 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295454{{!}}Disable the creation of synthetic main refs in production (T427484)]] (duration: 11m 26s) * 07:56 XioNoX: add no_p2p term to pfw1-codfw BGP_fundraising_export - [[phab:T423384|T423384]] * 07:52 wmde-fisch@deploy1003: lilients, wmde-fisch: Continuing with deployment * 07:51 wmde-fisch@deploy1003: lilients, wmde-fisch: Backport for [[gerrit:1295454{{!}}Disable the creation of synthetic main refs in production (T427484)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:47 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1295454{{!}}Disable the creation of synthetic main refs in production (T427484)]] * 07:45 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294826{{!}}Update VE core submodule to master (9cf5524e7) (T424232)]] (duration: 31m 34s) * 07:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 07:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 07:32 wmde-fisch@deploy1003: wmde-fisch: Continuing with deployment * 07:31 wmde-fisch@deploy1003: wmde-fisch: Backport for [[gerrit:1294826{{!}}Update VE core submodule to master (9cf5524e7) (T424232)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet * 07:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet * 07:13 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1294826{{!}}Update VE core submodule to master (9cf5524e7) (T424232)]] * 06:48 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 06:47 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. == 2026-05-31 == * 02:06 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 30s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-30 == * 16:21 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:21 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:21 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:21 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 06:39 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 06:39 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 06:39 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 06:38 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 27s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-29 == * 23:39 aokoth@cumin1003: END (PASS) - Cookbook sre.vrts.upgrade (exit_code=0) on VRTS host vrts1003.eqiad.wmnet * 23:37 aokoth@cumin1003: START - Cookbook sre.vrts.upgrade on VRTS host vrts1003.eqiad.wmnet * 21:42 catrope@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 21:41 catrope@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 17:40 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295487{{!}}Hide experiment if not active and no assigned group]] (duration: 06m 54s) * 17:35 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 17:34 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1295487{{!}}Hide experiment if not active and no assigned group]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:33 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1295487{{!}}Hide experiment if not active and no assigned group]] * 16:30 jgreen@dns1004: END - running authdns-update * 16:28 jgreen@dns1004: START - running authdns-update * 16:13 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:12 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 15:28 dancy@deploy1003: Installation of scap version "4.267.0" completed for 2 hosts * 15:26 dancy@deploy1003: Installing scap version "4.267.0" for 2 host(s) * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:15 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295466{{!}}GlobalPreferencesHandler: Cast auto-reveal expiry to int (T427625)]] (duration: 07m 58s) * 14:11 kharlan@deploy1003: kharlan: Continuing with deployment * 14:09 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295466{{!}}GlobalPreferencesHandler: Cast auto-reveal expiry to int (T427625)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:07 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295466{{!}}GlobalPreferencesHandler: Cast auto-reveal expiry to int (T427625)]] * 13:53 moritzm: imported OpenJDK 21 21.0.11+10-1~deb12u1 to component/jdk21 (backport of latest Java 21 security release for Bookworm) * 12:09 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader1006.wikimedia.org * 12:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1006.wikimedia.org with OS trixie * 11:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1006.wikimedia.org with reason: host reimage * 11:47 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1006.wikimedia.org with reason: host reimage * 11:36 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader1006.wikimedia.org with OS trixie * 11:15 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:15 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:13 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader1006.wikimedia.org on all recursors * 11:12 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader1006.wikimedia.org on all recursors * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:06 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:00 jmm@cumin2002: START - Cookbook sre.dns.netbox * 11:00 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader1006.wikimedia.org * 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader1005.wikimedia.org * 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1005.wikimedia.org with OS trixie * 10:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1005.wikimedia.org with reason: host reimage * 10:40 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2212: Pooling * 10:37 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1005.wikimedia.org with reason: host reimage * 10:27 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader1005.wikimedia.org with OS trixie * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 10:01 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:59 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:55 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 09:50 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 09:49 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:45 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:44 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup2014.codfw.wmnet with OS bookworm * 09:33 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:20 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup2014.codfw.wmnet with reason: host reimage * 09:12 jynus@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on backup2014.codfw.wmnet with reason: host reimage * 09:10 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 09:10 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 09:03 jelto@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM etherpad2002.codfw.wmnet * 08:59 jelto@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM etherpad2002.codfw.wmnet * 08:59 jelto: gnt-instance modify -B memory=4g,vcpus=1 etherpad2002.codfw.wmnet - [[phab:T427588|T427588]] * 08:54 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2014.codfw.wmnet with OS bookworm * 08:51 jelto@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM etherpad1004.eqiad.wmnet * 08:50 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams-internal: apply * 08:50 jynus@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host backup2014.codfw.wmnet with OS bookworm * 08:49 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams-internal: apply * 08:47 jelto@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM etherpad1004.eqiad.wmnet * 08:46 jelto: gnt-instance modify -B memory=4g,vcpus=1 etherpad1004.eqiad.wmnet - [[phab:T427588|T427588]] * 08:42 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2212: Pooling * 08:42 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 08:39 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2212: Pooling * 08:39 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 08:38 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams-internal: apply * 08:37 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams-internal: apply * 08:37 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams-internal: apply * 08:36 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams-internal: apply * 08:33 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2014.codfw.wmnet with OS bookworm * 08:31 jynus@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host backup2014.codfw.wmnet with OS bookworm * 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader1005.wikimedia.org on all recursors * 08:21 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader1005.wikimedia.org on all recursors * 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 08:21 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 08:18 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 08:17 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 08:16 jmm@cumin2002: START - Cookbook sre.dns.netbox * 08:16 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader1005.wikimedia.org * 08:05 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2212: Pooling * 07:59 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 07:59 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 07:54 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 07:54 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2212.codfw.wmnet * 07:54 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2212.codfw.wmnet * 07:22 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2014.codfw.wmnet with OS bookworm * 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader2006.wikimedia.org * 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader2006.wikimedia.org with OS trixie * 06:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader2006.wikimedia.org with reason: host reimage * 06:53 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader2006.wikimedia.org with reason: host reimage * 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader2006.wikimedia.org with OS trixie * 06:32 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:32 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader2006.wikimedia.org on all recursors * 06:31 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader2006.wikimedia.org on all recursors * 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:31 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:27 jmm@cumin2002: START - Cookbook sre.dns.netbox * 06:27 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader2006.wikimedia.org * 03:01 vriley@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts db1224.eqiad.wmnet * 03:00 vriley@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1224.eqiad.wmnet * 03:00 vriley@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1224.eqiad.wmnet * 02:56 vriley@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1224.eqiad.wmnet * 01:47 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5032.eqsin.wmnet with OS trixie * 01:18 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5032.eqsin.wmnet with reason: host reimage * 01:14 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5032.eqsin.wmnet with reason: host reimage * 00:31 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cp5032.eqsin.wmnet with OS trixie * 00:29 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host cp5032.eqsin.wmnet * 00:23 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 00:22 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply * 00:21 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 00:21 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply == 2026-05-28 == * 23:07 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 23:07 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new ae1.522 interface - pt1979@cumin2002" * 23:07 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new ae1.522 interface - pt1979@cumin2002" * 23:02 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 22:34 andrewbogott: reprepro includedeb trixie-wikimedia /home/andrew/magnum-cluster-api_0.36.6-1~wmf13u2_amd64.deb * 22:31 logmsgbot: dreamyjazz Deployed security patch for [[phab:T426388|T426388]] * 21:33 maryum: Deployed security fix for [[phab:T426867|T426867]] * 21:21 alexsanford: Deployed security fix for [[phab:T426889|T426889]] * 21:07 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host cp5032.eqsin.wmnet * 21:04 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "setup new eqsin vlan - pt1979@cumin2002 - [[phab:T427393|T427393]]" * 21:04 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "setup new eqsin vlan - pt1979@cumin2002 - [[phab:T427393|T427393]]" * 20:48 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295066{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T420336 T427098 T427354 T427082)]], [[gerrit:1295067{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T427082)]] (duration: 07m 34s) * 20:44 arlolra@deploy1003: arlolra: Continuing with deployment * 20:43 arlolra@deploy1003: arlolra: Backport for [[gerrit:1295066{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T420336 T427098 T427354 T427082)]], [[gerrit:1295067{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T427082)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:41 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1295066{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T420336 T427098 T427354 T427082)]], [[gerrit:1295067{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T427082)]] * 20:34 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293805{{!}}Deploy PRV to 7 wikis (T427331)]] (duration: 07m 20s) * 20:30 arlolra@deploy1003: arlolra: Continuing with deployment * 20:29 arlolra@deploy1003: arlolra: Backport for [[gerrit:1293805{{!}}Deploy PRV to 7 wikis (T427331)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:27 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1293805{{!}}Deploy PRV to 7 wikis (T427331)]] * 20:22 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291996{{!}}Replace deprecated Hooks::getInstance (T426981)]], [[gerrit:1294393{{!}}Permissions: Create wmf-officeit group on officewiki]], [[gerrit:1294229{{!}}Deploy IRS Direct Reporting feature to enwiki (T427369)]], [[gerrit:1295039{{!}}Add 2FA enforcement demotion config for phase 2 groups (T423119)]] (duration: 09m 07s) * 20:18 stran@deploy1003: alexsanford, stran, catrope, dreamyjazz: Continuing with deployment * 20:14 stran@deploy1003: alexsanford, stran, catrope, dreamyjazz: Backport for [[gerrit:1291996{{!}}Replace deprecated Hooks::getInstance (T426981)]], [[gerrit:1294393{{!}}Permissions: Create wmf-officeit group on officewiki]], [[gerrit:1294229{{!}}Deploy IRS Direct Reporting feature to enwiki (T427369)]], [[gerrit:1295039{{!}}Add 2FA enforcement demotion config for phase 2 groups (T423119)]] synced to the testservers (see https://wikitech. * 20:13 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5032.eqsin.wmnet with OS trixie * 20:13 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1291996{{!}}Replace deprecated Hooks::getInstance (T426981)]], [[gerrit:1294393{{!}}Permissions: Create wmf-officeit group on officewiki]], [[gerrit:1294229{{!}}Deploy IRS Direct Reporting feature to enwiki (T427369)]], [[gerrit:1295039{{!}}Add 2FA enforcement demotion config for phase 2 groups (T423119)]] * 19:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1018.eqiad.wmnet * 19:27 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1018.eqiad.wmnet * 19:09 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1018.eqiad.wmnet with reason: Kernel reboot * 19:09 brett: Stopping pybal/puppet/downtiming lvs1018.eqiad.wmnet for reboot * 19:05 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1019.eqiad.wmnet * 19:05 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1019.eqiad.wmnet * 18:52 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cp5032.eqsin.wmnet with OS trixie * 18:51 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:51 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change cp5032 IP - pt1979@cumin2002" * 18:51 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change cp5032 IP - pt1979@cumin2002" * 18:47 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 18:40 mutante: planet1003/planet2003 - apt-get upgrade - all pending package upgrades * 18:35 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: Kernel reboot * 18:34 brett: Stopping pybal/puppet/downtiming lvs1019.eqiad.wmnet for reboot and BIOS update/memory self-healing - [[phab:T426109|T426109]] * 18:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs2011.codfw.wmnet * 18:25 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs2011.codfw.wmnet * 18:19 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2011.codfw.wmnet with reason: Kernel reboot * 18:19 brett: Stopping pybal/puppet/downtiming lvs2011.codfw.wmnet for reboot * 18:09 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs2013.codfw.wmnet * 18:06 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs2013.codfw.wmnet * 18:00 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2013.codfw.wmnet with reason: Kernel reboot * 17:57 brett: Stopping pybal/puppet/downtiming lvs2013.codfw.wmnet for reboot * 17:19 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 16:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93393 and previous config saved to /var/cache/conftool/dbconfig/20260528-164514-fceratto.json * 16:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P93392 and previous config saved to /var/cache/conftool/dbconfig/20260528-163507-fceratto.json * 16:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P93391 and previous config saved to /var/cache/conftool/dbconfig/20260528-162459-fceratto.json * 16:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db1224.eqiad.wmnet with reason: unreachable [[phab:T427535|T427535]] * 16:17 swfrench-wmf: reprepro include xdebug_3.4.4-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:17 swfrench-wmf: reprepro include wikidiff2_1.14.1-2+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:17 swfrench-wmf: reprepro include php-yaml_2.2.4-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-xhprof_2.3.10-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-wmerrors_2.0.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-uuid_1.3.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-redis_6.2.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 swfrench-wmf: reprepro include php-pcov_1.0.12-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 swfrench-wmf: reprepro include php-memcached_3.3.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 16:15 swfrench-wmf: reprepro include php-luasandbox_4.1.2-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 16:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93390 and previous config saved to /var/cache/conftool/dbconfig/20260528-161452-fceratto.json * 16:14 swfrench-wmf: reprepro include php-imagick_3.7.0-13+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:14 swfrench-wmf: reprepro include php-excimer_1.2.5-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:09 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:09 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:06 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1251 ([[phab:T426633|T426633]])', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20260528-160646-fceratto.json * 16:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1251.eqiad.wmnet with reason: Maintenance * 16:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93388 and previous config saved to /var/cache/conftool/dbconfig/20260528-160613-fceratto.json * 15:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P93387 and previous config saved to /var/cache/conftool/dbconfig/20260528-155605-fceratto.json * 15:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P93386 and previous config saved to /var/cache/conftool/dbconfig/20260528-154557-fceratto.json * 15:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93385 and previous config saved to /var/cache/conftool/dbconfig/20260528-153550-fceratto.json * 15:27 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1235 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93384 and previous config saved to /var/cache/conftool/dbconfig/20260528-152736-fceratto.json * 15:27 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1235.eqiad.wmnet with reason: Maintenance * 15:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93383 and previous config saved to /var/cache/conftool/dbconfig/20260528-152708-fceratto.json * 15:20 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp5032.eqsin.wmnet with reason: Testing reimaging on new subnet * 15:18 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5032.* * 15:17 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P93382 and previous config saved to /var/cache/conftool/dbconfig/20260528-151701-fceratto.json * 15:17 jhathaway: dmarc ingress test on mx-in1001 * 15:14 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:14 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P93381 and previous config saved to /var/cache/conftool/dbconfig/20260528-150653-fceratto.json * 14:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93380 and previous config saved to /var/cache/conftool/dbconfig/20260528-145646-fceratto.json * 14:56 moritzm: installing nginx security updates * 14:49 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1234 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93379 and previous config saved to /var/cache/conftool/dbconfig/20260528-144936-fceratto.json * 14:49 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 14:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1234.eqiad.wmnet with reason: Maintenance * 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93378 and previous config saved to /var/cache/conftool/dbconfig/20260528-144909-fceratto.json * 14:48 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader2005.wikimedia.org * 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader2005.wikimedia.org with OS trixie * 14:47 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 14:39 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2189.codfw.wmnet * 14:39 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2189.codfw.wmnet * 14:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P93377 and previous config saved to /var/cache/conftool/dbconfig/20260528-143901-fceratto.json * 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader2005.wikimedia.org with reason: host reimage * 14:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P93376 and previous config saved to /var/cache/conftool/dbconfig/20260528-142854-fceratto.json * 14:28 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:28 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader2005.wikimedia.org with reason: host reimage * 14:27 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:19 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294998{{!}}ImageContentLookup: Fix issue created by strict types (T427505)]], [[gerrit:1295001{{!}}Enable hCaptcha for VisualEditor in group 1 (T425940)]] (duration: 11m 29s) * 14:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93375 and previous config saved to /var/cache/conftool/dbconfig/20260528-141846-fceratto.json * 14:15 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1232 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93374 and previous config saved to /var/cache/conftool/dbconfig/20260528-141029-fceratto.json * 14:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1232.eqiad.wmnet with reason: Maintenance * 14:10 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader2005.wikimedia.org with OS trixie * 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93373 and previous config saved to /var/cache/conftool/dbconfig/20260528-141001-fceratto.json * 14:09 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1294998{{!}}ImageContentLookup: Fix issue created by strict types (T427505)]], [[gerrit:1295001{{!}}Enable hCaptcha for VisualEditor in group 1 (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:08 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294998{{!}}ImageContentLookup: Fix issue created by strict types (T427505)]], [[gerrit:1295001{{!}}Enable hCaptcha for VisualEditor in group 1 (T425940)]] * 14:00 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down * 13:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P93371 and previous config saved to /var/cache/conftool/dbconfig/20260528-135951-fceratto.json * 13:58 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp6015.drmrs.wmnet,service=(cdn{{!}}ats-be) * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:55 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader2005.wikimedia.org on all recursors * 13:55 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader2005.wikimedia.org on all recursors * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:55 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P93370 and previous config saved to /var/cache/conftool/dbconfig/20260528-134944-fceratto.json * 13:40 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 13:40 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 13:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93369 and previous config saved to /var/cache/conftool/dbconfig/20260528-133936-fceratto.json * 13:39 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 13:38 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 13:36 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294986{{!}}Image Carousel: check candidate pages (T427336)]] (duration: 06m 40s) * 13:34 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 13:33 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 13:32 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93368 and previous config saved to /var/cache/conftool/dbconfig/20260528-133230-fceratto.json * 13:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance * 13:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93367 and previous config saved to /var/cache/conftool/dbconfig/20260528-133202-fceratto.json * 13:31 mlitn@deploy1003: mlitn: Continuing with deployment * 13:31 mlitn@deploy1003: mlitn: Backport for [[gerrit:1294986{{!}}Image Carousel: check candidate pages (T427336)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:29 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1294986{{!}}Image Carousel: check candidate pages (T427336)]] * 13:22 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 13:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P93366 and previous config saved to /var/cache/conftool/dbconfig/20260528-132155-fceratto.json * 13:21 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 13:17 elukey: clean up a lof ot stale Kafka ACLs on Kafka Jumbo - Details in [[phab:T425528|T425528]] * 13:14 jmm@cumin2002: START - Cookbook sre.dns.netbox * 13:14 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader2005.wikimedia.org * 13:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P93365 and previous config saved to /var/cache/conftool/dbconfig/20260528-131147-fceratto.json * 13:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93364 and previous config saved to /var/cache/conftool/dbconfig/20260528-130139-fceratto.json * 12:54 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1218 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93363 and previous config saved to /var/cache/conftool/dbconfig/20260528-125439-fceratto.json * 12:54 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1218.eqiad.wmnet with reason: Maintenance * 12:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93362 and previous config saved to /var/cache/conftool/dbconfig/20260528-125412-fceratto.json * 12:48 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 12:48 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 12:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P93361 and previous config saved to /var/cache/conftool/dbconfig/20260528-124404-fceratto.json * 12:44 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 12:43 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 12:39 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 12:38 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 12:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P93360 and previous config saved to /var/cache/conftool/dbconfig/20260528-123357-fceratto.json * 12:25 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest1006.eqiad.wmnet with OS trixie * 12:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93359 and previous config saved to /var/cache/conftool/dbconfig/20260528-122349-fceratto.json * 12:15 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93358 and previous config saved to /var/cache/conftool/dbconfig/20260528-121551-fceratto.json * 12:15 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance * 12:15 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie * 12:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93357 and previous config saved to /var/cache/conftool/dbconfig/20260528-121523-fceratto.json * 12:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P93356 and previous config saved to /var/cache/conftool/dbconfig/20260528-120515-fceratto.json * 12:02 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest1006.eqiad.wmnet with OS trixie * 12:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 12:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 12:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 12:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 11:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P93355 and previous config saved to /var/cache/conftool/dbconfig/20260528-115508-fceratto.json * 11:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93354 and previous config saved to /var/cache/conftool/dbconfig/20260528-114500-fceratto.json * 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93353 and previous config saved to /var/cache/conftool/dbconfig/20260528-113635-fceratto.json * 11:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 11:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance * 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93352 and previous config saved to /var/cache/conftool/dbconfig/20260528-113559-fceratto.json * 11:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P93351 and previous config saved to /var/cache/conftool/dbconfig/20260528-112551-fceratto.json * 11:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P93350 and previous config saved to /var/cache/conftool/dbconfig/20260528-111543-fceratto.json * 11:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93349 and previous config saved to /var/cache/conftool/dbconfig/20260528-110536-fceratto.json * 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1195 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93348 and previous config saved to /var/cache/conftool/dbconfig/20260528-105820-fceratto.json * 10:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie * 10:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1195.eqiad.wmnet with reason: Maintenance * 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93347 and previous config saved to /var/cache/conftool/dbconfig/20260528-105753-fceratto.json * 10:56 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 10:55 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 10:55 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 10:55 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 10:50 moritzm: update trixie netboot image for 13.5 point release [[phab:T427072|T427072]] * 10:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P93346 and previous config saved to /var/cache/conftool/dbconfig/20260528-104745-fceratto.json * 10:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P93345 and previous config saved to /var/cache/conftool/dbconfig/20260528-103738-fceratto.json * 10:29 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P13724 # [[phab:T406971|T406971]] * 10:28 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P14223 # [[phab:T422264|T422264]] * 10:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93344 and previous config saved to /var/cache/conftool/dbconfig/20260528-102730-fceratto.json * 10:26 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P1748 # [[phab:T422392|T422392]] * 10:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93343 and previous config saved to /var/cache/conftool/dbconfig/20260528-101900-fceratto.json * 10:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance * 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93342 and previous config saved to /var/cache/conftool/dbconfig/20260528-101829-fceratto.json * 10:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P93341 and previous config saved to /var/cache/conftool/dbconfig/20260528-100822-fceratto.json * 09:59 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290687{{!}}stream: webrequest.page_view (T426092 T426091)]] (duration: 06m 41s) * 09:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P93340 and previous config saved to /var/cache/conftool/dbconfig/20260528-095814-fceratto.json * 09:55 javiermonton@deploy1003: javiermonton: Continuing with deployment * 09:54 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1290687{{!}}stream: webrequest.page_view (T426092 T426091)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:52 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1290687{{!}}stream: webrequest.page_view (T426092 T426091)]] * 09:48 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294243{{!}}Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937{{!}}CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] (duration: 07m 37s) * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93339 and previous config saved to /var/cache/conftool/dbconfig/20260528-094807-fceratto.json * 09:44 dreamyjazz@deploy1003: dreamyjazz, stran: Continuing with deployment * 09:44 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:43 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:43 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:43 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:42 dreamyjazz@deploy1003: dreamyjazz, stran: Backport for [[gerrit:1294243{{!}}Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937{{!}}CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294243{{!}}Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937{{!}}CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] * 09:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93338 and previous config saved to /var/cache/conftool/dbconfig/20260528-093920-fceratto.json * 09:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: Maintenance * 09:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93337 and previous config saved to /var/cache/conftool/dbconfig/20260528-093849-fceratto.json * 09:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P93336 and previous config saved to /var/cache/conftool/dbconfig/20260528-092842-fceratto.json * 09:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance * 09:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93335 and previous config saved to /var/cache/conftool/dbconfig/20260528-092239-fceratto.json * 09:22 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pki-root1001.eqiad.wmnet * 09:22 elukey@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:22 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki-root1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1003" * 09:22 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki-root1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1003" * 09:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:18 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 09:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P93334 and previous config saved to /var/cache/conftool/dbconfig/20260528-091834-fceratto.json * 09:18 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 09:18 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 09:17 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1165: Reboot completed * 09:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 09:17 elukey@cumin1003: START - Cookbook sre.dns.netbox * 09:14 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 09:13 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:13 elukey@cumin1003: START - Cookbook sre.hosts.decommission for hosts pki-root1001.eqiad.wmnet * 09:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P93332 and previous config saved to /var/cache/conftool/dbconfig/20260528-091231-fceratto.json * 09:09 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:09 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93331 and previous config saved to /var/cache/conftool/dbconfig/20260528-090826-fceratto.json * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P93329 and previous config saved to /var/cache/conftool/dbconfig/20260528-090224-fceratto.json * 09:02 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Deploying to prod (duration: 02m 31s) * 09:01 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2216 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93328 and previous config saved to /var/cache/conftool/dbconfig/20260528-090114-fceratto.json * 09:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2216.codfw.wmnet with reason: Maintenance * 09:00 joal@deploy1003: Finished deploy [analytics/refinery@878cb24] (thin): Regular analytics weekly train THIN - 2[analytics/refinery@878cb24a] (duration: 02m 08s) * 08:59 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Deploying to prod * 08:58 joal@deploy1003: Started deploy [analytics/refinery@878cb24] (thin): Regular analytics weekly train THIN - 2[analytics/refinery@878cb24a] * 08:57 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Testing on backup host (duration: 00m 53s) * 08:56 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Testing on backup host * 08:56 joal@deploy1003: Finished deploy [analytics/refinery@878cb24]: Regular analytics weekly train - 2 [analytics/refinery@878cb24a] (duration: 06m 54s) * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93327 and previous config saved to /var/cache/conftool/dbconfig/20260528-085216-fceratto.json * 08:50 XioNoX: cr1-codfw# delete protocols bgp group fundraising family inet6 - [[phab:T423384|T423384]] * 08:49 joal@deploy1003: Started deploy [analytics/refinery@878cb24]: Regular analytics weekly train - 2 [analytics/refinery@878cb24a] * 08:49 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294925{{!}}hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] (duration: 09m 20s) * 08:49 joal@deploy1003: Finished deploy [analytics/refinery@878cb24] (hadoop-test): Regular analytics weekly train TEST -2 [analytics/refinery@878cb24a] (duration: 02m 00s) * 08:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93326 and previous config saved to /var/cache/conftool/dbconfig/20260528-084906-fceratto.json * 08:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1209.eqiad.wmnet with reason: Maintenance * 08:48 slyngshede@dns1004: END - running authdns-update * 08:47 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1165: Reboot completed * 08:47 joal@deploy1003: Started deploy [analytics/refinery@878cb24] (hadoop-test): Regular analytics weekly train TEST -2 [analytics/refinery@878cb24a] * 08:47 slyngs: Upgrade IDP to CAS 7.3.7.1 * 08:46 slyngshede@dns1004: START - running authdns-update * 08:45 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 08:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93324 and previous config saved to /var/cache/conftool/dbconfig/20260528-084149-fceratto.json * 08:41 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1294925{{!}}hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294925{{!}}hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] * 08:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet * 08:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet * 08:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93323 and previous config saved to /var/cache/conftool/dbconfig/20260528-083504-fceratto.json * 08:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1025].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 08:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1165.eqiad.wmnet with reason: Maintenance * 08:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93322 and previous config saved to /var/cache/conftool/dbconfig/20260528-083331-fceratto.json * 08:24 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1209: Test * 08:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P93320 and previous config saved to /var/cache/conftool/dbconfig/20260528-082324-fceratto.json * 08:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2189: repool after crash * 08:17 slyngshede@dns1004: END - running authdns-update * 08:16 slyngshede@dns1004: START - running authdns-update * 08:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P93318 and previous config saved to /var/cache/conftool/dbconfig/20260528-081316-fceratto.json * 08:10 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 08:09 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1209: Test * 08:05 hashar@deploy1003: Finished deploy [integration/docroot@2a51016]: build: update dependencies + eslint fix in comment. f021d3f..2a51016 (duration: 00m 13s) * 08:05 hashar@deploy1003: Started deploy [integration/docroot@2a51016]: build: update dependencies + eslint fix in comment. f021d3f..2a51016 * 08:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93315 and previous config saved to /var/cache/conftool/dbconfig/20260528-080309-fceratto.json * 07:56 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93314 and previous config saved to /var/cache/conftool/dbconfig/20260528-075631-fceratto.json * 07:56 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020,1022-1023].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 07:56 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance * 07:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93313 and previous config saved to /var/cache/conftool/dbconfig/20260528-075521-fceratto.json * 07:47 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab replica * 07:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93311 and previous config saved to /var/cache/conftool/dbconfig/20260528-074513-fceratto.json * 07:37 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2189: repool after crash * 07:36 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab replica * 07:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93309 and previous config saved to /var/cache/conftool/dbconfig/20260528-073506-fceratto.json * 07:34 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica * 07:29 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294808{{!}}Don't run the click intent experiment on mobile (T426743)]] (duration: 06m 29s) * 07:25 wmde-fisch@deploy1003: thiemowmde, wmde-fisch: Continuing with deployment * 07:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93308 and previous config saved to /var/cache/conftool/dbconfig/20260528-072458-fceratto.json * 07:24 wmde-fisch@deploy1003: thiemowmde, wmde-fisch: Backport for [[gerrit:1294808{{!}}Don't run the click intent experiment on mobile (T426743)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:24 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica * 07:23 tgr@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwikisource --logwiki=metawiki Ioed Renamed_user_4232d41570b9e8f46ef150e5e360e446 # [[phab:T427459|T427459]] * 07:22 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1294808{{!}}Don't run the click intent experiment on mobile (T426743)]] * 07:20 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1270986{{!}}Update wikimania wordmark for 2026 (T413331)]] (duration: 06m 54s) * 07:18 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93307 and previous config saved to /var/cache/conftool/dbconfig/20260528-071836-fceratto.json * 07:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1264.eqiad.wmnet with reason: Maintenance * 07:16 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1167: Reboot completed * 07:16 wmde-fisch@deploy1003: wmde-fisch, robertsky: Continuing with deployment * 07:15 wmde-fisch@deploy1003: wmde-fisch, robertsky: Backport for [[gerrit:1270986{{!}}Update wikimania wordmark for 2026 (T413331)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:13 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1270986{{!}}Update wikimania wordmark for 2026 (T413331)]] * 07:11 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289898{{!}}Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] (duration: 07m 15s) * 07:07 wmde-fisch@deploy1003: wmde-fisch, arthurtaylor: Continuing with deployment * 07:06 wmde-fisch@deploy1003: wmde-fisch, arthurtaylor: Backport for [[gerrit:1289898{{!}}Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:04 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1289898{{!}}Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] * 06:43 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1167: Reboot completed * 06:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93303 and previous config saved to /var/cache/conftool/dbconfig/20260528-064217-fceratto.json * 06:33 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93302 and previous config saved to /var/cache/conftool/dbconfig/20260528-063357-fceratto.json * 06:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 06:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance * 06:25 hashar: Restarting CI Jenkins for plugins upgrades * 06:16 fceratto@dns1005: END - running authdns-update * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1209 [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93301 and previous config saved to /var/cache/conftool/dbconfig/20260528-061609-fceratto.json * 06:14 fceratto@dns1005: START - running authdns-update * 06:11 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1193 to s8 primary and set section read-write [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93300 and previous config saved to /var/cache/conftool/dbconfig/20260528-061138-fceratto.json * 06:10 fceratto@cumin1003: dbctl commit (dc=all): 'Set s8 eqiad as read-only for maintenance - [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93299 and previous config saved to /var/cache/conftool/dbconfig/20260528-061048-fceratto.json * 06:10 federico3: Starting s8 eqiad failover from db1209 to db1193 - [[phab:T426095|T426095]] * 06:04 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1193 with weight 0 [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93298 and previous config saved to /var/cache/conftool/dbconfig/20260528-060412-fceratto.json * 06:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s8 [[phab:T426095|T426095]] * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 41s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:53 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 00:53 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new subnet in eqsin - pt1979@cumin2002" * 00:53 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new subnet in eqsin - pt1979@cumin2002" * 00:49 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 00:25 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294470{{!}}Activate conductwiki (T426984)]] (duration: 07m 12s) * 00:21 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 00:20 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1294470{{!}}Activate conductwiki (T426984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 00:18 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1294470{{!}}Activate conductwiki (T426984)]] * 00:12 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294438{{!}}Init conductwiki (T426984)]] (duration: 07m 25s) * 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 00:08 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 00:06 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1294438{{!}}Init conductwiki (T426984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 00:04 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1294438{{!}}Init conductwiki (T426984)]] * 00:04 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] == 2026-05-27 == * 23:13 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294432{{!}}Exclude more content from selection (T426308)]], [[gerrit:1285523{{!}}Remove MinervaNightMode config after skin cleanup (T426689)]] (duration: 08m 42s) * 23:09 jdlrobson@deploy1003: jdlrobson, h2o, egardner: Continuing with deployment * 23:06 jdlrobson@deploy1003: jdlrobson, h2o, egardner: Backport for [[gerrit:1294432{{!}}Exclude more content from selection (T426308)]], [[gerrit:1285523{{!}}Remove MinervaNightMode config after skin cleanup (T426689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:04 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294432{{!}}Exclude more content from selection (T426308)]], [[gerrit:1285523{{!}}Remove MinervaNightMode config after skin cleanup (T426689)]] * 22:58 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294435{{!}}passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] (duration: 07m 49s) * 22:55 ladsgroup@cumin1003: END (PASS) - Cookbook sre.mysql.sanitarium_restart (exit_code=0) * 22:54 catrope@deploy1003: catrope: Continuing with deployment * 22:52 catrope@deploy1003: catrope: Backport for [[gerrit:1294435{{!}}passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:50 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1294435{{!}}passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] * 22:46 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294360{{!}}Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322{{!}}Thumbnails are not being optimized in large mode (T427237)]] (duration: 06m 54s) * 22:42 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 22:41 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1294360{{!}}Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322{{!}}Thumbnails are not being optimized in large mode (T427237)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:40 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart * 22:40 ladsgroup@cumin1003: END (FAIL) - Cookbook sre.mysql.sanitarium_restart (exit_code=99) * 22:40 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart * 22:39 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294360{{!}}Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322{{!}}Thumbnails are not being optimized in large mode (T427237)]] * 22:39 ladsgroup@deploy1003: Finished scap sync-world: Add conduct.wikimedia.org ([[phab:T426984|T426984]]) (duration: 07m 16s) * 22:35 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 22:34 ladsgroup@deploy1003: ladsgroup: Add conduct.wikimedia.org ([[phab:T426984|T426984]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:33 ladsgroup@deploy1003: Started scap sync-world: Add conduct.wikimedia.org ([[phab:T426984|T426984]]) * 22:13 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294370{{!}}Carousel only on articles (T427336)]] (duration: 10m 00s) * 22:09 egardner@deploy1003: egardner: Continuing with deployment * 22:05 egardner@deploy1003: egardner: Backport for [[gerrit:1294370{{!}}Carousel only on articles (T427336)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:03 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1294370{{!}}Carousel only on articles (T427336)]] * 21:37 bking@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 15 days, 0:00:00 on relforge[1008-1010].eqiad.wmnet with reason: non-production environment * 21:20 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 21:20 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 21:20 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 21:19 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 21:04 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1288370{{!}}Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819{{!}}Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] (duration: 07m 38s) * 20:59 ebernhardson@deploy1003: matmarex, ebernhardson, pppery: Continuing with deployment * 20:58 ebernhardson@deploy1003: matmarex, ebernhardson, pppery: Backport for [[gerrit:1288370{{!}}Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819{{!}}Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:56 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1288370{{!}}Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819{{!}}Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] * 20:51 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294373{{!}}identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374{{!}}Revert^2 "cirrus: AB test query suggester variants" (T407432)]] (duration: 07m 30s) * 20:47 ebernhardson@deploy1003: ebernhardson: Continuing with deployment * 20:46 ebernhardson@deploy1003: ebernhardson: Backport for [[gerrit:1294373{{!}}identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374{{!}}Revert^2 "cirrus: AB test query suggester variants" (T407432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:44 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294373{{!}}identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374{{!}}Revert^2 "cirrus: AB test query suggester variants" (T407432)]] * 20:43 swfrench-wmf: reprepro include dh-php_5.5+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 20:39 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts lvs1016.eqiad.wmnet * 20:39 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:39 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brett@cumin2002" * 20:38 swfrench-wmf: reprepro include php-defaults_94+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 20:37 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brett@cumin2002" * 20:31 brett@cumin2002: START - Cookbook sre.dns.netbox * 20:27 swfrench-wmf: reprepro include php8.3_8.3.31-1+wmf12u2 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 20:25 brett@cumin2002: START - Cookbook sre.hosts.decommission for hosts lvs1016.eqiad.wmnet * 20:25 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294342{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344{{!}}frwiki: restrict Article Guidance experiment to junior editors (T426871)]] (duration: 08m 11s) * 20:21 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host lvs1016.eqiad.wmnet with OS bullseye * 20:21 sbisson@deploy1003: sbisson: Continuing with deployment * 20:20 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1020.eqiad.wmnet * 20:19 sbisson@deploy1003: sbisson: Backport for [[gerrit:1294342{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344{{!}}frwiki: restrict Article Guidance experiment to junior editors (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be v * 20:17 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294342{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344{{!}}frwiki: restrict Article Guidance experiment to junior editors (T426871)]] * 20:14 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet * 20:05 cmooney@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 12355 * 20:04 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 12355 * 19:51 brett@cumin2002: START - Cookbook sre.hosts.reimage for host lvs1016.eqiad.wmnet with OS bullseye * 19:48 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 19:45 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 19:45 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 19:32 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp6016.drmrs.wmnet,cp[1112,1114].eqiad.wmnet,cp[5024,5031-5032].eqsin.wmnet<nowiki>}</nowiki> and A:cp * 19:32 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5032.eqsin.wmnet * 19:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 19:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 19:01 joal@deploy1003: Finished deploy [analytics/refinery@96cf761] (thin): Regular analytics weekly train THIN [analytics/refinery@96cf761f] (duration: 02m 08s) * 18:59 joal@deploy1003: Started deploy [analytics/refinery@96cf761] (thin): Regular analytics weekly train THIN [analytics/refinery@96cf761f] * 18:58 joal@deploy1003: Finished deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] (duration: 05m 01s) * 18:53 joal@deploy1003: Started deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] * 18:53 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294376{{!}}Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375{{!}}auth: Mark the hidden token field used for reauth as skippable (T427398)]] (duration: 07m 41s) * 18:49 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5031.eqsin.wmnet * 18:49 catrope@deploy1003: catrope: Continuing with deployment * 18:47 catrope@deploy1003: catrope: Backport for [[gerrit:1294376{{!}}Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375{{!}}auth: Mark the hidden token field used for reauth as skippable (T427398)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:45 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1294376{{!}}Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375{{!}}auth: Mark the hidden token field used for reauth as skippable (T427398)]] * 18:40 joal@deploy1003: Finished deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] (duration: 01m 05s) * 18:39 joal@deploy1003: Started deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] * 18:37 joal@deploy1003: Finished deploy [analytics/refinery@96cf761] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96cf761f] (duration: 02m 04s) * 18:35 joal@deploy1003: Started deploy [analytics/refinery@96cf761] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96cf761f] * 18:29 swfrench@deploy1003: Finished scap sync-world: Helmfile-only deployment to clean up unused mesh listeners (duration: 06m 12s) * 18:25 swfrench@deploy1003: swfrench: Continuing with deployment * 18:24 swfrench@deploy1003: swfrench: Helmfile-only deployment to clean up unused mesh listeners synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:23 swfrench@deploy1003: Started scap sync-world: Helmfile-only deployment to clean up unused mesh listeners * 18:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93296 and previous config saved to /var/cache/conftool/dbconfig/20260527-181923-fceratto.json * 18:13 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 18:12 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 18:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 18:11 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 18:11 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 18:10 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 18:10 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 18:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93295 and previous config saved to /var/cache/conftool/dbconfig/20260527-180915-fceratto.json * 18:09 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 18:09 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293776{{!}}ProductionServices: Revert to discovery shellbox listeners]] (duration: 10m 24s) * 18:08 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1017.eqiad.wmnet * 18:08 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1017.eqiad.wmnet * 18:07 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5024.eqsin.wmnet * 18:03 swfrench@deploy1003: swfrench: Continuing with deployment * 18:02 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 18:02 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 18:02 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 18:01 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:01 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 18:01 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 18:00 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 18:00 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293776{{!}}ProductionServices: Revert to discovery shellbox listeners]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:00 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93294 and previous config saved to /var/cache/conftool/dbconfig/20260527-175908-fceratto.json * 17:58 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293776{{!}}ProductionServices: Revert to discovery shellbox listeners]] * 17:55 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93293 and previous config saved to /var/cache/conftool/dbconfig/20260527-174900-fceratto.json * 17:43 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293774{{!}}ProductionServices: Temporarily use shellbox in codfw]] (duration: 15m 01s) * 17:38 swfrench@deploy1003: swfrench: Continuing with deployment * 17:31 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293774{{!}}ProductionServices: Temporarily use shellbox in codfw]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:28 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293774{{!}}ProductionServices: Temporarily use shellbox in codfw]] * 17:25 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp1114.eqiad.wmnet * 17:18 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:15 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:15 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:14 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:05 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293775{{!}}ProductionServices: Temporarily use shellbox in eqiad]] (duration: 08m 44s) * 17:00 swfrench@deploy1003: swfrench: Continuing with deployment * 16:58 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293775{{!}}ProductionServices: Temporarily use shellbox in eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:56 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293775{{!}}ProductionServices: Temporarily use shellbox in eqiad]] * 16:53 atsuko@dns1004: END - running authdns-update * 16:51 atsuko@dns1004: START - running authdns-update * 16:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93292 and previous config saved to /var/cache/conftool/dbconfig/20260527-164846-fceratto.json * 16:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1264.eqiad.wmnet with reason: Maintenance * 16:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93291 and previous config saved to /var/cache/conftool/dbconfig/20260527-164815-fceratto.json * 16:43 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp1112.eqiad.wmnet * 16:41 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1017.eqiad.wmnet with reason: Setting up * 16:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P93290 and previous config saved to /var/cache/conftool/dbconfig/20260527-163808-fceratto.json * 16:37 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2163: Repooling after testing patch * 16:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P93287 and previous config saved to /var/cache/conftool/dbconfig/20260527-162800-fceratto.json * 16:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93285 and previous config saved to /var/cache/conftool/dbconfig/20260527-161753-fceratto.json * 16:14 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 16:13 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 16:13 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 16:12 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 16:11 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93284 and previous config saved to /var/cache/conftool/dbconfig/20260527-161101-fceratto.json * 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: Maintenance * 16:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93283 and previous config saved to /var/cache/conftool/dbconfig/20260527-161034-fceratto.json * 16:10 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 16:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1178: Recovering from failure in cookbook * 16:10 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 16:05 sukhe@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host durum5003.eqsin.wmnet with OS trixie * 16:03 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp6016.drmrs.wmnet * 16:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220', diff saved to https://phabricator.wikimedia.org/P93280 and previous config saved to /var/cache/conftool/dbconfig/20260527-160027-fceratto.json * 15:59 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1017.eqiad.wmnet * 15:53 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2163.codfw.wmnet * 15:53 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2163.codfw.wmnet * 15:52 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs1017.eqiad.wmnet * 15:52 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Repooling after testing patch * 15:52 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp6016.drmrs.wmnet,cp[1112,1114].eqiad.wmnet,cp[5024,5031-5032].eqsin.wmnet<nowiki>}</nowiki> and A:cp * 15:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2163: Testing cookbook * 15:50 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2163: Testing cookbook * 15:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220', diff saved to https://phabricator.wikimedia.org/P93276 and previous config saved to /var/cache/conftool/dbconfig/20260527-155019-fceratto.json * 15:45 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:45 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93274 and previous config saved to /var/cache/conftool/dbconfig/20260527-154011-fceratto.json * 15:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 15:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2163: Migration of db2163.codfw.wmnet completed * 15:32 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Migration of db2163.codfw.wmnet completed * 15:32 cwilliams@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2163: Migration of db2163.codfw.wmnet completed * 15:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1178: Recovering from failure in cookbook * 15:22 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1178.eqiad.wmnet * 15:22 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1178.eqiad.wmnet * 15:19 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 15:19 cdanis: 💙cdanis@cp4047.ulsfo.wmnet ~ 🕦☕ sudo apt install lua5.4-ciderbloom lua5.4-ciderbloom-dbgsym * 15:13 cdanis: 💙cdanis@cp5026.eqsin.wmnet ~ 🕚☕ sudo apt install lua5.4-ciderbloom lua5.4-ciderbloom-dbgsym * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:11 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:11 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:11 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Icinga wait failed during run * 15:10 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:10 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:10 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:09 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:09 cdanis: 💔cdanis@apt1002.wikimedia.org ~ 🕚☕ sudo -i reprepro --component main --restrict cidergrinder update trixie-wikimedia * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:05 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93268 and previous config saved to /var/cache/conftool/dbconfig/20260527-150508-fceratto.json * 15:05 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1220.eqiad.wmnet with reason: Maintenance * 15:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93267 and previous config saved to /var/cache/conftool/dbconfig/20260527-150438-fceratto.json * 14:59 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Migration of db2163.codfw.wmnet completed * 14:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P93264 and previous config saved to /var/cache/conftool/dbconfig/20260527-145430-fceratto.json * 14:54 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 14:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2163.codfw.wmnet with OS trixie * 14:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply * 14:50 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply * 14:46 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290926{{!}}Re-enable ReadingLists QuickSurvey (T426781)]] (duration: 08m 32s) * 14:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1178.eqiad.wmnet with OS trixie * 14:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P93263 and previous config saved to /var/cache/conftool/dbconfig/20260527-144423-fceratto.json * 14:42 aude@deploy1003: aude: Continuing with deployment * 14:40 aude@deploy1003: aude: Backport for [[gerrit:1290926{{!}}Re-enable ReadingLists QuickSurvey (T426781)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:38 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db2189.codfw.wmnet with reason: crashed [[phab:T427376|T427376]] * 14:38 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1290926{{!}}Re-enable ReadingLists QuickSurvey (T426781)]] * 14:35 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290924{{!}}Make logging of title and page ID optional (T426457)]] (duration: 11m 30s) * 14:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93262 and previous config saved to /var/cache/conftool/dbconfig/20260527-143416-fceratto.json * 14:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2163.codfw.wmnet with reason: host reimage * 14:29 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2163.codfw.wmnet with reason: host reimage * 14:29 aude@deploy1003: aude: Continuing with deployment * 14:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1178.eqiad.wmnet with reason: host reimage * 14:27 aude@deploy1003: aude: Backport for [[gerrit:1290924{{!}}Make logging of title and page ID optional (T426457)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:27 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93260 and previous config saved to /var/cache/conftool/dbconfig/20260527-142659-fceratto.json * 14:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1179.eqiad.wmnet with reason: Maintenance * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:23 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1290924{{!}}Make logging of title and page ID optional (T426457)]] * 14:22 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1178.eqiad.wmnet with reason: host reimage * 14:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1033.eqiad.wmnet with reason: Maintenance * 14:18 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294247{{!}}Update Direct Reporting email (T427358)]] (duration: 33m 01s) * 14:10 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2163.codfw.wmnet with OS trixie * 14:09 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1178.eqiad.wmnet with OS trixie * 14:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2163: Upgrading db2163.codfw.wmnet * 14:08 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2163: Upgrading db2163.codfw.wmnet * 14:08 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1178: Upgrading db1178.eqiad.wmnet * 14:07 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1178: Upgrading db1178.eqiad.wmnet * 14:06 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:06 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:06 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:06 stran@deploy1003: stran: Continuing with deployment * 14:02 stran@deploy1003: stran: Backport for [[gerrit:1294247{{!}}Update Direct Reporting email (T427358)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:56 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 13:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2164: Migration of db2164.codfw.wmnet completed * 13:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1192: Migration of db1192.eqiad.wmnet completed * 13:45 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1294247{{!}}Update Direct Reporting email (T427358)]] * 13:40 phuedx@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294217{{!}}ext.wikimediaEvents: Add hoisting error detection test (T427092)]] (duration: 11m 35s) * 13:36 phuedx@deploy1003: phuedx: Continuing with deployment * 13:30 phuedx@deploy1003: phuedx: Backport for [[gerrit:1294217{{!}}ext.wikimediaEvents: Add hoisting error detection test (T427092)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:28 phuedx@deploy1003: Started scap sync-world: Backport for [[gerrit:1294217{{!}}ext.wikimediaEvents: Add hoisting error detection test (T427092)]] * 13:21 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290781{{!}}mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264{{!}}MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] (duration: 13m 23s) * 13:15 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2189: Test * 13:15 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2189: Test * 13:15 mlitn@deploy1003: krinkle, mlitn: Continuing with deployment * 13:13 mlitn@deploy1003: krinkle, mlitn: Backport for [[gerrit:1290781{{!}}mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264{{!}}MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 13:10 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2164: Migration of db2164.codfw.wmnet completed * 13:08 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1290781{{!}}mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264{{!}}MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] * 13:06 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 13:05 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db2212.codfw.wmnet with reason: failed to reboot [[phab:T427388|T427388]] [[phab:T426633|T426633]] * 13:05 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1192: Migration of db1192.eqiad.wmnet completed * 13:01 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2164.codfw.wmnet with OS trixie * 12:57 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1192.eqiad.wmnet with OS trixie * 12:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2164.codfw.wmnet with reason: host reimage * 12:40 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1192.eqiad.wmnet with reason: host reimage * 12:40 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2164.codfw.wmnet with reason: host reimage * 12:35 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1192.eqiad.wmnet with reason: host reimage * 12:28 Amir1: deleting binlogs older than a year * 12:22 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2164.codfw.wmnet with OS trixie * 12:21 cmooney@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 36692 * 12:21 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1192.eqiad.wmnet with OS trixie * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1077 * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1080 * 12:20 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1077 * 12:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2164: Upgrading db2164.codfw.wmnet * 12:20 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 36692 * 12:20 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1080 * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1078 * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1079 * 12:20 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2164: Upgrading db2164.codfw.wmnet * 12:19 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:19 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1079 * 12:19 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1078 * 12:19 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:19 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1192: Upgrading db1192.eqiad.wmnet * 12:19 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:18 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1192: Upgrading db1192.eqiad.wmnet * 12:18 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:15 jclark@cumin1003: START - Cookbook sre.dns.netbox * 12:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2165: Migration of db2165.codfw.wmnet completed * 12:14 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:14 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:14 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:12 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db2189: Test * 12:11 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2189: Test * 12:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1193: Migration of db1193.eqiad.wmnet completed * 12:09 jclark@cumin1003: START - Cookbook sre.dns.netbox * 12:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93243 and previous config saved to /var/cache/conftool/dbconfig/20260527-120452-fceratto.json * 12:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: Maintenance * 12:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93242 and previous config saved to /var/cache/conftool/dbconfig/20260527-120205-fceratto.json * 12:01 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 11:58 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 11:58 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "is everything alright? /cc effie - ayounsi@cumin1003" * 11:58 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "is everything alright? /cc effie - ayounsi@cumin1003" * 11:56 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 11:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P93239 and previous config saved to /var/cache/conftool/dbconfig/20260527-115157-fceratto.json * 11:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P93237 and previous config saved to /var/cache/conftool/dbconfig/20260527-114149-fceratto.json * 11:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93235 and previous config saved to /var/cache/conftool/dbconfig/20260527-113142-fceratto.json * 11:29 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2165: Migration of db2165.codfw.wmnet completed * 11:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1193: Migration of db1193.eqiad.wmnet completed * 11:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93231 and previous config saved to /var/cache/conftool/dbconfig/20260527-112327-fceratto.json * 11:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2188.codfw.wmnet with reason: Maintenance * 11:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93230 and previous config saved to /var/cache/conftool/dbconfig/20260527-112257-fceratto.json * 11:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2165.codfw.wmnet with OS trixie * 11:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1193.eqiad.wmnet with OS trixie * 11:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P93229 and previous config saved to /var/cache/conftool/dbconfig/20260527-111250-fceratto.json * 11:10 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:10 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:08 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:08 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:02 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P93227 and previous config saved to /var/cache/conftool/dbconfig/20260527-110242-fceratto.json * 11:02 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:02 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 11:01 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 11:01 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2165.codfw.wmnet with reason: host reimage * 11:00 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2189', diff saved to https://phabricator.wikimedia.org/P93226 and previous config saved to /var/cache/conftool/dbconfig/20260527-110016-marostegui.json * 10:58 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1193.eqiad.wmnet with reason: host reimage * 10:57 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2165.codfw.wmnet with reason: host reimage * 10:56 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 10:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93225 and previous config saved to /var/cache/conftool/dbconfig/20260527-105235-fceratto.json * 10:52 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1193.eqiad.wmnet with reason: host reimage * 10:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1050: repool after maintenance * 10:45 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2176 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93223 and previous config saved to /var/cache/conftool/dbconfig/20260527-104518-fceratto.json * 10:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2176.codfw.wmnet with reason: Maintenance * 10:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93222 and previous config saved to /var/cache/conftool/dbconfig/20260527-104449-fceratto.json * 10:39 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2165.codfw.wmnet with OS trixie * 10:38 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1193.eqiad.wmnet with OS trixie * 10:36 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1193: Upgrading db1193.eqiad.wmnet * 10:35 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1193: Upgrading db1193.eqiad.wmnet * 10:35 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2165: Upgrading db2165.codfw.wmnet * 10:35 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2165: Upgrading db2165.codfw.wmnet * 10:34 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P93218 and previous config saved to /var/cache/conftool/dbconfig/20260527-103441-fceratto.json * 10:29 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:29 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P93217 and previous config saved to /var/cache/conftool/dbconfig/20260527-102434-fceratto.json * 10:22 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:21 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93215 and previous config saved to /var/cache/conftool/dbconfig/20260527-101426-fceratto.json * 10:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1203: Migration of db1203.eqiad.wmnet completed * 10:10 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2166: Migration of db2166.codfw.wmnet completed * 10:08 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93212 and previous config saved to /var/cache/conftool/dbconfig/20260527-100701-fceratto.json * 10:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance * 10:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93211 and previous config saved to /var/cache/conftool/dbconfig/20260527-100632-fceratto.json * 10:05 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1050: repool after maintenance * 10:04 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:02 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1050.eqiad.wmnet with OS trixie * 09:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P93208 and previous config saved to /var/cache/conftool/dbconfig/20260527-095624-fceratto.json * 09:47 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 09:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P93206 and previous config saved to /var/cache/conftool/dbconfig/20260527-094616-fceratto.json * 09:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1050.eqiad.wmnet with reason: host reimage * 09:43 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 09:41 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1050.eqiad.wmnet with reason: host reimage * 09:38 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 09:38 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 09:37 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 09:37 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 09:36 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 09:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93203 and previous config saved to /var/cache/conftool/dbconfig/20260527-093609-fceratto.json * 09:34 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 09:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93202 and previous config saved to /var/cache/conftool/dbconfig/20260527-092842-fceratto.json * 09:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance * 09:28 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1203: Migration of db1203.eqiad.wmnet completed * 09:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93200 and previous config saved to /var/cache/conftool/dbconfig/20260527-092814-fceratto.json * 09:27 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1050.eqiad.wmnet with OS trixie * 09:26 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1050: Upgrading es1050.eqiad.wmnet * 09:25 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1050: Upgrading es1050.eqiad.wmnet * 09:25 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1050: repool after maintenance * 09:25 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1050: repool after maintenance * 09:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2166: Migration of db2166.codfw.wmnet completed * 09:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2051: repool after maintenance * 09:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1203.eqiad.wmnet with OS trixie * 09:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P93196 and previous config saved to /var/cache/conftool/dbconfig/20260527-091806-fceratto.json * 09:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2166.codfw.wmnet with OS trixie * 09:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P93194 and previous config saved to /var/cache/conftool/dbconfig/20260527-090759-fceratto.json * 09:03 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp3074.* * 09:03 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp3066.* * 09:03 fabfur: repooling cp3074 and cp3066 ([[phab:T419825|T419825]]) * 09:02 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp6015.drmrs.wmnet * 09:02 slyngshede@cumin1003: START - Cookbook sre.hosts.remove-downtime for cp6015.drmrs.wmnet * 09:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1203.eqiad.wmnet with reason: host reimage * 09:02 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: name=cp6015.* * 08:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2166.codfw.wmnet with reason: host reimage * 08:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93193 and previous config saved to /var/cache/conftool/dbconfig/20260527-085751-fceratto.json * 08:55 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1203.eqiad.wmnet with reason: host reimage * 08:54 Emperor: restart swift on ms-fe2011 [[phab:T360913|T360913]] * 08:54 jayme@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:54 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2166.codfw.wmnet with reason: host reimage * 08:54 jayme@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 08:51 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 08:51 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 08:51 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp3066.* * 08:51 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp3074.* * 08:51 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 08:50 fabfur: depooling and installing haproxy-awslc on cp3074 and cp3066 ([[phab:T419825|T419825]]) * 08:50 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93191 and previous config saved to /var/cache/conftool/dbconfig/20260527-085024-fceratto.json * 08:50 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance * 08:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93190 and previous config saved to /var/cache/conftool/dbconfig/20260527-085005-fceratto.json * 08:41 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1203.eqiad.wmnet with OS trixie * 08:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P93189 and previous config saved to /var/cache/conftool/dbconfig/20260527-083957-fceratto.json * 08:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2051: repool after maintenance * 08:37 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 08:36 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1203: Upgrading db1203.eqiad.wmnet * 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1004.wikimedia.org * 08:36 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1203: Upgrading db1203.eqiad.wmnet * 08:36 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:35 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2166.codfw.wmnet with OS trixie * 08:35 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2051.codfw.wmnet with OS trixie * 08:34 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2166: Upgrading db2166.codfw.wmnet * 08:33 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2166: Upgrading db2166.codfw.wmnet * 08:33 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader1004.wikimedia.org * 08:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org * 08:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P93185 and previous config saved to /var/cache/conftool/dbconfig/20260527-082950-fceratto.json * 08:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org * 08:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93184 and previous config saved to /var/cache/conftool/dbconfig/20260527-081942-fceratto.json * 08:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2051.codfw.wmnet with reason: host reimage * 08:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2051.codfw.wmnet with reason: host reimage * 08:11 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 08:11 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2153 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93183 and previous config saved to /var/cache/conftool/dbconfig/20260527-081112-fceratto.json * 08:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance * 08:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93182 and previous config saved to /var/cache/conftool/dbconfig/20260527-081054-fceratto.json * 08:07 jmm@dns1004: END - running authdns-update * 08:05 jmm@dns1004: START - running authdns-update * 08:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248', diff saved to https://phabricator.wikimedia.org/P93181 and previous config saved to /var/cache/conftool/dbconfig/20260527-080046-fceratto.json * 07:59 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2051.codfw.wmnet with OS trixie * 07:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248', diff saved to https://phabricator.wikimedia.org/P93180 and previous config saved to /var/cache/conftool/dbconfig/20260527-075039-fceratto.json * 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1026.eqiad.wmnet * 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1026.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:43 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1026.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2051: Upgrading es2051.codfw.wmnet * 07:42 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2051: Upgrading es2051.codfw.wmnet * 07:41 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93178 and previous config saved to /var/cache/conftool/dbconfig/20260527-074031-fceratto.json * 07:40 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294125{{!}}Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126{{!}}Add script to demote ineligible members of restricted global groups (T425395)]] (duration: 06m 42s) * 07:36 mszwarc@deploy1003: mszwarc: Continuing with deployment * 07:35 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1294125{{!}}Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126{{!}}Add script to demote ineligible members of restricted global groups (T425395)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93177 and previous config saved to /var/cache/conftool/dbconfig/20260527-073504-fceratto.json * 07:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2248.codfw.wmnet with reason: Maintenance * 07:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93176 and previous config saved to /var/cache/conftool/dbconfig/20260527-073434-fceratto.json * 07:33 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1294125{{!}}Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126{{!}}Add script to demote ineligible members of restricted global groups (T425395)]] * 07:28 jmm@cumin2002: START - Cookbook sre.dns.netbox * 07:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P93175 and previous config saved to /var/cache/conftool/dbconfig/20260527-072426-fceratto.json * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) * 07:23 marostegui@cumin1003: Removing pc1014 from zarcillo [[phab:T427190|T427190]] * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1014.eqiad.wmnet * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 07:23 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 07:18 marostegui@cumin1003: START - Cookbook sre.dns.netbox * 07:15 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1026.eqiad.wmnet * 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1025.eqiad.wmnet * 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P93174 and previous config saved to /var/cache/conftool/dbconfig/20260527-071418-fceratto.json * 07:13 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1014.eqiad.wmnet * 07:13 marostegui@cumin1003: START - Cookbook sre.mysql.decommission * 07:13 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org * 07:07 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2055: repool after maintenance * 07:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org * 07:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1003.wikimedia.org * 07:06 jmm@cumin2002: START - Cookbook sre.dns.netbox * 07:06 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1190.eqiad.wmnet with reason: Maintenance on db1190 * 07:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93172 and previous config saved to /var/cache/conftool/dbconfig/20260527-070410-fceratto.json * 07:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader1003.wikimedia.org * 06:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93171 and previous config saved to /var/cache/conftool/dbconfig/20260527-065545-fceratto.json * 06:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2247.codfw.wmnet with reason: Maintenance * 06:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93170 and previous config saved to /var/cache/conftool/dbconfig/20260527-065526-fceratto.json * 06:54 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1025.eqiad.wmnet * 06:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P93168 and previous config saved to /var/cache/conftool/dbconfig/20260527-064519-fceratto.json * 06:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P93166 and previous config saved to /var/cache/conftool/dbconfig/20260527-063511-fceratto.json * 06:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93165 and previous config saved to /var/cache/conftool/dbconfig/20260527-062503-fceratto.json * 06:22 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2055: repool after maintenance * 06:21 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 06:21 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2055.codfw.wmnet with OS trixie * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2246 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93163 and previous config saved to /var/cache/conftool/dbconfig/20260527-061643-fceratto.json * 06:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2246.codfw.wmnet with reason: Maintenance * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93162 and previous config saved to /var/cache/conftool/dbconfig/20260527-061613-fceratto.json * 06:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P93161 and previous config saved to /var/cache/conftool/dbconfig/20260527-060606-fceratto.json * 06:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2055.codfw.wmnet with reason: host reimage * 05:56 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2055.codfw.wmnet with reason: host reimage * 05:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P93160 and previous config saved to /var/cache/conftool/dbconfig/20260527-055558-fceratto.json * 05:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93159 and previous config saved to /var/cache/conftool/dbconfig/20260527-054550-fceratto.json * 05:41 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2055.codfw.wmnet with OS trixie * 05:40 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2055: Upgrading es2055.codfw.wmnet * 05:40 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2055: Upgrading es2055.codfw.wmnet * 05:40 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:38 moritzm: remove ganeti1026 from eqiad Ganeti cluster [[phab:T424680|T424680]] * 05:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2245 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93157 and previous config saved to /var/cache/conftool/dbconfig/20260527-053727-fceratto.json * 05:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: Maintenance * 05:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93156 and previous config saved to /var/cache/conftool/dbconfig/20260527-053708-fceratto.json * 05:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P93155 and previous config saved to /var/cache/conftool/dbconfig/20260527-052700-fceratto.json * 05:26 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1014 from dbctl [[phab:T427270|T427270]]', diff saved to https://phabricator.wikimedia.org/P93154 and previous config saved to /var/cache/conftool/dbconfig/20260527-052624-marostegui.json * 05:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P93153 and previous config saved to /var/cache/conftool/dbconfig/20260527-051653-fceratto.json * 05:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93152 and previous config saved to /var/cache/conftool/dbconfig/20260527-050645-fceratto.json * 04:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2237 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93151 and previous config saved to /var/cache/conftool/dbconfig/20260527-045827-fceratto.json * 04:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: Maintenance * 04:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93150 and previous config saved to /var/cache/conftool/dbconfig/20260527-045759-fceratto.json * 04:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P93149 and previous config saved to /var/cache/conftool/dbconfig/20260527-044751-fceratto.json * 04:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P93148 and previous config saved to /var/cache/conftool/dbconfig/20260527-043744-fceratto.json * 04:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93147 and previous config saved to /var/cache/conftool/dbconfig/20260527-042737-fceratto.json * 04:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2236 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93146 and previous config saved to /var/cache/conftool/dbconfig/20260527-041921-fceratto.json * 04:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2236.codfw.wmnet with reason: Maintenance * 04:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93145 and previous config saved to /var/cache/conftool/dbconfig/20260527-041852-fceratto.json * 04:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P93144 and previous config saved to /var/cache/conftool/dbconfig/20260527-040844-fceratto.json * 03:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P93143 and previous config saved to /var/cache/conftool/dbconfig/20260527-035836-fceratto.json * 03:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93142 and previous config saved to /var/cache/conftool/dbconfig/20260527-034828-fceratto.json * 03:40 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93141 and previous config saved to /var/cache/conftool/dbconfig/20260527-034008-fceratto.json * 03:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2219.codfw.wmnet with reason: Maintenance * 03:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93140 and previous config saved to /var/cache/conftool/dbconfig/20260527-033938-fceratto.json * 03:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P93139 and previous config saved to /var/cache/conftool/dbconfig/20260527-032931-fceratto.json * 03:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P93138 and previous config saved to /var/cache/conftool/dbconfig/20260527-031923-fceratto.json * 03:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93137 and previous config saved to /var/cache/conftool/dbconfig/20260527-030915-fceratto.json * 03:00 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93136 and previous config saved to /var/cache/conftool/dbconfig/20260527-030045-fceratto.json * 03:00 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2210.codfw.wmnet with reason: Maintenance * 03:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93135 and previous config saved to /var/cache/conftool/dbconfig/20260527-030016-fceratto.json * 02:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P93134 and previous config saved to /var/cache/conftool/dbconfig/20260527-025008-fceratto.json * 02:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P93133 and previous config saved to /var/cache/conftool/dbconfig/20260527-024000-fceratto.json * 02:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93132 and previous config saved to /var/cache/conftool/dbconfig/20260527-022953-fceratto.json * 02:21 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93131 and previous config saved to /var/cache/conftool/dbconfig/20260527-022133-fceratto.json * 02:21 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2206.codfw.wmnet with reason: Maintenance * 02:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93130 and previous config saved to /var/cache/conftool/dbconfig/20260527-022100-fceratto.json * 02:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P93129 and previous config saved to /var/cache/conftool/dbconfig/20260527-021053-fceratto.json * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 29s) * 02:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P93128 and previous config saved to /var/cache/conftool/dbconfig/20260527-020045-fceratto.json * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93127 and previous config saved to /var/cache/conftool/dbconfig/20260527-015037-fceratto.json * 01:42 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93126 and previous config saved to /var/cache/conftool/dbconfig/20260527-014204-fceratto.json * 01:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2179.codfw.wmnet with reason: Maintenance * 01:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93125 and previous config saved to /var/cache/conftool/dbconfig/20260527-014134-fceratto.json * 01:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P93124 and previous config saved to /var/cache/conftool/dbconfig/20260527-013126-fceratto.json * 01:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P93123 and previous config saved to /var/cache/conftool/dbconfig/20260527-012119-fceratto.json * 01:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93122 and previous config saved to /var/cache/conftool/dbconfig/20260527-011111-fceratto.json * 01:02 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2172 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93121 and previous config saved to /var/cache/conftool/dbconfig/20260527-010234-fceratto.json * 01:02 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance * 01:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93120 and previous config saved to /var/cache/conftool/dbconfig/20260527-010205-fceratto.json * 00:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P93119 and previous config saved to /var/cache/conftool/dbconfig/20260527-005157-fceratto.json * 00:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P93118 and previous config saved to /var/cache/conftool/dbconfig/20260527-004149-fceratto.json * 00:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93117 and previous config saved to /var/cache/conftool/dbconfig/20260527-003141-fceratto.json * 00:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2155 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93116 and previous config saved to /var/cache/conftool/dbconfig/20260527-002309-fceratto.json * 00:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance * 00:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93115 and previous config saved to /var/cache/conftool/dbconfig/20260527-002228-fceratto.json * 00:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P93114 and previous config saved to /var/cache/conftool/dbconfig/20260527-001220-fceratto.json * 00:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P93113 and previous config saved to /var/cache/conftool/dbconfig/20260527-000209-fceratto.json == 2026-05-26 == * 23:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93112 and previous config saved to /var/cache/conftool/dbconfig/20260526-235201-fceratto.json * 23:44 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93111 and previous config saved to /var/cache/conftool/dbconfig/20260526-234451-fceratto.json * 23:44 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance * 23:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93110 and previous config saved to /var/cache/conftool/dbconfig/20260526-234421-fceratto.json * 23:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P93109 and previous config saved to /var/cache/conftool/dbconfig/20260526-233414-fceratto.json * 23:27 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5026.* * 23:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P93108 and previous config saved to /var/cache/conftool/dbconfig/20260526-232406-fceratto.json * 23:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93107 and previous config saved to /var/cache/conftool/dbconfig/20260526-231358-fceratto.json * 23:07 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5026.* * 23:06 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93106 and previous config saved to /var/cache/conftool/dbconfig/20260526-230650-fceratto.json * 23:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance * 23:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93105 and previous config saved to /var/cache/conftool/dbconfig/20260526-230620-fceratto.json * 22:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P93104 and previous config saved to /var/cache/conftool/dbconfig/20260526-225612-fceratto.json * 22:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P93103 and previous config saved to /var/cache/conftool/dbconfig/20260526-224604-fceratto.json * 22:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93101 and previous config saved to /var/cache/conftool/dbconfig/20260526-223556-fceratto.json * 22:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93100 and previous config saved to /var/cache/conftool/dbconfig/20260526-222848-fceratto.json * 22:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance * 22:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93099 and previous config saved to /var/cache/conftool/dbconfig/20260526-222828-fceratto.json * 22:23 robh@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts cp6015.drmrs.wmnet * 22:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P93098 and previous config saved to /var/cache/conftool/dbconfig/20260526-221819-fceratto.json * 22:10 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1009.eqiad.wmnet with OS trixie * 22:08 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1008.eqiad.wmnet with OS trixie * 22:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P93097 and previous config saved to /var/cache/conftool/dbconfig/20260526-220811-fceratto.json * 22:04 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293701{{!}}MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] (duration: 09m 30s) * 22:03 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1009.eqiad.wmnet with reason: host reimage * 22:00 egardner@deploy1003: egardner, mfossati: Continuing with deployment * 21:59 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1008.eqiad.wmnet with reason: host reimage * 21:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93096 and previous config saved to /var/cache/conftool/dbconfig/20260526-215803-fceratto.json * 21:57 egardner@deploy1003: egardner, mfossati: Backport for [[gerrit:1293701{{!}}MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:56 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp6015.drmrs.wmnet * 21:56 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1010.eqiad.wmnet with OS trixie * 21:56 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp6015.drmrs.wmnet * 21:55 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1293701{{!}}MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] * 21:54 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1009.eqiad.wmnet with reason: host reimage * 21:51 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1008.eqiad.wmnet with reason: host reimage * 21:50 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93095 and previous config saved to /var/cache/conftool/dbconfig/20260526-215043-fceratto.json * 21:50 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance * 21:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93094 and previous config saved to /var/cache/conftool/dbconfig/20260526-215011-fceratto.json * 21:49 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1010.eqiad.wmnet with reason: host reimage * 21:47 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp6015.drmrs.wmnet * 21:44 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1009 * 21:44 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host relforge1009 * 21:43 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host relforge1009 * 21:43 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) relforge1009.eqiad.wmnet 120.48.64.10.in-addr.arpa 0.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:43 bking@cumin2002: START - Cookbook sre.dns.wipe-cache relforge1009.eqiad.wmnet 120.48.64.10.in-addr.arpa 0.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:43 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:42 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1009 - bking@cumin2002" * 21:42 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1010.eqiad.wmnet with reason: host reimage * 21:42 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1009 - bking@cumin2002" * 21:41 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1008 * 21:40 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host relforge1008 * 21:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P93093 and previous config saved to /var/cache/conftool/dbconfig/20260526-214003-fceratto.json * 21:36 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host relforge1008 * 21:36 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) relforge1008.eqiad.wmnet 100.32.64.10.in-addr.arpa 0.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:36 bking@cumin2002: START - Cookbook sre.dns.wipe-cache relforge1008.eqiad.wmnet 100.32.64.10.in-addr.arpa 0.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:36 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:36 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1008 - bking@cumin2002" * 21:36 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1008 - bking@cumin2002" * 21:35 bking@cumin2002: START - Cookbook sre.dns.netbox * 21:32 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1010 * 21:32 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1010 * 21:31 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1010.eqiad.wmnet with OS trixie * 21:31 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1009 * 21:30 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1009.eqiad.wmnet with OS trixie * 21:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P93092 and previous config saved to /var/cache/conftool/dbconfig/20260526-212955-fceratto.json * 21:29 bking@cumin2002: START - Cookbook sre.dns.netbox * 21:29 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1008 * 21:29 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1008.eqiad.wmnet with OS trixie * 21:27 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist "all.dblist - mediamoderation-continuous-scan.dblist - preinstall.dblist" extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` in tmux session - [[phab:T421688|T421688]] * 21:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93091 and previous config saved to /var/cache/conftool/dbconfig/20260526-211948-fceratto.json * 21:19 jhathaway: dmarc ingress test run mx-in1001 * 21:15 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_codfw and A:cp * 21:15 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2057.codfw.wmnet * 21:14 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_codfw and A:cp * 21:14 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2058.codfw.wmnet * 21:12 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2222 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93090 and previous config saved to /var/cache/conftool/dbconfig/20260526-211238-fceratto.json * 21:12 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance * 21:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93089 and previous config saved to /var/cache/conftool/dbconfig/20260526-211207-fceratto.json * 21:06 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 21:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P93088 and previous config saved to /var/cache/conftool/dbconfig/20260526-210159-fceratto.json * 20:55 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on phab2003.codfw.wmnet with reason: WIP * 20:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P93087 and previous config saved to /var/cache/conftool/dbconfig/20260526-205152-fceratto.json * 20:50 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:50 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 20:50 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 20:45 dzahn@cumin2002: START - Cookbook sre.dns.netbox * 20:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93086 and previous config saved to /var/cache/conftool/dbconfig/20260526-204143-fceratto.json * 20:38 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2055.codfw.wmnet * 20:34 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93085 and previous config saved to /var/cache/conftool/dbconfig/20260526-203430-fceratto.json * 20:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance * 20:34 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2056.codfw.wmnet * 20:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93084 and previous config saved to /var/cache/conftool/dbconfig/20260526-203357-fceratto.json * 20:32 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:32 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:32 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:31 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P93083 and previous config saved to /var/cache/conftool/dbconfig/20260526-202349-fceratto.json * 20:18 alexsanford@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293161{{!}}Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794{{!}}Re-enable ReadingLists survey on beta cluster (T426781)]] (duration: 09m 14s) * 20:14 alexsanford@deploy1003: alexsanford, aude: Continuing with deployment * 20:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P93082 and previous config saved to /var/cache/conftool/dbconfig/20260526-201341-fceratto.json * 20:11 alexsanford@deploy1003: alexsanford, aude: Backport for [[gerrit:1293161{{!}}Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794{{!}}Re-enable ReadingLists survey on beta cluster (T426781)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:09 alexsanford@deploy1003: Started scap sync-world: Backport for [[gerrit:1293161{{!}}Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794{{!}}Re-enable ReadingLists survey on beta cluster (T426781)]] * 20:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93081 and previous config saved to /var/cache/conftool/dbconfig/20260526-200333-fceratto.json * 19:59 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2053.codfw.wmnet * 19:58 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2029.codfw.wmnet with OS trixie * 19:57 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2028.codfw.wmnet with OS trixie * 19:56 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2208 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93080 and previous config saved to /var/cache/conftool/dbconfig/20260526-195632-fceratto.json * 19:56 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance * 19:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93079 and previous config saved to /var/cache/conftool/dbconfig/20260526-195557-fceratto.json * 19:55 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2054.codfw.wmnet * 19:51 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:51 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P93078 and previous config saved to /var/cache/conftool/dbconfig/20260526-194549-fceratto.json * 19:45 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 19:44 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:43 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2029 * 19:43 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 * 19:43 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 * 19:43 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 19:40 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb2014.codfw.wmnet with OS trixie * 19:40 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:40 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb2013.codfw.wmnet with OS trixie * 19:40 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:39 brett@cumin2002: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 19:38 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 19:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P93077 and previous config saved to /var/cache/conftool/dbconfig/20260526-193541-fceratto.json * 19:35 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:35 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 19:30 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 19:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93076 and previous config saved to /var/cache/conftool/dbconfig/20260526-192533-fceratto.json * 19:24 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:21 dzahn@cumin2002: START - Cookbook sre.dns.netbox * 19:20 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2051.codfw.wmnet * 19:19 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:19 brett@cumin2002: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 19:18 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93075 and previous config saved to /var/cache/conftool/dbconfig/20260526-191818-fceratto.json * 19:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance * 19:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93074 and previous config saved to /var/cache/conftool/dbconfig/20260526-191748-fceratto.json * 19:16 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2052.codfw.wmnet * 19:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P93073 and previous config saved to /var/cache/conftool/dbconfig/20260526-190740-fceratto.json * 19:07 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb2014.codfw.wmnet with reason: host reimage * 19:03 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb2013.codfw.wmnet with reason: host reimage * 18:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1026.eqiad.wmnet * 18:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P93072 and previous config saved to /var/cache/conftool/dbconfig/20260526-185732-fceratto.json * 18:56 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2014.codfw.wmnet with reason: host reimage * 18:56 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2013.codfw.wmnet with reason: host reimage * 18:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93071 and previous config saved to /var/cache/conftool/dbconfig/20260526-184724-fceratto.json * 18:44 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb2014.codfw.wmnet with OS trixie * 18:43 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb2013.codfw.wmnet with OS trixie * 18:41 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host rdb2014.codfw.wmnet with OS trixie * 18:41 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2049.codfw.wmnet * 18:40 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93070 and previous config saved to /var/cache/conftool/dbconfig/20260526-184009-fceratto.json * 18:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance * 18:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93069 and previous config saved to /var/cache/conftool/dbconfig/20260526-183939-fceratto.json * 18:37 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2050.codfw.wmnet * 18:30 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 18:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P93068 and previous config saved to /var/cache/conftool/dbconfig/20260526-182931-fceratto.json * 18:29 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:29 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_magru-v4 - dzahn@cumin2002" * 18:29 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_magru-v4 - dzahn@cumin2002" * 18:24 dzahn@cumin2002: START - Cookbook sre.dns.netbox * 18:21 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 18:21 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 18:21 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 18:20 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 18:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P93066 and previous config saved to /var/cache/conftool/dbconfig/20260526-181923-fceratto.json * 18:15 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 18:15 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 18:15 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 18:15 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 18:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93065 and previous config saved to /var/cache/conftool/dbconfig/20260526-180915-fceratto.json * 18:02 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93064 and previous config saved to /var/cache/conftool/dbconfig/20260526-180205-fceratto.json * 18:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2159.codfw.wmnet with reason: Maintenance * 18:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93063 and previous config saved to /var/cache/conftool/dbconfig/20260526-180132-fceratto.json * 18:00 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2047.codfw.wmnet * 17:59 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2048.codfw.wmnet * 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:54 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:54 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:54 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P93062 and previous config saved to /var/cache/conftool/dbconfig/20260526-175124-fceratto.json * 17:42 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293779{{!}}Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] (duration: 07m 25s) * 17:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P93060 and previous config saved to /var/cache/conftool/dbconfig/20260526-174117-fceratto.json * 17:39 mvernon@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ms-be2089.codfw.wmnet * 17:37 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 17:37 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:36 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:36 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:36 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1293779{{!}}Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:36 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:34 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1293779{{!}}Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] * 17:33 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:33 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:33 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:33 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93059 and previous config saved to /var/cache/conftool/dbconfig/20260526-173109-fceratto.json * 17:27 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:26 jclark@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:25 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:25 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:25 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:24 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:24 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1001 to eqiad - jclark@cumin1003" * 17:24 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:24 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1001 to eqiad - jclark@cumin1003" * 17:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93058 and previous config saved to /var/cache/conftool/dbconfig/20260526-172332-fceratto.json * 17:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2227.codfw.wmnet with reason: Maintenance * 17:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93057 and previous config saved to /var/cache/conftool/dbconfig/20260526-172303-fceratto.json * 17:21 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2045.codfw.wmnet * 17:20 jclark@cumin1003: START - Cookbook sre.dns.netbox * 17:20 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2046.codfw.wmnet * 17:18 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:17 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:17 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:17 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:17 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:17 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:17 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:17 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:16 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:15 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 17:14 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:14 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:14 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:14 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:13 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:13 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:13 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:13 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:13 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P93056 and previous config saved to /var/cache/conftool/dbconfig/20260526-171255-fceratto.json * 17:11 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:11 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:11 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:11 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:07 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:05 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:05 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:05 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:05 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P93055 and previous config saved to /var/cache/conftool/dbconfig/20260526-170247-fceratto.json * 17:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:57 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:55 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:52 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93054 and previous config saved to /var/cache/conftool/dbconfig/20260526-165240-fceratto.json * 16:50 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:50 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:50 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:50 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:45 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:45 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:45 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:45 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:45 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:44 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:44 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2209 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93053 and previous config saved to /var/cache/conftool/dbconfig/20260526-164421-fceratto.json * 16:44 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:44 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1002 to eqiad - jclark@cumin1003" * 16:44 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2209.codfw.wmnet with reason: Maintenance * 16:44 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1002 to eqiad - jclark@cumin1003" * 16:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93052 and previous config saved to /var/cache/conftool/dbconfig/20260526-164352-fceratto.json * 16:42 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2043.codfw.wmnet * 16:41 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2044.codfw.wmnet * 16:40 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:40 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:40 jclark@cumin1003: START - Cookbook sre.dns.netbox * 16:40 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:40 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:40 brett: reboot lvs 101[345].eqiad.wmnet * 16:39 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:39 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:39 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:39 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:37 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:37 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 16:37 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:37 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:37 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:37 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 16:37 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 16:36 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 16:36 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:36 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:36 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:36 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:35 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 16:34 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 16:34 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:34 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:34 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:34 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:33 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_codfw and A:cp * 16:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P93051 and previous config saved to /var/cache/conftool/dbconfig/20260526-163344-fceratto.json * 16:33 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_codfw and A:cp * 16:31 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:31 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:30 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:30 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P93050 and previous config saved to /var/cache/conftool/dbconfig/20260526-162336-fceratto.json * 16:13 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2089.codfw.wmnet * 16:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93049 and previous config saved to /var/cache/conftool/dbconfig/20260526-161328-fceratto.json * 16:11 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:11 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:10 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:10 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:07 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search,name=eqiad * 16:06 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:06 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:06 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:06 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93047 and previous config saved to /var/cache/conftool/dbconfig/20260526-160450-fceratto.json * 16:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2194.codfw.wmnet with reason: Maintenance * 16:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93046 and previous config saved to /var/cache/conftool/dbconfig/20260526-160420-fceratto.json * 16:03 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:03 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:03 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] (duration: 00m 28s) * 16:02 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] * 16:00 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:00 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:00 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:00 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:55 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:55 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:55 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:55 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] (duration: 00m 22s) * 15:55 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:55 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] * 15:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P93045 and previous config saved to /var/cache/conftool/dbconfig/20260526-155413-fceratto.json * 15:46 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search,name=eqiad * 15:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P93044 and previous config saved to /var/cache/conftool/dbconfig/20260526-154405-fceratto.json * 15:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93043 and previous config saved to /var/cache/conftool/dbconfig/20260526-153357-fceratto.json * 15:30 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:30 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:30 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:30 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93042 and previous config saved to /var/cache/conftool/dbconfig/20260526-152629-fceratto.json * 15:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2190.codfw.wmnet with reason: Maintenance * 15:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93041 and previous config saved to /var/cache/conftool/dbconfig/20260526-152559-fceratto.json * 15:24 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:24 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:24 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:24 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:23 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:22 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:22 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:22 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P93040 and previous config saved to /var/cache/conftool/dbconfig/20260526-151552-fceratto.json * 15:12 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2196: Rack maintenance completed * 15:10 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2196.codfw.wmnet * 15:10 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2196.codfw.wmnet * 15:07 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw * 15:06 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2222: Rack maintenance completed * 15:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P93037 and previous config saved to /var/cache/conftool/dbconfig/20260526-150546-fceratto.json * 15:04 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2221: Rack maintenance completed * 15:04 brennen@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab1004 for [[phab:T427286|T427286]] (duration: 00m 39s) * 15:03 brennen@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab1004 for [[phab:T427286|T427286]] * 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2002 for [[phab:T427286|T427286]] (duration: 00m 45s) * 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2002 for [[phab:T427286|T427286]] * 15:02 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2002.codfw.wmnet with reason: Phabricator deploy * 15:01 bjensen: uploading prometheus-memcached-exporter_0.16.0-1_amd64 on apt1002 * 15:01 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator deploy * 15:00 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2223: switch maintenance * 14:56 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2196: Rack maintenance completed * 14:55 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2221.codfw.wmnet * 14:55 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2221.codfw.wmnet * 14:55 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2222.codfw.wmnet * 14:55 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2222.codfw.wmnet * 14:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93033 and previous config saved to /var/cache/conftool/dbconfig/20260526-145538-fceratto.json * 14:55 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet * 14:54 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1026.eqiad.wmnet * 14:52 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet * 14:52 moritzm: remove ganeti1025 from eqiad Ganeti cluster [[phab:T424680|T424680]] * 14:51 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2030.codfw.wmnet to cluster codfw and group A * 14:51 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2222: Rack maintenance completed * 14:49 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:49 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2221: Rack maintenance completed * 14:49 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2030.codfw.wmnet to cluster codfw and group A * 14:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2029.codfw.wmnet to cluster codfw and group A * 14:47 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2029.codfw.wmnet to cluster codfw and group A * 14:47 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2177 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93030 and previous config saved to /var/cache/conftool/dbconfig/20260526-144718-fceratto.json * 14:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance * 14:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93029 and previous config saved to /var/cache/conftool/dbconfig/20260526-144651-fceratto.json * 14:45 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=wdqs-scholarly,name=codfw * 14:45 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=wdqs-scholarly,name=codfw * 14:43 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw * 14:40 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:40 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2167: Migration of db2167.codfw.wmnet completed * 14:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P93026 and previous config saved to /var/cache/conftool/dbconfig/20260526-143643-fceratto.json * 14:31 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1054.eqiad.wmnet with OS trixie * 14:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P93023 and previous config saved to /var/cache/conftool/dbconfig/20260526-142636-fceratto.json * 14:26 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:25 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:24 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1014: Rack maintenance completed * 14:24 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.parsercache (exit_code=99) * 14:24 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache * 14:24 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc1014: Rack maintenance completed * 14:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1025.eqiad.wmnet * 14:19 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for backup2015.codfw.wmnet,db2197.codfw.wmnet * 14:19 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for backup2015.codfw.wmnet,db2197.codfw.wmnet * 14:18 jynus: restarting mediabackups@codfw after maintenance on a codfw backup media storage server [[phab:T426199|T426199]] * 14:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93021 and previous config saved to /var/cache/conftool/dbconfig/20260526-141628-fceratto.json * 14:16 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:14 fabfur: repooled cp2043 ([[phab:T426199|T426199]]) * 14:14 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2223: switch maintenance * 14:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1054.eqiad.wmnet with reason: host reimage * 14:14 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2043.* * 14:13 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293710{{!}}Site info should output thumblimits as array (T427066)]] (duration: 06m 40s) * 14:12 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:10 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1054.eqiad.wmnet with reason: host reimage * 14:10 fabfur@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2011.codfw.wmnet * 14:10 fabfur@cumin1003: START - Cookbook sre.hosts.remove-downtime for lvs2011.codfw.wmnet * 14:09 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 14:09 fabfur: restoring lvs2011 as primary ([[phab:T426199|T426199]]) * 14:08 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1293710{{!}}Site info should output thumblimits as array (T427066)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:08 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 14:08 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 14:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93017 and previous config saved to /var/cache/conftool/dbconfig/20260526-140748-fceratto.json * 14:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance * 14:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93016 and previous config saved to /var/cache/conftool/dbconfig/20260526-140718-fceratto.json * 14:07 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1293710{{!}}Site info should output thumblimits as array (T427066)]] * 14:05 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.decommission (exit_code=99) * 14:05 marostegui@cumin1003: Removing pc1013 from zarcillo [[phab:T427190|T427190]] * 14:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1013.eqiad.wmnet * 14:04 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:04 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 14:04 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 14:00 marostegui@cumin1003: START - Cookbook sre.dns.netbox * 13:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P93014 and previous config saved to /var/cache/conftool/dbconfig/20260526-135711-fceratto.json * 13:56 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1054.eqiad.wmnet with OS trixie * 13:55 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2167: Migration of db2167.codfw.wmnet completed * 13:53 Amir1: drop flaggedrevs tables on cawikinews ([[phab:T423577|T423577]]) * 13:49 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1013.eqiad.wmnet * 13:49 marostegui@cumin1003: START - Cookbook sre.mysql.decommission * 13:48 Lucas_WMDE: UTC afternoon backport+config window done * 13:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P93012 and previous config saved to /var/cache/conftool/dbconfig/20260526-134703-fceratto.json * 13:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2167.codfw.wmnet with OS trixie * 13:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93011 and previous config saved to /var/cache/conftool/dbconfig/20260526-133656-fceratto.json * 13:36 XioNoX: reboot lsw1-a2-codfw for software upgrade - [[phab:T426199|T426199]] * 13:36 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2223: switch maintenance * 13:35 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2223: switch maintenance * 13:35 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2222: switch maintenance * 13:35 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2222: switch maintenance * 13:35 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2221: switch maintenance * 13:35 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293662{{!}}Enable IRS Direct Reporting on testwiki (T425025)]] (duration: 09m 28s) * 13:34 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2221: switch maintenance * 13:34 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2196: switch maintenance * 13:34 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2196: switch maintenance * 13:31 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 13:30 stran@deploy1003: stran: Continuing with deployment * 13:29 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 13:29 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93006 and previous config saved to /var/cache/conftool/dbconfig/20260526-132927-fceratto.json * 13:29 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2167.codfw.wmnet with reason: host reimage * 13:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance * 13:29 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 34 hosts with reason: Switch maintenance * 13:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93005 and previous config saved to /var/cache/conftool/dbconfig/20260526-132857-fceratto.json * 13:28 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lsw1-a2-codfw,lsw1-a2-codfw IPv6,lsw1-a2-codfw.mgmt with reason: Switch maintenance * 13:27 stran@deploy1003: stran: Backport for [[gerrit:1293662{{!}}Enable IRS Direct Reporting on testwiki (T425025)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:25 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1293662{{!}}Enable IRS Direct Reporting on testwiki (T425025)]] * 13:25 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2167.codfw.wmnet with reason: host reimage * 13:22 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293706{{!}}Disable the `no` language code for translation (T424613)]] (duration: 08m 30s) * 13:22 ladsgroup@dns1004: END - running authdns-update * 13:20 ladsgroup@dns1004: START - running authdns-update * 13:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P93004 and previous config saved to /var/cache/conftool/dbconfig/20260526-131850-fceratto.json * 13:18 lucaswerkmeister-wmde@deploy1003: jhsoby, lucaswerkmeister-wmde: Continuing with deployment * 13:16 lucaswerkmeister-wmde@deploy1003: jhsoby, lucaswerkmeister-wmde: Backport for [[gerrit:1293706{{!}}Disable the `no` language code for translation (T424613)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:14 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1293706{{!}}Disable the `no` language code for translation (T424613)]] * 13:12 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293177{{!}}Instrumentation: log new articles namespace and source (T422146)]] (duration: 07m 09s) * 13:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P93003 and previous config saved to /var/cache/conftool/dbconfig/20260526-130842-fceratto.json * 13:08 sbisson@deploy1003: sbisson: Continuing with deployment * 13:07 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2167.codfw.wmnet with OS trixie * 13:07 sbisson@deploy1003: sbisson: Backport for [[gerrit:1293177{{!}}Instrumentation: log new articles namespace and source (T422146)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:05 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2167: Upgrading db2167.codfw.wmnet * 13:05 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1293177{{!}}Instrumentation: log new articles namespace and source (T422146)]] * 13:04 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2167: Upgrading db2167.codfw.wmnet * 13:04 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:04 kart_: Update Recommendation API to 2026-05-26-074931-production * 13:03 kartik@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 13:00 topranks: deactivate CR BGP to doh2002 to test backup path via doh2001 * 12:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93000 and previous config saved to /var/cache/conftool/dbconfig/20260526-125834-fceratto.json * 12:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2226 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92999 and previous config saved to /var/cache/conftool/dbconfig/20260526-125135-fceratto.json * 12:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance * 12:51 kartik@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92998 and previous config saved to /var/cache/conftool/dbconfig/20260526-125105-fceratto.json * 12:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P92997 and previous config saved to /var/cache/conftool/dbconfig/20260526-124059-fceratto.json * 12:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2003.wikimedia.org * 12:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1214: Migration of db1214.eqiad.wmnet completed * 12:33 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc2003.wikimedia.org * 12:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P92995 and previous config saved to /var/cache/conftool/dbconfig/20260526-123052-fceratto.json * 12:26 fabfur: depooled cp204 for network activity ([[phab:T426199|T426199]]) * 12:26 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2043.* * 12:24 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-a1-codfw,ssw1-a1-codfw IPv6,ssw1-a1-codfw.mgmt with reason: Switch maintenance * 12:24 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mirror1001.wikimedia.org * 12:23 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:23 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:22 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92993 and previous config saved to /var/cache/conftool/dbconfig/20260526-122044-fceratto.json * 12:20 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:19 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mirror1001.wikimedia.org * 12:13 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2225 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92991 and previous config saved to /var/cache/conftool/dbconfig/20260526-121336-fceratto.json * 12:13 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance * 12:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92990 and previous config saved to /var/cache/conftool/dbconfig/20260526-121306-fceratto.json * 12:09 fabfur@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2011.codfw.wmnet with reason: Planned downtime for rack maintenance * 12:08 fabfur: downtime, disable puppet and stop pybal for rack maintenance ([[phab:T426199|T426199]]) * 12:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2181: Migration of db2181.codfw.wmnet completed * 12:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P92987 and previous config saved to /var/cache/conftool/dbconfig/20260526-120258-fceratto.json * 12:01 XioNoX: start ssw1-a1-codfw network maintenance (no impact expected as the spines are redundant) * 11:59 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293167{{!}}hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055{{!}}hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] (duration: 15m 26s) * 11:56 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on backup2015.codfw.wmnet,db2197.codfw.wmnet with reason: network maintenance * 11:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1005.eqiad.wmnet * 11:55 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment * 11:54 jynus: stopping mediabackups@codfw for maintenance on a codfw backup media storage server [[phab:T426199|T426199]] * 11:54 jmm@dns1004: END - running authdns-update * 11:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P92985 and previous config saved to /var/cache/conftool/dbconfig/20260526-115251-fceratto.json * 11:52 jmm@dns1004: START - running authdns-update * 11:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1005.eqiad.wmnet * 11:49 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1214: Migration of db1214.eqiad.wmnet completed * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1004.eqiad.wmnet * 11:47 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet * 11:46 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1293167{{!}}hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055{{!}}hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:45 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1004.eqiad.wmnet * 11:44 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1293167{{!}}hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055{{!}}hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] * 11:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92983 and previous config saved to /var/cache/conftool/dbconfig/20260526-114243-fceratto.json * 11:42 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet * 11:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1214.eqiad.wmnet with OS trixie * 11:35 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293691{{!}}Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] (duration: 06m 46s) * 11:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92981 and previous config saved to /var/cache/conftool/dbconfig/20260526-113542-fceratto.json * 11:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance * 11:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92980 and previous config saved to /var/cache/conftool/dbconfig/20260526-113521-fceratto.json * 11:31 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 11:31 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1293691{{!}}Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1222: Migration of db1222.eqiad.wmnet completed * 11:29 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1293691{{!}}Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] * 11:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P92978 and previous config saved to /var/cache/conftool/dbconfig/20260526-112513-fceratto.json * 11:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1214.eqiad.wmnet with reason: host reimage * 11:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc4 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92977 and previous config saved to /var/cache/conftool/dbconfig/20260526-112326-marostegui.json * 11:22 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2181: Migration of db2181.codfw.wmnet completed * 11:22 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1024 to dbctl [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92975 and previous config saved to /var/cache/conftool/dbconfig/20260526-112215-marostegui.json * 11:20 fceratto@cumin1003: dbctl commit (dc=all): 'Switchover es2042 es2041 for [[phab:T426199|T426199]]', diff saved to https://phabricator.wikimedia.org/P92974 and previous config saved to /var/cache/conftool/dbconfig/20260526-112028-fceratto.json * 11:17 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1214.eqiad.wmnet with reason: host reimage * 11:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P92972 and previous config saved to /var/cache/conftool/dbconfig/20260526-111506-fceratto.json * 11:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2181.codfw.wmnet with OS trixie * 11:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92971 and previous config saved to /var/cache/conftool/dbconfig/20260526-110458-fceratto.json * 11:02 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1214.eqiad.wmnet with OS trixie * 11:00 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293095{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] (duration: 15m 50s) * 11:00 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1214: Upgrading db1214.eqiad.wmnet * 10:59 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1214: Upgrading db1214.eqiad.wmnet * 10:59 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92968 and previous config saved to /var/cache/conftool/dbconfig/20260526-105755-fceratto.json * 10:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance * 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92967 and previous config saved to /var/cache/conftool/dbconfig/20260526-105726-fceratto.json * 10:56 jiji@deploy1003: jiji: Continuing with deployment * 10:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2181.codfw.wmnet with reason: host reimage * 10:51 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2181.codfw.wmnet with reason: host reimage * 10:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P92966 and previous config saved to /var/cache/conftool/dbconfig/20260526-104718-fceratto.json * 10:46 jiji@deploy1003: jiji: Backport for [[gerrit:1293095{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:44 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1293095{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] * 10:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P92964 and previous config saved to /var/cache/conftool/dbconfig/20260526-103711-fceratto.json * 10:36 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2181.codfw.wmnet with OS trixie * 10:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply * 10:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply * 10:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92963 and previous config saved to /var/cache/conftool/dbconfig/20260526-102703-fceratto.json * 10:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1226: Migration of db1226.eqiad.wmnet completed * 10:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2181: Upgrading db2181.codfw.wmnet * 10:24 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2181: Upgrading db2181.codfw.wmnet * 10:24 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92960 and previous config saved to /var/cache/conftool/dbconfig/20260526-101936-fceratto.json * 10:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance * 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92959 and previous config saved to /var/cache/conftool/dbconfig/20260526-101842-fceratto.json * 10:16 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-codfw@codfw * 10:16 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 10:15 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 10:10 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293668{{!}}hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] (duration: 06m 42s) * 10:09 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-codfw@codfw * 10:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92957 and previous config saved to /var/cache/conftool/dbconfig/20260526-100834-fceratto.json * 10:06 kharlan@deploy1003: kharlan: Continuing with deployment * 10:05 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293668{{!}}hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:03 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293668{{!}}hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] * 10:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2195: Migration of db2195.codfw.wmnet completed * 10:01 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>kubestage200*<nowiki>}</nowiki> and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw) * 10:01 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet * 10:01 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet * 10:00 jmm@cumin2002: END (PASS) - Cookbook sre.netbox.restart-reboot (exit_code=0) rolling reboot on A:netbox * 09:58 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 09:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92955 and previous config saved to /var/cache/conftool/dbconfig/20260526-095827-fceratto.json * 09:58 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 09:58 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 09:57 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 09:56 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-eqiad@eqiad * 09:56 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:55 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 09:55 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:55 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:55 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet * 09:54 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet * 09:54 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet * 09:54 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet * 09:53 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>kubestage100*<nowiki>}</nowiki> and (A:wikikube-staging-master-eqiad or A:wikikube-staging-worker-eqiad) * 09:53 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet * 09:53 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet * 09:52 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-eqiad@eqiad * 09:52 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293665{{!}}hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] (duration: 08m 07s) * 09:51 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2043.* * 09:51 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2044.* * 09:48 fabfur: repooling cp2043 and cp2044 (haproxy-awslc) ([[phab:T419825|T419825]]) * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92953 and previous config saved to /var/cache/conftool/dbconfig/20260526-094819-fceratto.json * 09:47 kharlan@deploy1003: kharlan: Continuing with deployment * 09:46 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet * 09:45 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293665{{!}}hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:44 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs3009.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:44 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293665{{!}}hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] * 09:41 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet * 09:41 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet * 09:41 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet * 09:41 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92951 and previous config saved to /var/cache/conftool/dbconfig/20260526-094115-fceratto.json * 09:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2229.codfw.wmnet with reason: Maintenance * 09:41 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs3009.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92950 and previous config saved to /var/cache/conftool/dbconfig/20260526-094045-fceratto.json * 09:40 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1226: Migration of db1226.eqiad.wmnet completed * 09:39 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-codfw@codfw * 09:39 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:38 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:34 fabfur: depooling cp2044 to install haproxy-awslc ([[phab:T419825|T419825]]) * 09:34 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet * 09:34 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet * 09:34 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2044.* * 09:33 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet * 09:33 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet * 09:33 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet * 09:33 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2043.* * 09:32 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293661{{!}}hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] (duration: 06m 52s) * 09:32 fabfur: depooling cp2043 to install haproxy-awslc ([[phab:T419825|T419825]]) * 09:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1226.eqiad.wmnet with OS trixie * 09:30 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-codfw@codfw * 09:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P92947 and previous config saved to /var/cache/conftool/dbconfig/20260526-093031-fceratto.json * 09:29 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet * 09:29 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet * 09:29 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet * 09:28 kharlan@deploy1003: kharlan: Continuing with deployment * 09:28 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs3008.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:28 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293661{{!}}hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:27 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet * 09:26 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet * 09:26 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet * 09:26 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet * 09:26 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293661{{!}}hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] * 09:25 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs3008.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:25 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs3010.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet * 09:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet * 09:21 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs3010.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:20 fabfur: start rebooting esams liberica instances ([[phab:T426563|T426563]]) * 09:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P92946 and previous config saved to /var/cache/conftool/dbconfig/20260526-092024-fceratto.json * 09:20 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet * 09:16 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2195: Migration of db2195.codfw.wmnet completed * 09:15 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet * 09:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1226.eqiad.wmnet with reason: host reimage * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>kubestage100*<nowiki>}</nowiki> and (A:wikikube-staging-master-eqiad or A:wikikube-staging-worker-eqiad) * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>kubestage200*<nowiki>}</nowiki> and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw) * 09:14 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293658{{!}}Fix TypeError in Mandatory2FAChecker (T427251)]] (duration: 06m 47s) * 09:10 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1226.eqiad.wmnet with reason: host reimage * 09:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92944 and previous config saved to /var/cache/conftool/dbconfig/20260526-091016-fceratto.json * 09:09 mszwarc@deploy1003: mszwarc: Continuing with deployment * 09:09 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1293658{{!}}Fix TypeError in Mandatory2FAChecker (T427251)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2195.codfw.wmnet with OS trixie * 09:07 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293658{{!}}Fix TypeError in Mandatory2FAChecker (T427251)]] * 09:06 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs4009.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 09:03 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92943 and previous config saved to /var/cache/conftool/dbconfig/20260526-090315-fceratto.json * 09:03 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2224.codfw.wmnet with reason: Maintenance * 09:03 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs4009.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92942 and previous config saved to /var/cache/conftool/dbconfig/20260526-090256-fceratto.json * 08:57 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs4008.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 08:56 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox.discovery.wmnet. on all recursors * 08:56 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors * 08:55 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1226.eqiad.wmnet with OS trixie * 08:53 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs4008.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 08:53 fabfur: start rebooting ulsfo liberica instances ([[phab:T426563|T426563]]) * 08:53 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293594{{!}}Allow to remove passkeys when there's only one standard 2FA method (T426872)]] (duration: 07m 23s) * 08:53 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs5005.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:53 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1226: Upgrading db1226.eqiad.wmnet * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P92941 and previous config saved to /var/cache/conftool/dbconfig/20260526-085248-fceratto.json * 08:51 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox.discovery.wmnet. on all recursors * 08:51 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors * 08:51 jmm@cumin2002: START - Cookbook sre.netbox.restart-reboot rolling reboot on A:netbox * 08:50 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1226: Upgrading db1226.eqiad.wmnet * 08:50 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs5005.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:50 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2195.codfw.wmnet with reason: host reimage * 08:49 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1222: Migration of db1222.eqiad.wmnet completed * 08:48 mszwarc@deploy1003: mszwarc: Continuing with deployment * 08:47 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1293594{{!}}Allow to remove passkeys when there's only one standard 2FA method (T426872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:46 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293594{{!}}Allow to remove passkeys when there's only one standard 2FA method (T426872)]] * 08:43 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs5004.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet * 08:43 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2195.codfw.wmnet with reason: host reimage * 08:43 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1292032{{!}}Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964{{!}}hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] (duration: 09m 56s) * 08:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P92939 and previous config saved to /var/cache/conftool/dbconfig/20260526-084240-fceratto.json * 08:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1222.eqiad.wmnet with OS trixie * 08:40 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs5004.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:40 fabfur: start rebooting eqsin liberica instances ([[phab:T426563|T426563]]) * 08:39 kartik@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 08:39 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet * 08:39 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 08:39 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs5006.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:35 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs5006.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1024.eqiad.wmnet * 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1024.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 08:35 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1292032{{!}}Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964{{!}}hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:33 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs6002.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:33 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1292032{{!}}Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964{{!}}hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] * 08:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92938 and previous config saved to /var/cache/conftool/dbconfig/20260526-083233-fceratto.json * 08:30 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs6002.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:25 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2217 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92937 and previous config saved to /var/cache/conftool/dbconfig/20260526-082531-fceratto.json * 08:25 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2217.codfw.wmnet with reason: Maintenance * 08:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92936 and previous config saved to /var/cache/conftool/dbconfig/20260526-082458-fceratto.json * 08:23 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2195.codfw.wmnet with OS trixie * 08:23 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage * 08:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2195: Upgrading db2195.codfw.wmnet * 08:20 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2195: Upgrading db2195.codfw.wmnet * 08:19 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:18 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage * 08:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P92934 and previous config saved to /var/cache/conftool/dbconfig/20260526-081451-fceratto.json * 08:13 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs6001.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 08:10 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs6001.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:09 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1024.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 08:04 jmm@cumin2002: START - Cookbook sre.dns.netbox * 08:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P92932 and previous config saved to /var/cache/conftool/dbconfig/20260526-080443-fceratto.json * 08:01 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1222.eqiad.wmnet with OS trixie * 08:00 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs6003.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet * 07:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet * 07:59 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1024.eqiad.wmnet * 07:59 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1023.eqiad.wmnet * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1023.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:59 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 07:59 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:58 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1023.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:56 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs6003.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 07:56 fabfur: start rebooting drmrs liberica instances ([[phab:T426563|T426563]]) * 07:56 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs7002.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:54 jmm@cumin2002: START - Cookbook sre.dns.netbox * 07:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92931 and previous config saved to /var/cache/conftool/dbconfig/20260526-075435-fceratto.json * 07:52 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs7002.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:51 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1047.eqiad.wmnet * 07:51 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:51 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1047.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:49 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1023.eqiad.wmnet * 07:47 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2193 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92930 and previous config saved to /var/cache/conftool/dbconfig/20260526-074739-fceratto.json * 07:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2193.codfw.wmnet with reason: Maintenance * 07:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92929 and previous config saved to /var/cache/conftool/dbconfig/20260526-074710-fceratto.json * 07:46 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet * 07:45 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet * 07:45 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:45 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs7001.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:44 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1025.eqiad.wmnet * 07:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:43 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:41 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs7001.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:40 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1046.eqiad.wmnet * 07:40 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1046.eqiad.wmnet * 07:38 arthurtaylor@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291951{{!}}Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] (duration: 12m 01s) * 07:38 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1047.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P92928 and previous config saved to /var/cache/conftool/dbconfig/20260526-073702-fceratto.json * 07:37 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet * 07:36 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet * 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance * 07:35 fabfur: start rebooting magru liberica instances ([[phab:T426563|T426563]]) * 07:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92926 and previous config saved to /var/cache/conftool/dbconfig/20260526-073459-fceratto.json * 07:32 arthurtaylor@deploy1003: arthurtaylor: Continuing with deployment * 07:31 arthurtaylor@deploy1003: arthurtaylor: Backport for [[gerrit:1291951{{!}}Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:30 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1046.eqiad.wmnet * 07:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20260526-072643-fceratto.json * 07:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1046.eqiad.wmnet * 07:26 arthurtaylor@deploy1003: Started scap sync-world: Backport for [[gerrit:1291951{{!}}Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] * 07:25 jiji@cumin1003: START - Cookbook sre.dns.netbox * 07:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P92924 and previous config saved to /var/cache/conftool/dbconfig/20260526-072452-fceratto.json * 07:24 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet * 07:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1047.eqiad.wmnet * 07:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1047.eqiad.wmnet * 07:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92923 and previous config saved to /var/cache/conftool/dbconfig/20260526-071635-fceratto.json * 07:15 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet * 07:15 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1026.eqiad.wmnet * 07:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P92922 and previous config saved to /var/cache/conftool/dbconfig/20260526-071444-fceratto.json * 07:13 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet * 07:11 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1025.eqiad.wmnet * 07:10 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1025.eqiad.wmnet * 07:09 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92921 and previous config saved to /var/cache/conftool/dbconfig/20260526-070946-fceratto.json * 07:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2180.codfw.wmnet with reason: Maintenance * 07:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92920 and previous config saved to /var/cache/conftool/dbconfig/20260526-070916-fceratto.json * 07:09 moritzm: failover Ganeti master in eqiad to ganeti1048 * 07:09 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1047.eqiad.wmnet * 07:07 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1046.eqiad.wmnet * 07:07 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:06 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1046.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org * 07:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92919 and previous config saved to /var/cache/conftool/dbconfig/20260526-070436-fceratto.json * 07:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet * 07:04 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1046.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet * 07:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org * 06:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P92918 and previous config saved to /var/cache/conftool/dbconfig/20260526-065909-fceratto.json * 06:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org * 06:58 jiji@cumin1003: START - Cookbook sre.dns.netbox * 06:58 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet * 06:55 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet * 06:53 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1046.eqiad.wmnet * 06:53 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1045.eqiad.wmnet * 06:53 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:53 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1045.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 06:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org * 06:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P92917 and previous config saved to /var/cache/conftool/dbconfig/20260526-064901-fceratto.json * 06:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92916 and previous config saved to /var/cache/conftool/dbconfig/20260526-064833-fceratto.json * 06:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1222.eqiad.wmnet with reason: Maintenance * 06:47 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1222: Switchover * 06:41 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92914 and previous config saved to /var/cache/conftool/dbconfig/20260526-063853-fceratto.json * 06:35 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 06:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92912 and previous config saved to /var/cache/conftool/dbconfig/20260526-063155-fceratto.json * 06:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance * 06:28 fceratto@cumin1003: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance * 06:23 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1222: Switchover * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1222 [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92910 and previous config saved to /var/cache/conftool/dbconfig/20260526-061656-fceratto.json * 06:15 fceratto@dns1005: END - running authdns-update * 06:14 fceratto@dns1005: START - running authdns-update * 06:11 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1162 to s2 primary and set section read-write [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92909 and previous config saved to /var/cache/conftool/dbconfig/20260526-061114-fceratto.json * 06:10 fceratto@cumin1003: dbctl commit (dc=all): 'Set s2 eqiad as read-only for maintenance - [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92908 and previous config saved to /var/cache/conftool/dbconfig/20260526-061021-fceratto.json * 06:10 federico3: Starting s2 eqiad failover from db1222 to db1162 - [[phab:T425622|T425622]] * 06:04 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1162 with weight 0 [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92907 and previous config saved to /var/cache/conftool/dbconfig/20260526-060443-fceratto.json * 06:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s2 [[phab:T425622|T425622]] * 06:02 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:02 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:01 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:00 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 05:15 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1014.eqiad.wmnet: Maintenance on pc4 * 05:15 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 05:15 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 05:15 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1014.eqiad.wmnet: Maintenance on pc4 * 05:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2024.codfw.wmnet,pc[1014,1024].eqiad.wmnet with reason: Maintenance on pc4 * 04:37 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 04:34 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.1 (duration: 02m 32s) * 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] (duration: 36m 24s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 20s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-25 == * 21:00 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1045.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:49 jiji@cumin1003: START - Cookbook sre.dns.netbox * 20:38 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1045.eqiad.wmnet * 20:37 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1044.eqiad.wmnet * 20:37 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:37 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1044.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:25 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1044.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:15 moritzm: truncate krb5kdc.log1 (which made log rotation fail) * 20:06 jiji@cumin1003: START - Cookbook sre.dns.netbox * 19:57 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1044.eqiad.wmnet * 19:25 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1043.eqiad.wmnet * 19:25 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:25 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1043.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 19:22 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1043.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 18:49 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_eqiad * 18:49 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1115.eqiad.wmnet * 18:34 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5023.eqsin.wmnet [reason: manually pooling after reboot as icinga was down] * 18:33 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5030.eqsin.wmnet [reason: manually pooling after reboot as icinga was down] * 18:22 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp5030*<nowiki>}</nowiki> and A:cp * 18:22 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5030.eqsin.wmnet * 18:15 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp5023*<nowiki>}</nowiki> and A:cp * 18:15 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5023.eqsin.wmnet * 18:10 jiji@cumin1003: START - Cookbook sre.dns.netbox * 18:10 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp5030*<nowiki>}</nowiki> and A:cp * 18:09 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp1113*<nowiki>}</nowiki> and A:cp * 18:09 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet * 18:09 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet * 18:03 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp1113*<nowiki>}</nowiki> and A:cp * 18:02 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp5023*<nowiki>}</nowiki> and A:cp * 18:01 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-text_eqiad * 18:01 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-upload_eqsin * 18:01 sukhe: sre.cdn.roll-reboot cookbooks stalled due to icinga reboot * 18:00 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-text_eqsin * 17:35 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1043.eqiad.wmnet * 17:31 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1110.eqiad.wmnet [reason: manually pooling after reboot as icinga was down] * 17:30 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1042.eqiad.wmnet * 17:30 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:30 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1042.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:29 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1111.eqiad.wmnet * 17:28 sukhe: sukhe@alert1002:~$ sudo systemctl restart icinga.service * 17:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92903 and previous config saved to /var/cache/conftool/dbconfig/20260525-171310-fceratto.json * 17:11 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1042.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:06 jiji@cumin1003: START - Cookbook sre.dns.netbox * 17:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P92902 and previous config saved to /var/cache/conftool/dbconfig/20260525-170302-fceratto.json * 16:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P92901 and previous config saved to /var/cache/conftool/dbconfig/20260525-165255-fceratto.json * 16:51 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1042.eqiad.wmnet * 16:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92900 and previous config saved to /var/cache/conftool/dbconfig/20260525-164247-fceratto.json * 16:42 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1041.eqiad.wmnet * 16:42 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:42 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1041.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:41 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1041.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:40 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5021.eqsin.wmnet * 16:39 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5029.eqsin.wmnet * 16:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92899 and previous config saved to /var/cache/conftool/dbconfig/20260525-163559-fceratto.json * 16:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2158.codfw.wmnet with reason: Maintenance * 16:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92898 and previous config saved to /var/cache/conftool/dbconfig/20260525-163512-fceratto.json * 16:34 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1108.eqiad.wmnet * 16:30 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1109.eqiad.wmnet * 16:26 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249', diff saved to https://phabricator.wikimedia.org/P92897 and previous config saved to /var/cache/conftool/dbconfig/20260525-162505-fceratto.json * 16:20 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1041.eqiad.wmnet * 16:20 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1040.eqiad.wmnet * 16:20 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:20 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1040.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:16 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1040.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249', diff saved to https://phabricator.wikimedia.org/P92896 and previous config saved to /var/cache/conftool/dbconfig/20260525-161457-fceratto.json * 16:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92895 and previous config saved to /var/cache/conftool/dbconfig/20260525-160450-fceratto.json * 16:02 jiji@cumin1003: START - Cookbook sre.dns.netbox * 15:59 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2249 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92894 and previous config saved to /var/cache/conftool/dbconfig/20260525-155930-fceratto.json * 15:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2249.codfw.wmnet with reason: Maintenance * 15:57 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5020.eqsin.wmnet * 15:57 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5028.eqsin.wmnet * 15:52 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1106.eqiad.wmnet * 15:51 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1107.eqiad.wmnet * 15:29 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1040.eqiad.wmnet * 15:29 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1039.eqiad.wmnet * 15:29 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:29 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1039.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:27 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1039.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:17 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1013 from dbctl [[phab:T427190|T427190]]', diff saved to https://phabricator.wikimedia.org/P92893 and previous config saved to /var/cache/conftool/dbconfig/20260525-151718-marostegui.json * 15:15 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5019.eqsin.wmnet * 15:15 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5027.eqsin.wmnet * 15:12 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1104.eqiad.wmnet * 15:11 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1105.eqiad.wmnet * 15:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92892 and previous config saved to /var/cache/conftool/dbconfig/20260525-150309-fceratto.json * 14:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P92891 and previous config saved to /var/cache/conftool/dbconfig/20260525-145301-fceratto.json * 14:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P92890 and previous config saved to /var/cache/conftool/dbconfig/20260525-144253-fceratto.json * 14:33 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1102.eqiad.wmnet * 14:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92889 and previous config saved to /var/cache/conftool/dbconfig/20260525-143246-fceratto.json * 14:32 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5026.eqsin.wmnet * 14:32 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5018.eqsin.wmnet * 14:31 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1103.eqiad.wmnet * 14:25 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2228 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92888 and previous config saved to /var/cache/conftool/dbconfig/20260525-142551-fceratto.json * 14:25 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance * 14:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92887 and previous config saved to /var/cache/conftool/dbconfig/20260525-142520-fceratto.json * 14:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P92885 and previous config saved to /var/cache/conftool/dbconfig/20260525-141513-fceratto.json * 14:12 jiji@cumin1003: START - Cookbook sre.dns.netbox * 14:06 sukhe: curl localhost:9090/pools/inference-staging-grpc_30051 shows ml-staging200[1-3].codfw.wmnet as enabled and pooled: [[phab:T424049|T424049]] * 14:05 sukhe: sukhe@lvs2013:~$ sudo systemctl restart pybal.service: [[phab:T424049|T424049]] * 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P92884 and previous config saved to /var/cache/conftool/dbconfig/20260525-140505-fceratto.json * 14:03 sukhe: sudo cumin 'A:lvs and A:lvs-low-traffic-codfw' 'run-puppet-agent --enable "adding new ml-serve (grpc) [[phab:T424049|T424049]]"' * 14:02 sukhe: sukhe@lvs2014:~$ sudo systemctl restart pybal.service": [[phab:T424049|T424049]] * 14:02 sukhe: sukhe@lvs2014:~$ sudo systemctl restart pybal.service * 14:00 sukhe: sudo cumin 'A:lvs and A:lvs-secondary-codfw' 'run-puppet-agent --enable "adding new ml-serve (grpc) [[phab:T424049|T424049]]"' * 13:59 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1039.eqiad.wmnet * 13:58 sukhe: sudo cumin 'A:lvs and A:eqiad' 'run-puppet-agent --enable "adding new ml-serve (grpc) [[phab:T424049|T424049]]": NOOP change, since service is codfw only * 13:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92882 and previous config saved to /var/cache/conftool/dbconfig/20260525-135458-fceratto.json * 13:52 Msz2001: Everything deployed, UTC afternoon config+backport window done * 13:52 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293119{{!}}Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] (duration: 09m 43s) * 13:51 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1101.eqiad.wmnet * 13:51 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1100.eqiad.wmnet * 13:50 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5025.eqsin.wmnet * 13:50 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5017.eqsin.wmnet * 13:49 kart_: Updated Recommendation API to 2026-05-21-044522-production * 13:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2223 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92881 and previous config saved to /var/cache/conftool/dbconfig/20260525-134807-fceratto.json * 13:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2223.codfw.wmnet with reason: Maintenance * 13:47 mszwarc@deploy1003: vadymts1, mszwarc: Continuing with deployment * 13:47 kartik@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 13:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92880 and previous config saved to /var/cache/conftool/dbconfig/20260525-134737-fceratto.json * 13:45 mszwarc@deploy1003: vadymts1, mszwarc: Backport for [[gerrit:1293119{{!}}Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:45 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1162: Reboot * 13:43 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293119{{!}}Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] * 13:40 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqiad * 13:39 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqiad * 13:38 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290813{{!}}Article Guidance: enable experiment on phase 2 wikis (T426871)]] (duration: 08m 14s) * 13:38 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqsin * 13:38 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqsin * 13:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P92878 and previous config saved to /var/cache/conftool/dbconfig/20260525-133729-fceratto.json * 13:34 sbisson@deploy1003: sbisson: Continuing with deployment * 13:33 kartik@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 13:32 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1038.eqiad.wmnet * 13:32 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:32 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1038.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 13:31 sbisson@deploy1003: sbisson: Backport for [[gerrit:1290813{{!}}Article Guidance: enable experiment on phase 2 wikis (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:30 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1290813{{!}}Article Guidance: enable experiment on phase 2 wikis (T426871)]] * 13:27 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293094{{!}}Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953{{!}}Update logo, wordmark and tagline for zghwiki (T426406)]] (duration: 07m 43s) * 13:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P92876 and previous config saved to /var/cache/conftool/dbconfig/20260525-132722-fceratto.json * 13:23 mszwarc@deploy1003: mszwarc, jhsoby: Continuing with deployment * 13:21 mszwarc@deploy1003: mszwarc, jhsoby: Backport for [[gerrit:1293094{{!}}Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953{{!}}Update logo, wordmark and tagline for zghwiki (T426406)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:20 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1038.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 13:20 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293094{{!}}Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953{{!}}Update logo, wordmark and tagline for zghwiki (T426406)]] * 13:19 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291966{{!}}Modify various configurations for English Wikibooks (T426992)]] (duration: 15m 53s) * 13:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92875 and previous config saved to /var/cache/conftool/dbconfig/20260525-131714-fceratto.json * 13:12 mszwarc@deploy1003: vadymts1, mszwarc: Continuing with deployment * 13:12 jiji@cumin1003: START - Cookbook sre.dns.netbox * 13:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92873 and previous config saved to /var/cache/conftool/dbconfig/20260525-131023-fceratto.json * 13:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2211.codfw.wmnet with reason: Maintenance * 13:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92872 and previous config saved to /var/cache/conftool/dbconfig/20260525-130950-fceratto.json * 13:07 mszwarc@deploy1003: vadymts1, mszwarc: Backport for [[gerrit:1291966{{!}}Modify various configurations for English Wikibooks (T426992)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:03 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1291966{{!}}Modify various configurations for English Wikibooks (T426992)]] * 12:59 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1162: Reboot * 12:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P92870 and previous config saved to /var/cache/conftool/dbconfig/20260525-125942-fceratto.json * 12:59 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1162: Reboot * 12:59 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1162: Reboot * 12:58 kart_: Updated cxserver to 2026-05-24-103047-production ([[phab:T426808|T426808]], [[phab:T373418|T373418]]) * 12:56 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply * 12:56 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply * 12:54 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db1162: Reboot * 12:54 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1162: Reboot * 12:54 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply * 12:53 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply * 12:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1162.eqiad.wmnet with reason: Reboot * 12:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P92868 and previous config saved to /var/cache/conftool/dbconfig/20260525-124934-fceratto.json * 12:40 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply * 12:39 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply * 12:39 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1038.eqiad.wmnet * 12:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92867 and previous config saved to /var/cache/conftool/dbconfig/20260525-123927-fceratto.json * 12:32 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2192 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92866 and previous config saved to /var/cache/conftool/dbconfig/20260525-123239-fceratto.json * 12:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2192.codfw.wmnet with reason: Maintenance * 12:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92865 and previous config saved to /var/cache/conftool/dbconfig/20260525-123208-fceratto.json * 12:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P92864 and previous config saved to /var/cache/conftool/dbconfig/20260525-122201-fceratto.json * 12:17 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1037.eqiad.wmnet * 12:17 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:17 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1037.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 12:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P92863 and previous config saved to /var/cache/conftool/dbconfig/20260525-121153-fceratto.json * 12:10 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1037.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 12:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92862 and previous config saved to /var/cache/conftool/dbconfig/20260525-120145-fceratto.json * 11:58 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2178 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92861 and previous config saved to /var/cache/conftool/dbconfig/20260525-115504-fceratto.json * 11:54 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2178.codfw.wmnet with reason: Maintenance * 11:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92860 and previous config saved to /var/cache/conftool/dbconfig/20260525-115434-fceratto.json * 11:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P92859 and previous config saved to /var/cache/conftool/dbconfig/20260525-114426-fceratto.json * 11:43 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1037.eqiad.wmnet * 11:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P92858 and previous config saved to /var/cache/conftool/dbconfig/20260525-113419-fceratto.json * 11:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2160.codfw.wmnet with OS trixie * 11:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92857 and previous config saved to /var/cache/conftool/dbconfig/20260525-112411-fceratto.json * 11:17 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2171 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92856 and previous config saved to /var/cache/conftool/dbconfig/20260525-111717-fceratto.json * 11:17 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: Maintenance * 11:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92855 and previous config saved to /var/cache/conftool/dbconfig/20260525-111648-fceratto.json * 11:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P92854 and previous config saved to /var/cache/conftool/dbconfig/20260525-110640-fceratto.json * 11:05 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2160.codfw.wmnet with reason: host reimage * 11:00 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2160.codfw.wmnet with reason: host reimage * 10:58 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:57 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:56 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P92853 and previous config saved to /var/cache/conftool/dbconfig/20260525-105633-fceratto.json * 10:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92852 and previous config saved to /var/cache/conftool/dbconfig/20260525-104625-fceratto.json * 10:43 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2160.codfw.wmnet with OS trixie * 10:41 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc3 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92851 and previous config saved to /var/cache/conftool/dbconfig/20260525-104141-marostegui.json * 10:40 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1023 to pc3 as master [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92850 and previous config saved to /var/cache/conftool/dbconfig/20260525-104055-marostegui.json * 10:40 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1023 to dbctl', diff saved to https://phabricator.wikimedia.org/P92849 and previous config saved to /var/cache/conftool/dbconfig/20260525-104027-marostegui.json * 10:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92848 and previous config saved to /var/cache/conftool/dbconfig/20260525-103944-fceratto.json * 10:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance * 10:31 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply * 10:30 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply * 10:27 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 10:18 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 10:16 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1011.eqiad.wmnet * 10:08 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1011.eqiad.wmnet * 10:08 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1007.eqiad.wmnet * 09:59 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1007.eqiad.wmnet * 09:59 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1006.eqiad.wmnet * 09:57 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:49 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1006.eqiad.wmnet * 09:48 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:46 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:45 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:40 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:40 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:28 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:17 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:13 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92847 and previous config saved to /var/cache/conftool/dbconfig/20260525-091302-fceratto.json * 09:12 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231', diff saved to https://phabricator.wikimedia.org/P92846 and previous config saved to /var/cache/conftool/dbconfig/20260525-090255-fceratto.json * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231', diff saved to https://phabricator.wikimedia.org/P92845 and previous config saved to /var/cache/conftool/dbconfig/20260525-085247-fceratto.json * 08:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92844 and previous config saved to /var/cache/conftool/dbconfig/20260525-084239-fceratto.json * 08:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92843 and previous config saved to /var/cache/conftool/dbconfig/20260525-083540-fceratto.json * 08:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2231.codfw.wmnet with reason: Maintenance * 08:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92842 and previous config saved to /var/cache/conftool/dbconfig/20260525-083511-fceratto.json * 08:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215', diff saved to https://phabricator.wikimedia.org/P92841 and previous config saved to /var/cache/conftool/dbconfig/20260525-082504-fceratto.json * 08:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215', diff saved to https://phabricator.wikimedia.org/P92840 and previous config saved to /var/cache/conftool/dbconfig/20260525-081456-fceratto.json * 08:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92839 and previous config saved to /var/cache/conftool/dbconfig/20260525-080448-fceratto.json * 07:57 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2215 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92838 and previous config saved to /var/cache/conftool/dbconfig/20260525-075739-fceratto.json * 07:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2215.codfw.wmnet with reason: Maintenance * 07:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92837 and previous config saved to /var/cache/conftool/dbconfig/20260525-075708-fceratto.json * 07:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196', diff saved to https://phabricator.wikimedia.org/P92836 and previous config saved to /var/cache/conftool/dbconfig/20260525-074700-fceratto.json * 07:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196', diff saved to https://phabricator.wikimedia.org/P92835 and previous config saved to /var/cache/conftool/dbconfig/20260525-073653-fceratto.json * 07:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92834 and previous config saved to /var/cache/conftool/dbconfig/20260525-072645-fceratto.json * 07:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92833 and previous config saved to /var/cache/conftool/dbconfig/20260525-071953-fceratto.json * 07:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2196.codfw.wmnet with reason: Maintenance * 07:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92832 and previous config saved to /var/cache/conftool/dbconfig/20260525-071924-fceratto.json * 07:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186', diff saved to https://phabricator.wikimedia.org/P92831 and previous config saved to /var/cache/conftool/dbconfig/20260525-070917-fceratto.json * 07:03 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2233.codfw.wmnet with OS trixie * 06:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186', diff saved to https://phabricator.wikimedia.org/P92830 and previous config saved to /var/cache/conftool/dbconfig/20260525-065909-fceratto.json * 06:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92829 and previous config saved to /var/cache/conftool/dbconfig/20260525-064902-fceratto.json * 06:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92828 and previous config saved to /var/cache/conftool/dbconfig/20260525-064305-fceratto.json * 06:42 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance * 06:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2233.codfw.wmnet with reason: host reimage * 06:35 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2233.codfw.wmnet with reason: host reimage * 06:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2233.codfw.wmnet with OS trixie * 06:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2233.codfw.wmnet with reason: Reimage to Trixie * 06:17 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 06:17 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:15 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2160.codfw.wmnet with reason: Reboot upgrade m2 * 06:15 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2233.codfw.wmnet with reason: Reboot upgrade m2 * 06:08 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy1027.eqiad.wmnet with reason: Reboot * 05:18 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2023.codfw.wmnet,pc[1013,1023].eqiad.wmnet with reason: Maintenance on pc3 * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1013.eqiad.wmnet: Maintenance on pc3 * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 05:17 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 05:17 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1013.eqiad.wmnet: Maintenance on pc3 * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 43s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-24 == * 19:08 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down * 02:06 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 23s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-23 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 35s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-22 == * 23:39 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 23:39 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 23:39 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 23:39 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 23:38 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 23:37 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 23:37 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 23:37 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 22:20 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 22:12 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 22:11 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 20:29 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 20:28 inflatador: bking@deploy1003 set eqiad prod cirrus `node_concurrent_recoveries` up to 7 from 4 [[phab:T426585|T426585]] * 20:27 inflatador: bking@deploy1003 set codfw prod cirrus `node_concurrent_recoveries` back down to 4 from 7 [[phab:T426585|T426585]] * 18:39 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 17:34 topranks: enable ttl protection on esams CRs IBGP session * 17:28 topranks: enable ttl protection on ulsfo CRs IBGP session * 16:50 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 16:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 16:16 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 16:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 16:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 15:58 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 15:14 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 15:02 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 15:02 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 14:34 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudnet2008-dev.codfw.wmnet * 14:34 andrew@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:34 andrew@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2008-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 14:33 andrew@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2008-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 14:33 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb[1020,1022-1025].eqiad.wmnet * 14:29 andrew@cumin2002: START - Cookbook sre.dns.netbox * 14:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 14:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 14:23 andrew@cumin2002: START - Cookbook sre.hosts.decommission for hosts cloudnet2008-dev.codfw.wmnet * 14:23 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudnet2007-dev.codfw.wmnet * 14:23 andrew@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:23 andrew@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2007-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 14:03 andrew@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2007-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 13:59 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb[1020,1022-1025].eqiad.wmnet * 13:58 andrew@cumin2002: START - Cookbook sre.dns.netbox * 13:53 andrew@cumin2002: START - Cookbook sre.hosts.decommission for hosts cloudnet2007-dev.codfw.wmnet * 13:52 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1018.eqiad.wmnet * 13:50 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 13:50 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 13:46 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1018.eqiad.wmnet * 13:25 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for clouddb1018.eqiad.wmnet * 13:25 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1018.eqiad.wmnet * 13:25 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for 6 hosts * 13:16 inflatador: bking@deploy1002 set search_codfw cluster recovery settings from 4 to 7 [[phab:T426560|T426560]] * 13:15 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for 6 hosts * 13:15 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 13:11 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp5017.eqsin.wmnet<nowiki>}</nowiki> and A:cp * 13:11 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5017.eqsin.wmnet * 13:10 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1017.eqiad.wmnet * 13:09 elukey: uploaded spicerack_12.6.0 to apt.wikimedia.org bookworm-wikimedia * 13:08 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for clouddb1017.eqiad.wmnet * 12:59 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp5017.eqsin.wmnet<nowiki>}</nowiki> and A:cp * 12:57 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp308[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 12:57 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3081.esams.wmnet * 12:54 isaranto@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 12:41 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 12:15 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3080.esams.wmnet * 12:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 12:03 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp308[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[2-3].esams.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3073.esams.wmnet * 11:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2154: Migration of db2154.codfw.wmnet completed * 11:19 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3072.esams.wmnet * 11:15 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 11:11 fnegri@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb1017.eqiad.wmnet with reason: Rebooting clouddb1017 * 11:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1172: Migration of db1172.eqiad.wmnet completed * 11:07 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[2-3].esams.wmnet<nowiki>}</nowiki> and A:cp * 11:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1058.eqiad.wmnet * 11:01 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 11:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3079.esams.wmnet * 10:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1058.eqiad.wmnet * 10:55 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 10:55 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 10:48 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 10:47 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 10:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1024.eqiad.wmnet * 10:43 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:43 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:43 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:42 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:42 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:42 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2154: Migration of db2154.codfw.wmnet completed * 10:42 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:41 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1024.eqiad.wmnet * 10:37 moritzm: remove ganeti1024 foom eqiad Ganeti cluster [[phab:T424680|T424680]] * 10:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2154.codfw.wmnet with OS trixie * 10:31 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2010.codfw.wmnet with OS trixie * 10:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1024.eqiad.wmnet * 10:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1172: Migration of db1172.eqiad.wmnet completed * 10:19 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3078.esams.wmnet * 10:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2154.codfw.wmnet with reason: host reimage * 10:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1172.eqiad.wmnet with OS trixie * 10:15 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1017.eqiad.wmnet * 10:13 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2154.codfw.wmnet with reason: host reimage * 10:07 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 10:06 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 10:06 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3071.esams.wmnet * 09:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1172.eqiad.wmnet with reason: host reimage * 09:56 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2154.codfw.wmnet with OS trixie * 09:55 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage * 09:53 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1172.eqiad.wmnet with reason: host reimage * 09:51 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage * 09:39 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2154: Upgrading db2154.codfw.wmnet * 09:39 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2154: Upgrading db2154.codfw.wmnet * 09:38 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:38 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1172.eqiad.wmnet with OS trixie * 09:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1172: Upgrading db1172.eqiad.wmnet * 09:34 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1172: Upgrading db1172.eqiad.wmnet * 09:34 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:34 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2009.codfw.wmnet with OS trixie * 09:33 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS trixie * 09:26 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 09:26 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 09:26 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3070.esams.wmnet * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 09:16 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie * 09:14 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 09:11 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[6-7].esams.wmnet<nowiki>}</nowiki> and A:cp * 09:11 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3077.esams.wmnet * 09:04 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 09:03 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie * 08:47 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 08:46 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie * 08:40 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 08:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 08:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 08:30 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3076.esams.wmnet * 08:18 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[6-7].esams.wmnet<nowiki>}</nowiki> and A:cp * 08:15 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti1058.eqiad.wmnet on all recursors * 08:15 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:15 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change records for ganeti1058 - cmooney@cumin1003" * 08:15 cmooney@cumin1003: START - Cookbook sre.dns.wipe-cache ganeti1058.eqiad.wmnet on all recursors * 08:15 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change records for ganeti1058 - cmooney@cumin1003" * 08:09 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 08:07 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp306[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 08:07 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3069.esams.wmnet * 08:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 08:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 07:31 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1024.eqiad.wmnet * 07:26 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3068.esams.wmnet * 07:14 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp306[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 07:11 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1057.eqiad.wmnet to cluster eqiad and group A * 07:10 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3075.esams.wmnet<nowiki>}</nowiki> and A:cp * 07:10 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3075.esams.wmnet * 07:06 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1057.eqiad.wmnet to cluster eqiad and group A * 07:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1057.eqiad.wmnet * 07:02 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1057 * 07:01 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1057 * 06:58 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3075.esams.wmnet<nowiki>}</nowiki> and A:cp * 06:58 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3067.esams.wmnet<nowiki>}</nowiki> and A:cp * 06:58 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3067.esams.wmnet * 06:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1057.eqiad.wmnet * 06:46 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3067.esams.wmnet<nowiki>}</nowiki> and A:cp * 06:13 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1024.eqiad.wmnet * 06:08 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1024.eqiad.wmnet * 06:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org * 06:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org * 05:25 marostegui@dns1004: END - running authdns-update * 05:24 marostegui@dns1004: START - running authdns-update * 05:23 marostegui: Failover m5-master [[phab:T426633|T426633]] * 05:19 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy1028.eqiad.wmnet with reason: Reboot * 05:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy2005.codfw.wmnet with reason: Reboot * 05:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1012.eqiad.wmnet * 05:11 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 05:11 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1012.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 05:06 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1012.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 05:03 marostegui@cumin1003: START - Cookbook sre.dns.netbox * 04:56 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1012.eqiad.wmnet == 2026-05-21 == * 23:43 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290954{{!}}Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957{{!}}Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958{{!}}Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960{{!}}Drop unused wpReportIncidentLocalLinks]] (duration: 06m 42s) * 23:38 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 23:38 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1290954{{!}}Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957{{!}}Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958{{!}}Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960{{!}}Drop unused wpReportIncidentLocalLinks]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified * 23:36 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1290954{{!}}Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957{{!}}Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958{{!}}Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960{{!}}Drop unused wpReportIncidentLocalLinks]] * 22:26 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul2002.codfw.wmnet with OS trixie * 22:08 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul2002.codfw.wmnet with reason: host reimage * 22:03 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul2002.codfw.wmnet with reason: host reimage * 22:02 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 21:49 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 21:49 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 21:44 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host zuul2002.codfw.wmnet with OS trixie * 21:25 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:25 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 20:26 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 20:16 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 19:22 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-reboot (exit_code=0) rolling reboot on A:restbase * 19:10 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:59 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:53 papaul: rebooting msw1-codfw * 18:50 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:39 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:52 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:52 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:50 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:49 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:49 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:48 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:46 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 17:46 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 17:43 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:43 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:43 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:42 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:42 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:41 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:41 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:41 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:41 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:41 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:41 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:41 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:40 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:40 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:40 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:39 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 * 17:39 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:38 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down * 17:37 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 17:36 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:36 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:30 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:25 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:25 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:24 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:23 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:22 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1016.eqiad.wmnet * 17:22 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:14 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2031.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:14 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2030.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:13 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1016.eqiad.wmnet * 17:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:08 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repool pc2 ([[phab:T421705|T421705]])', diff saved to https://phabricator.wikimedia.org/P92810 and previous config saved to /var/cache/conftool/dbconfig/20260521-170823-ladsgroup.json * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:07 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2031.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:07 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2030.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:06 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:03 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:03 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:00 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2029 * 16:58 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2031 * 16:58 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:58 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 * 16:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 * 16:55 papaul: rebooting msw-d3-codfw * 16:55 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 16:52 papaul: rebooting msw-c7-codfw * 16:51 papaul: rebooting msw-c6-codfw * 16:48 papaul: rebooting msw-b7-codfw * 16:48 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1014.eqiad.wmnet * 16:45 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1014.eqiad.wmnet * 16:43 papaul: rebooting msw-b6-codfw * 16:40 papaul: rebooting msw-a1-codfw * 16:37 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 16:37 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1014.eqiad.wmnet * 16:37 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:36 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:35 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 16:35 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2030 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2030 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 * 16:34 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 16:34 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:33 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2028 to codfw - jhancock@cumin2002" * 16:33 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2028 to codfw - jhancock@cumin2002" * 16:26 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 16:24 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on pc1022.eqiad.wmnet with reason: Move to nftables * 16:24 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on pc2022.codfw.wmnet with reason: Move to nftables * 16:18 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2048: Repooling * 16:18 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depool pc2 ([[phab:T421705|T421705]])', diff saved to https://phabricator.wikimedia.org/P92807 and previous config saved to /var/cache/conftool/dbconfig/20260521-161808-ladsgroup.json * 16:15 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:15 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:15 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:15 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:52 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 15:42 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es2048: Repooling * 15:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92804 and previous config saved to /var/cache/conftool/dbconfig/20260521-154108-fceratto.json * 15:39 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:38 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:34 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:34 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:34 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:34 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:34 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92803 and previous config saved to /var/cache/conftool/dbconfig/20260521-153400-fceratto.json * 15:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2048.codfw.wmnet with reason: Maintenance * 15:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92802 and previous config saved to /var/cache/conftool/dbconfig/20260521-153331-fceratto.json * 15:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:25 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:24 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:24 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:24 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:24 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040', diff saved to https://phabricator.wikimedia.org/P92801 and previous config saved to /var/cache/conftool/dbconfig/20260521-152323-fceratto.json * 15:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1045.eqiad.wmnet * 15:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1045.eqiad.wmnet * 15:19 claime: Enabling puppet on A:cp-text - [[phab:T426323|T426323]] * 15:15 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1045.eqiad.wmnet * 15:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040', diff saved to https://phabricator.wikimedia.org/P92800 and previous config saved to /var/cache/conftool/dbconfig/20260521-151316-fceratto.json * 15:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1014.eqiad.wmnet * 15:11 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1045.eqiad.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2034.codfw.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2034.codfw.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1037.eqiad.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1037.eqiad.wmnet * 15:07 elukey@cumin1003: END (PASS) - Cookbook sre.misc-clusters.restart-reboot-config-master (exit_code=0) rolling reboot on A:config-master * 15:06 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1014.eqiad.wmnet * 15:05 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors * 15:05 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors * 15:04 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290805{{!}}hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] (duration: 10m 11s) * 15:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92799 and previous config saved to /var/cache/conftool/dbconfig/20260521-150308-fceratto.json * 15:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1037.eqiad.wmnet * 15:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2034.codfw.wmnet * 15:00 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors * 15:00 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors * 15:00 elukey@cumin1003: START - Cookbook sre.misc-clusters.restart-reboot-config-master rolling reboot on A:config-master * 15:00 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 15:00 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-lab1002.eqiad.wmnet * 14:59 elukey@cumin1003: END (PASS) - Cookbook sre.pki.restart-reboot (exit_code=0) rolling reboot on A:pki * 14:57 claime: Disabling puppet on A:cp-text - [[phab:T426323|T426323]] * 14:56 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1290805{{!}}hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:55 klausman@cumin1003: START - Cookbook sre.hosts.reboot-single for host ml-lab1002.eqiad.wmnet * 14:54 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-build1001.eqiad.wmnet * 14:54 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1290805{{!}}hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] * 14:54 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet * 14:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1013.eqiad.wmnet * 14:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1037.eqiad.wmnet * 14:53 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1028.eqiad.wmnet * 14:53 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>ml-serve1001.eqiad.wmnet<nowiki>}</nowiki> and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) * 14:53 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1001.eqiad.wmnet * 14:53 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1001.eqiad.wmnet * 14:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1028.eqiad.wmnet * 14:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92798 and previous config saved to /var/cache/conftool/dbconfig/20260521-145132-fceratto.json * 14:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2040.codfw.wmnet with reason: Maintenance * 14:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92797 and previous config saved to /var/cache/conftool/dbconfig/20260521-145103-fceratto.json * 14:50 klausman@cumin1003: START - Cookbook sre.hosts.reboot-single for host ml-build1001.eqiad.wmnet * 14:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: Migration of db2241.codfw.wmnet completed * 14:48 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1001.eqiad.wmnet * 14:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1013.eqiad.wmnet * 14:46 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1028.eqiad.wmnet * 14:45 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:44 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:42 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1001.eqiad.wmnet * 14:42 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>ml-serve1001.eqiad.wmnet<nowiki>}</nowiki> and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) * 14:42 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1028.eqiad.wmnet * 14:42 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-worker-eqiad * 14:42 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1011.eqiad.wmnet * 14:42 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1011.eqiad.wmnet * 14:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039', diff saved to https://phabricator.wikimedia.org/P92795 and previous config saved to /var/cache/conftool/dbconfig/20260521-144055-fceratto.json * 14:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1012.eqiad.wmnet * 14:38 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet. on all recursors * 14:37 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet. on all recursors * 14:37 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1011.eqiad.wmnet * 14:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1027.eqiad.wmnet * 14:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1027.eqiad.wmnet * 14:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1011.eqiad.wmnet * 14:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1012.eqiad.wmnet * 14:32 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1010.eqiad.wmnet * 14:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1010.eqiad.wmnet * 14:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039', diff saved to https://phabricator.wikimedia.org/P92793 and previous config saved to /var/cache/conftool/dbconfig/20260521-143045-fceratto.json * 14:30 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet. on all recursors * 14:30 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet. on all recursors * 14:29 elukey@cumin1003: START - Cookbook sre.pki.restart-reboot rolling reboot on A:pki * 14:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1027.eqiad.wmnet * 14:27 slyngshede@cumin1003: END (FAIL) - Cookbook sre.cdn.roll-reboot (exit_code=1) rolling reboot on P<nowiki>{</nowiki>cp601[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 14:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1027.eqiad.wmnet * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1054.eqiad.wmnet * 14:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1054.eqiad.wmnet * 14:24 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1010.eqiad.wmnet * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1011.eqiad.wmnet * 14:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92792 and previous config saved to /var/cache/conftool/dbconfig/20260521-142037-fceratto.json * 14:19 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1054.eqiad.wmnet * 14:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1054.eqiad.wmnet * 14:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1053.eqiad.wmnet * 14:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1053.eqiad.wmnet * 14:14 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1010.eqiad.wmnet * 14:14 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1009.eqiad.wmnet * 14:14 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1009.eqiad.wmnet * 14:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:13 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1011.eqiad.wmnet * 14:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:12 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2218: repool after maintenance * 14:11 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1053.eqiad.wmnet * 14:09 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92789 and previous config saved to /var/cache/conftool/dbconfig/20260521-140906-fceratto.json * 14:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2039.codfw.wmnet with reason: Maintenance * 14:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92788 and previous config saved to /var/cache/conftool/dbconfig/20260521-140837-fceratto.json * 14:08 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1009.eqiad.wmnet * 14:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:07 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1053.eqiad.wmnet * 14:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1035.eqiad.wmnet * 14:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1035.eqiad.wmnet * 14:04 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2241: Migration of db2241.codfw.wmnet completed * 14:03 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1009.eqiad.wmnet * 14:03 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1008.eqiad.wmnet * 14:03 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1008.eqiad.wmnet * 14:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2241.codfw.wmnet with OS trixie * 13:59 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 13:59 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1035.eqiad.wmnet * 13:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92786 and previous config saved to /var/cache/conftool/dbconfig/20260521-135830-fceratto.json * 13:58 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1008.eqiad.wmnet * 13:53 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1008.eqiad.wmnet * 13:53 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1007.eqiad.wmnet * 13:53 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1007.eqiad.wmnet * 13:51 Lucas_WMDE: UTC afternoon backport+config window done * 13:51 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290743{{!}}composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347{{!}}Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342{{!}}fix: simplify to show only one icon type for password reveal (T419413)]] (duration: 07m 20s) * 13:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92784 and previous config saved to /var/cache/conftool/dbconfig/20260521-134822-fceratto.json * 13:48 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1007.eqiad.wmnet * 13:47 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 13:46 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Continuing with deployment * 13:45 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 13:45 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Backport for [[gerrit:1290743{{!}}composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347{{!}}Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342{{!}}fix: simplify to show only one icon type for password reveal (T419413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes * 13:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2241.codfw.wmnet with reason: host reimage * 13:44 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 13:43 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1290743{{!}}composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347{{!}}Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342{{!}}fix: simplify to show only one icon type for password reveal (T419413)]] * 13:43 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 13:43 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1007.eqiad.wmnet * 13:42 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1006.eqiad.wmnet * 13:42 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1006.eqiad.wmnet * 13:41 dbrant@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290035{{!}}docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] (duration: 06m 52s) * 13:41 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 13:40 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2241.codfw.wmnet with reason: host reimage * 13:39 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1035.eqiad.wmnet * 13:38 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in codfw/ml-serve-codfw: maintenance * 13:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92782 and previous config saved to /var/cache/conftool/dbconfig/20260521-133815-fceratto.json * 13:37 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1006.eqiad.wmnet * 13:37 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in codfw/ml-serve-codfw: maintenance * 13:37 dbrant@deploy1003: dbrant: Continuing with deployment * 13:36 dbrant@deploy1003: dbrant: Backport for [[gerrit:1290035{{!}}docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1032.eqiad.wmnet * 13:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1032.eqiad.wmnet * 13:35 dbrant@deploy1003: Started scap sync-world: Backport for [[gerrit:1290035{{!}}docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] * 13:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1006.eqiad.wmnet * 13:32 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1005.eqiad.wmnet * 13:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1005.eqiad.wmnet * 13:31 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290014{{!}}Enable AG on phase 2 wikis (T426871)]] (duration: 09m 11s) * 13:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92781 and previous config saved to /var/cache/conftool/dbconfig/20260521-133116-fceratto.json * 13:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1048.eqiad.wmnet with reason: Maintenance * 13:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92780 and previous config saved to /var/cache/conftool/dbconfig/20260521-133048-fceratto.json * 13:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1032.eqiad.wmnet * 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1032.eqiad.wmnet * 13:27 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1005.eqiad.wmnet * 13:27 sbisson@deploy1003: sbisson: Continuing with deployment * 13:27 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2218: repool after maintenance * 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1031.eqiad.wmnet * 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1031.eqiad.wmnet * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:25 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2241.codfw.wmnet with OS trixie * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:24 sbisson@deploy1003: sbisson: Backport for [[gerrit:1290014{{!}}Enable AG on phase 2 wikis (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:23 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: Upgrading db2241.codfw.wmnet * 13:23 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2241: Upgrading db2241.codfw.wmnet * 13:23 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:22 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1290014{{!}}Enable AG on phase 2 wikis (T426871)]] * 13:22 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1005.eqiad.wmnet * 13:22 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1004.eqiad.wmnet * 13:22 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1004.eqiad.wmnet * 13:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040', diff saved to https://phabricator.wikimedia.org/P92778 and previous config saved to /var/cache/conftool/dbconfig/20260521-132041-fceratto.json * 13:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1031.eqiad.wmnet * 13:20 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290088{{!}}Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032{{!}}Enable 'flood' user group at en.wikiversity (T426882)]] (duration: 11m 55s) * 13:18 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet * 13:17 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1018.eqiad.wmnet with OS trixie * 13:16 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1031.eqiad.wmnet * 13:16 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1039: Repooling * 13:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1030.eqiad.wmnet * 13:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1030.eqiad.wmnet * 13:15 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Continuing with deployment * 13:15 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1004.eqiad.wmnet * 13:14 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet * 13:11 eevans@cumin1003: START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:restbase * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . * 13:10 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1004.eqiad.wmnet * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . * 13:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040', diff saved to https://phabricator.wikimedia.org/P92776 and previous config saved to /var/cache/conftool/dbconfig/20260521-131033-fceratto.json * 13:10 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1003.eqiad.wmnet * 13:10 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1003.eqiad.wmnet * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 13:10 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2241 [[phab:T426936|T426936]]', diff saved to https://phabricator.wikimedia.org/P92775 and previous config saved to /var/cache/conftool/dbconfig/20260521-131025-cwilliams.json * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'readability' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'logo-detection' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . * 13:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1030.eqiad.wmnet * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . * 13:10 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Backport for [[gerrit:1290088{{!}}Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032{{!}}Enable 'flood' user group at en.wikiversity (T426882)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'llm' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . * 13:08 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1290088{{!}}Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032{{!}}Enable 'flood' user group at en.wikiversity (T426882)]] * 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet * 13:06 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp601[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 13:06 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3074.esams.wmnet<nowiki>}</nowiki> and A:cp * 13:06 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3074.esams.wmnet * 13:06 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2162 to x3 primary [[phab:T426936|T426936]]', diff saved to https://phabricator.wikimedia.org/P92774 and previous config saved to /var/cache/conftool/dbconfig/20260521-130609-cwilliams.json * 13:04 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 13:04 cezmunsta: Starting x3 codfw failover from db2241 to db2162 - [[phab:T426936|T426936]] * 13:04 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1003.eqiad.wmnet * 13:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1030.eqiad.wmnet * 13:03 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet * 13:00 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 13:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92772 and previous config saved to /var/cache/conftool/dbconfig/20260521-130018-fceratto.json * 12:59 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1003.eqiad.wmnet * 12:59 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1018.eqiad.wmnet with reason: host reimage * 12:59 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1002.eqiad.wmnet * 12:59 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1002.eqiad.wmnet * 12:58 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:57 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:56 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2162 with weight 0 [[phab:T426936|T426936]]', diff saved to https://phabricator.wikimedia.org/P92771 and previous config saved to /var/cache/conftool/dbconfig/20260521-125645-cwilliams.json * 12:56 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Primary switchover x3 [[phab:T426936|T426936]] * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:55 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1029.eqiad.wmnet * 12:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1029.eqiad.wmnet * 12:54 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3074.esams.wmnet<nowiki>}</nowiki> and A:cp * 12:54 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1002.eqiad.wmnet * 12:54 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp600[7-8].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 12:54 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6008.drmrs.wmnet * 12:53 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:52 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1018.eqiad.wmnet with reason: host reimage * 12:51 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:49 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1002.eqiad.wmnet * 12:49 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-worker-eqiad * 12:48 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1029.eqiad.wmnet * 12:48 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3066.esams.wmnet<nowiki>}</nowiki> and A:cp * 12:48 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3066.esams.wmnet * 12:47 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:47 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92770 and previous config saved to /var/cache/conftool/dbconfig/20260521-124707-fceratto.json * 12:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1040.eqiad.wmnet with reason: Maintenance * 12:46 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es1039: Repooling * 12:46 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:45 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1029.eqiad.wmnet * 12:45 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:44 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:43 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:43 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290727{{!}}hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] (duration: 07m 54s) * 12:42 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92768 and previous config saved to /var/cache/conftool/dbconfig/20260521-124014-fceratto.json * 12:39 kharlan@deploy1003: kharlan: Continuing with deployment * 12:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1052.eqiad.wmnet * 12:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1052.eqiad.wmnet * 12:37 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1018.eqiad.wmnet with OS trixie * 12:37 kharlan@deploy1003: kharlan: Backport for [[gerrit:1290727{{!}}hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:36 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:36 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3066.esams.wmnet<nowiki>}</nowiki> and A:cp * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:35 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1290727{{!}}hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:34 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1017.eqiad.wmnet with OS trixie * 12:34 kart_: Updated cxserver to 2026-05-20-034002-production ([[phab:T388690|T388690]], [[phab:T404295|T404295]], [[phab:T391703|T391703]], [[phab:T426605|T426605]]) * 12:34 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet * 12:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1052.eqiad.wmnet * 12:30 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply * 12:30 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply * 12:30 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet * 12:29 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:29 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92767 and previous config saved to /var/cache/conftool/dbconfig/20260521-122905-fceratto.json * 12:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1039.eqiad.wmnet with reason: Maintenance * 12:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92766 and previous config saved to /var/cache/conftool/dbconfig/20260521-122839-fceratto.json * 12:27 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply * 12:27 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply * 12:26 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:23 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-staging-worker * 12:23 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2003.codfw.wmnet * 12:23 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2003.codfw.wmnet * 12:22 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1052.eqiad.wmnet * 12:21 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply * 12:21 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply * 12:21 moritzm: installing nginx security updates * 12:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1051.eqiad.wmnet * 12:20 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in codfw/ml-serve-codfw: maintenance * 12:19 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1017.eqiad.wmnet with reason: host reimage * 12:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1051.eqiad.wmnet * 12:19 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in codfw/ml-serve-codfw: maintenance * 12:19 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in codfw/ml-staging-codfw: maintenance * 12:19 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in codfw/ml-staging-codfw: maintenance * 12:19 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in codfw/ml-staging-codfw: maintenance * 12:18 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in codfw/ml-staging-codfw: maintenance * 12:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047', diff saved to https://phabricator.wikimedia.org/P92765 and previous config saved to /var/cache/conftool/dbconfig/20260521-121832-fceratto.json * 12:17 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2003.codfw.wmnet * 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet * 12:15 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1017.eqiad.wmnet with reason: host reimage * 12:14 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1051.eqiad.wmnet * 12:13 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6007.drmrs.wmnet * 12:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet * 12:10 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1051.eqiad.wmnet * 12:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047', diff saved to https://phabricator.wikimedia.org/P92764 and previous config saved to /var/cache/conftool/dbconfig/20260521-120824-fceratto.json * 12:07 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2003.codfw.wmnet * 12:07 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2002.codfw.wmnet * 12:07 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2002.codfw.wmnet * 12:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1050.eqiad.wmnet * 12:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1050.eqiad.wmnet * 12:02 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp600[7-8].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp601[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6014.drmrs.wmnet * 12:00 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1017.eqiad.wmnet with OS trixie * 12:00 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2002.codfw.wmnet * 11:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1002.wikimedia.org * 11:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92763 and previous config saved to /var/cache/conftool/dbconfig/20260521-115817-fceratto.json * 11:57 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1050.eqiad.wmnet * 11:53 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt1002.wikimedia.org * 11:51 taavi: disabling puppet on C:bird to roll out {{Gerrit|1289919}} * 11:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92762 and previous config saved to /var/cache/conftool/dbconfig/20260521-115112-fceratto.json * 11:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2047.codfw.wmnet with reason: Maintenance * 11:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1050.eqiad.wmnet * 11:50 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2002.codfw.wmnet * 11:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92761 and previous config saved to /var/cache/conftool/dbconfig/20260521-115043-fceratto.json * 11:50 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2001.codfw.wmnet * 11:50 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2001.codfw.wmnet * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1049.eqiad.wmnet * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt2002.wikimedia.org * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1049.eqiad.wmnet * 11:45 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2001.codfw.wmnet * 11:45 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker-exp1001.eqiad.wmnet * 11:44 kartik@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 11:44 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1049.eqiad.wmnet * 11:43 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt2002.wikimedia.org * 11:42 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet * 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet * 11:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037', diff saved to https://phabricator.wikimedia.org/P92760 and previous config saved to /var/cache/conftool/dbconfig/20260521-114036-fceratto.json * 11:39 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker-exp1001.eqiad.wmnet * 11:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker-exp2001.codfw.wmnet * 11:38 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host testreduce1002.eqiad.wmnet * 11:37 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1049.eqiad.wmnet * 11:36 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet * 11:36 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet * 11:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1038.eqiad.wmnet * 11:35 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2001.codfw.wmnet * 11:35 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-staging-worker * 11:35 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet * 11:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1038.eqiad.wmnet * 11:34 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host testreduce1002.eqiad.wmnet * 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker-exp2001.codfw.wmnet * 11:32 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet * 11:31 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet * 11:30 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt-staging2001.codfw.wmnet * 11:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037', diff saved to https://phabricator.wikimedia.org/P92759 and previous config saved to /var/cache/conftool/dbconfig/20260521-113028-fceratto.json * 11:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2014.codfw.wmnet * 11:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1038.eqiad.wmnet * 11:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt-staging2001.codfw.wmnet * 11:26 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet * 11:24 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1038.eqiad.wmnet * 11:24 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1034.eqiad.wmnet * 11:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1034.eqiad.wmnet * 11:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2014.codfw.wmnet * 11:20 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6013.drmrs.wmnet * 11:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92758 and previous config saved to /var/cache/conftool/dbconfig/20260521-112021-fceratto.json * 11:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1034.eqiad.wmnet * 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling reboot on A:ldap-replicas-eqiad * 11:13 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2013.codfw.wmnet * 11:11 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1034.eqiad.wmnet * 11:09 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp601[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92757 and previous config saved to /var/cache/conftool/dbconfig/20260521-110851-fceratto.json * 11:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2037.codfw.wmnet with reason: Maintenance * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92756 and previous config saved to /var/cache/conftool/dbconfig/20260521-110822-fceratto.json * 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1033.eqiad.wmnet * 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1033.eqiad.wmnet * 11:05 jmm@cumin2002: START - Cookbook sre.ldap.roll-restart-reboot-replica rolling reboot on A:ldap-replicas-eqiad * 11:05 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2013.codfw.wmnet * 11:04 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp600[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 11:04 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6006.drmrs.wmnet * 11:02 jmm@cumin2002: END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling reboot on A:ldap-replicas-codfw * 11:00 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1033.eqiad.wmnet * 10:59 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1016.eqiad.wmnet with reason: host reimage * 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036', diff saved to https://phabricator.wikimedia.org/P92753 and previous config saved to /var/cache/conftool/dbconfig/20260521-105815-fceratto.json * 10:57 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1033.eqiad.wmnet * 10:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1044.eqiad.wmnet * 10:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1044.eqiad.wmnet * 10:55 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1016.eqiad.wmnet with reason: host reimage * 10:54 jmm@cumin2002: START - Cookbook sre.ldap.roll-restart-reboot-replica rolling reboot on A:ldap-replicas-codfw * 10:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2012.codfw.wmnet * 10:51 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:51 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:51 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1044.eqiad.wmnet * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036', diff saved to https://phabricator.wikimedia.org/P92752 and previous config saved to /var/cache/conftool/dbconfig/20260521-104807-fceratto.json * 10:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2012.codfw.wmnet * 10:46 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1044.eqiad.wmnet * 10:44 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290709{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] (duration: 08m 02s) * 10:43 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:41 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:40 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet * 10:40 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:39 jiji@deploy1003: jiji: Continuing with deployment * 10:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92751 and previous config saved to /var/cache/conftool/dbconfig/20260521-103759-fceratto.json * 10:37 jiji@deploy1003: jiji: Backport for [[gerrit:1290709{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:36 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1290709{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] * 10:35 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet * 10:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1043.eqiad.wmnet * 10:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1043.eqiad.wmnet * 10:34 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:29 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 10:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1043.eqiad.wmnet * 10:27 dcausse: [[phab:T423993|T423993]]: reindexing all archive indices * 10:27 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . * 10:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92749 and previous config saved to /var/cache/conftool/dbconfig/20260521-102630-fceratto.json * 10:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance * 10:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1043.eqiad.wmnet * 10:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92748 and previous config saved to /var/cache/conftool/dbconfig/20260521-102601-fceratto.json * 10:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2011.codfw.wmnet * 10:24 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6005.drmrs.wmnet * 10:22 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1042.eqiad.wmnet * 10:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1042.eqiad.wmnet * 10:17 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2011.codfw.wmnet * 10:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1042.eqiad.wmnet * 10:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047', diff saved to https://phabricator.wikimedia.org/P92747 and previous config saved to /var/cache/conftool/dbconfig/20260521-101552-fceratto.json * 10:15 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:14 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . * 10:13 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1042.eqiad.wmnet * 10:13 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1041.eqiad.wmnet * 10:12 moritzm: installing postgresql security updates * 10:12 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp600[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 10:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1041.eqiad.wmnet * 10:10 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet * 10:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org * 10:09 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 10:08 fnegri@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb1013.eqiad.wmnet * 10:08 fnegri@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb1013.eqiad.wmnet * 10:07 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1013.eqiad.wmnet * 10:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1041.eqiad.wmnet * 10:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047', diff saved to https://phabricator.wikimedia.org/P92746 and previous config saved to /var/cache/conftool/dbconfig/20260521-100545-fceratto.json * 10:05 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet * 10:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1041.eqiad.wmnet * 10:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1040.eqiad.wmnet * 10:04 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet * 10:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1040.eqiad.wmnet * 10:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org * 10:01 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1013.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 10:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1040.eqiad.wmnet * 10:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org * 09:59 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet * 09:58 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-codfw * 09:58 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2005.codfw.wmnet * 09:58 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2005.codfw.wmnet * 09:56 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1040.eqiad.wmnet * 09:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1039.eqiad.wmnet * 09:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1039.eqiad.wmnet * 09:56 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 09:56 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:55 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ml-serve1013.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 09:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92745 and previous config saved to /var/cache/conftool/dbconfig/20260521-095536-fceratto.json * 09:54 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1384.eqiad.wmnet * 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org * 09:54 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:54 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:52 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2005.codfw.wmnet * 09:52 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2005.codfw.wmnet * 09:52 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply * 09:52 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2004.codfw.wmnet * 09:52 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2004.codfw.wmnet * 09:51 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply * 09:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1039.eqiad.wmnet * 09:49 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1384.eqiad.wmnet * 09:49 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 09:49 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1383.eqiad.wmnet * 09:48 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1039.eqiad.wmnet * 09:48 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1036.eqiad.wmnet * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92744 and previous config saved to /var/cache/conftool/dbconfig/20260521-094829-fceratto.json * 09:48 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1036.eqiad.wmnet * 09:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1047.eqiad.wmnet with reason: Maintenance * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92743 and previous config saved to /var/cache/conftool/dbconfig/20260521-094801-fceratto.json * 09:47 fnegri@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1013.eqiad.wmnet * 09:47 fnegri@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1013.eqiad.wmnet with reason: Rebooting clouddb1013 [[phab:T426563|T426563]] * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2004.codfw.wmnet * 09:45 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2004.codfw.wmnet * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet * 09:45 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet * 09:45 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet * 09:44 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1383.eqiad.wmnet * 09:44 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 09:44 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1382.eqiad.wmnet * 09:42 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2002.codfw.wmnet * 09:40 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1036.eqiad.wmnet * 09:39 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet * 09:38 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1382.eqiad.wmnet * 09:38 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1381.eqiad.wmnet * 09:38 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1036.eqiad.wmnet * 09:38 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2003.codfw.wmnet * 09:38 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2003.codfw.wmnet * 09:38 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2002.codfw.wmnet * 09:38 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2002.codfw.wmnet * 09:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037', diff saved to https://phabricator.wikimedia.org/P92742 and previous config saved to /var/cache/conftool/dbconfig/20260521-093754-fceratto.json * 09:37 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 09:37 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet * 09:37 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet * 09:37 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet * 09:37 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet * 09:36 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host build2002.codfw.wmnet * 09:36 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 09:35 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp601[1-2].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 09:35 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6012.drmrs.wmnet * 09:34 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet * 09:33 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet * 09:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1381.eqiad.wmnet * 09:33 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1380.eqiad.wmnet * 09:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1023.eqiad.wmnet * 09:31 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet * 09:31 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2002.codfw.wmnet * 09:31 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2002.codfw.wmnet * 09:31 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2001.codfw.wmnet * 09:31 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2001.codfw.wmnet * 09:30 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet * 09:30 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet * 09:30 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet * 09:30 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet * 09:29 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet * 09:29 jayme@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad * 09:29 jayme@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw * 09:29 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet * 09:28 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet * 09:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037', diff saved to https://phabricator.wikimedia.org/P92741 and previous config saved to /var/cache/conftool/dbconfig/20260521-092746-fceratto.json * 09:27 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1380.eqiad.wmnet * 09:27 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1379.eqiad.wmnet * 09:27 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet * 09:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1023.eqiad.wmnet * 09:25 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet * 09:24 jayme@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw * 09:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1056.eqiad.wmnet to cluster eqiad and group A * 09:23 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet * 09:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad * 09:22 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1379.eqiad.wmnet * 09:22 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1378.eqiad.wmnet * 09:21 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2001.codfw.wmnet * 09:21 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2001.codfw.wmnet * 09:21 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-codfw * 09:21 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1056.eqiad.wmnet to cluster eqiad and group A * 09:20 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 09:18 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-jumbo1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:18 moritzm: remove ganeti1023 foom eqiad Ganeti cluster [[phab:T424680|T424680]] * 09:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92740 and previous config saved to /var/cache/conftool/dbconfig/20260521-091738-fceratto.json * 09:16 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1378.eqiad.wmnet * 09:16 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1377.eqiad.wmnet * 09:12 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1377.eqiad.wmnet * 09:12 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1376.eqiad.wmnet * 09:07 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1036: Repooling * 09:07 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1376.eqiad.wmnet * 09:07 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1375.eqiad.wmnet * 09:06 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92738 and previous config saved to /var/cache/conftool/dbconfig/20260521-090609-fceratto.json * 09:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1037.eqiad.wmnet with reason: Maintenance * 09:02 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1375.eqiad.wmnet * 09:01 btullis@cumin1003: START - Cookbook sre.hosts.provision for host kafka-jumbo1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:55 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6011.drmrs.wmnet * 08:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1023.eqiad.wmnet * 08:47 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 08:47 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1256: Migration of db1256.eqiad.wmnet completed * 08:44 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp601[1-2].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 08:42 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp600[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 08:42 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6004.drmrs.wmnet * 08:37 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es1036: Repooling * 08:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92733 and previous config saved to /var/cache/conftool/dbconfig/20260521-082951-fceratto.json * 08:29 hashar@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.3 refs [[phab:T423912|T423912]] * 08:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92731 and previous config saved to /var/cache/conftool/dbconfig/20260521-081642-fceratto.json * 08:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1036.eqiad.wmnet with reason: Maintenance * 08:02 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1256: Migration of db1256.eqiad.wmnet completed * 08:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6003.drmrs.wmnet * 08:00 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1256.eqiad.wmnet with OS trixie * 07:52 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp600[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:51 marostegui@dns1004: END - running authdns-update * 07:50 marostegui@dns1004: START - running authdns-update * 07:48 marostegui: Failover m3-master [[phab:T426633|T426633]] * 07:47 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1023.eqiad.wmnet * 07:46 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp6010.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:46 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6010.drmrs.wmnet * 07:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster1005.eqiad.wmnet to plain * 07:44 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster1005.eqiad.wmnet to plain * 07:43 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1256.eqiad.wmnet with reason: host reimage * 07:42 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster1005.eqiad.wmnet to drbd * 07:38 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1256.eqiad.wmnet with reason: host reimage * 07:35 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp6010.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:35 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp6002.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:35 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6002.drmrs.wmnet * 07:27 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster1005.eqiad.wmnet to drbd * 07:24 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp6002.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:24 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1256.eqiad.wmnet with OS trixie * 07:22 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1256: Upgrading db1256.eqiad.wmnet * 07:21 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1256: Upgrading db1256.eqiad.wmnet * 07:21 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to plain * 07:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to plain * 07:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbproxy1025.eqiad.wmnet with reason: Rebooting * 07:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to drbd * 06:54 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to drbd * 06:53 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to plain * 06:52 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to plain * 06:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to drbd * 06:42 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists1004.wikimedia.org * 06:40 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org * 06:39 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1003.eqiad.wmnet * 06:34 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org * 06:34 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host lists1004.wikimedia.org * 06:33 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host vrts1003.eqiad.wmnet * 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to drbd * 06:23 arnaudb@cumin1003: END (FAIL) - Cookbook sre.gerrit.reboot-gerrit (exit_code=99) Rebooting Gerrit on gerrit2003 * 06:22 arnaudb@cumin1003: START - Cookbook sre.gerrit.reboot-gerrit Rebooting Gerrit on gerrit2003 * 06:15 marostegui@dns1004: END - running authdns-update * 06:14 marostegui: Failover m2-master [[phab:T426633|T426633]] * 06:13 marostegui@dns1004: START - running authdns-update * 05:39 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1012 from dbctl [[phab:T426930|T426930]]', diff saved to https://phabricator.wikimedia.org/P92728 and previous config saved to /var/cache/conftool/dbconfig/20260521-053858-marostegui.json * 05:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc2 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92727 and previous config saved to /var/cache/conftool/dbconfig/20260521-053000-marostegui.json * 05:29 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1022 to pc2 master [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92726 and previous config saved to /var/cache/conftool/dbconfig/20260521-052905-marostegui.json * 05:21 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc1012.eqiad.wmnet with reason: Cloning * 02:41 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on planet1003.eqiad.wmnet with reason: debug wip * 02:11 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 29s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:29 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1027.eqiad.wmnet * 01:22 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1027.eqiad.wmnet * 00:55 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> scafeabh3a386lzmidfb70z3z39g9j5 2426653 2426652 2026-06-14T11:03:25Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2426653 wikitext text/x-wiki == 2026-06-14 == * 11:03 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 11:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 11:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 11:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:06 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-06-13 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 35s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-06-12 == * 19:54 dwisehaupt@dns1004: END - running authdns-update * 19:52 dwisehaupt@dns1004: START - running authdns-update * 18:33 dwisehaupt@dns1006: END - running authdns-update * 18:32 dwisehaupt@dns1006: START - running authdns-update * 16:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:10 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:10 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:59 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 15:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:43 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1301371{{!}}Hotfix for T428620 (T428620)]] (duration: 11m 17s) * 14:36 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 14:35 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1301371{{!}}Hotfix for T428620 (T428620)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:31 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1301371{{!}}Hotfix for T428620 (T428620)]] * 14:29 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:28 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 13:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 12:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 12:22 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:22 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:22 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:22 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 12:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 12:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 12:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 12:04 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:04 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:04 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 12:02 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.changedisk (exit_code=99) for changing disk type of prometheus5003.eqsin.wmnet to drbd * 12:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus5003.eqsin.wmnet to drbd * 11:40 moritzm: installing Linux 5.10.257 on Bullseye hosts * 11:36 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 11:35 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:24 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 11:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:56 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 10:56 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 10:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:49 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 10:49 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 10:40 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply * 10:37 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply * 10:36 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply * 10:35 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply * 10:35 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply * 10:35 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply * 10:12 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 10:12 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 10:08 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 09:59 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 09:58 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 09:57 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 06:13 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) * 06:11 jmm@cumin2002: START - Cookbook sre.puppet.disable-merges * 03:07 ryankemper: [[phab:T427951|T427951]] sorry, `[eqiad,codfw].mediawiki.page_html_content_change.rc0` (accidentally a word) * 03:06 ryankemper: [[phab:T427951|T427951]] Deleted all 20 unused dev/test topics on kafka-jumbo (verified empty first); 2 (`[eqiad,codfw]page_html_content_change.rc0`) were immediately auto-recreated empty by a still-running `dse-k8s` enrichment consumer; awaiting owner confirmation before final re-delete * 02:01 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 01m 13s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:00 bblack@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () == 2026-06-11 == * 22:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 22:26 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 22:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 22:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 22:05 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300906{{!}}Restore MediaViewer toggle in Special:Preferences (T428742)]] (duration: 30m 51s) * 21:58 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host releases2003.codfw.wmnet with OS trixie * 21:52 egardner@deploy1003: egardner: Continuing with deployment * 21:51 egardner@deploy1003: egardner: Backport for [[gerrit:1300906{{!}}Restore MediaViewer toggle in Special:Preferences (T428742)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:34 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1300906{{!}}Restore MediaViewer toggle in Special:Preferences (T428742)]] * 21:34 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet with reason: host reimage * 21:29 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300913{{!}}Avoid the escaping from nowiki processing (T398967)]] (duration: 09m 09s) * 21:28 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on releases2003.codfw.wmnet with reason: host reimage * 21:25 arlolra@deploy1003: arlolra: Continuing with deployment * 21:22 arlolra@deploy1003: arlolra: Backport for [[gerrit:1300913{{!}}Avoid the escaping from nowiki processing (T398967)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:20 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1300913{{!}}Avoid the escaping from nowiki processing (T398967)]] * 21:07 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300911{{!}}hCaptcha: Enable for badlogin for all small wikis (T426875)]], [[gerrit:1300905{{!}}RadioRangeBallot: Fix strict mode issue (T428947)]] (duration: 10m 43s) * 21:06 bblack@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text and not P<nowiki>{</nowiki>cp7008*<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 21:01 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 21:00 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300911{{!}}hCaptcha: Enable for badlogin for all small wikis (T426875)]], [[gerrit:1300905{{!}}RadioRangeBallot: Fix strict mode issue (T428947)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:56 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300911{{!}}hCaptcha: Enable for badlogin for all small wikis (T426875)]], [[gerrit:1300905{{!}}RadioRangeBallot: Fix strict mode issue (T428947)]] * 20:51 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300842{{!}}Donor Delight Badge: Unify on "Remove badge" language across treatments (T427313)]], [[gerrit:1300843{{!}}[A11y] Donor Badge: Remove Badge button disappears too quickly (T428646)]], [[gerrit:1300896{{!}}Donor Delight Badge, styles: Amending to final design review feedback (T427313)]] (duration: 34m 10s) * 20:39 jdrewniak@deploy1003: annet, jdrewniak: Continuing with deployment * 20:35 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host releases2003.codfw.wmnet with OS trixie * 20:34 jdrewniak@deploy1003: annet, jdrewniak: Backport for [[gerrit:1300842{{!}}Donor Delight Badge: Unify on "Remove badge" language across treatments (T427313)]], [[gerrit:1300843{{!}}[A11y] Donor Badge: Remove Badge button disappears too quickly (T428646)]], [[gerrit:1300896{{!}}Donor Delight Badge, styles: Amending to final design review feedback (T427313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug * 20:17 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1300842{{!}}Donor Delight Badge: Unify on "Remove badge" language across treatments (T427313)]], [[gerrit:1300843{{!}}[A11y] Donor Badge: Remove Badge button disappears too quickly (T428646)]], [[gerrit:1300896{{!}}Donor Delight Badge, styles: Amending to final design review feedback (T427313)]] * 19:12 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 18:12 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 18:12 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 17:52 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300865{{!}}UploadWizard.config.php: Fix cc-by-4.0-heirs msg issue (T428935 T405146)]] (duration: 08m 15s) * 17:48 reedy@deploy1003: reedy: Continuing with deployment * 17:46 reedy@deploy1003: reedy: Backport for [[gerrit:1300865{{!}}UploadWizard.config.php: Fix cc-by-4.0-heirs msg issue (T428935 T405146)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:44 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1300865{{!}}UploadWizard.config.php: Fix cc-by-4.0-heirs msg issue (T428935 T405146)]] * 17:26 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 17:25 blake@deploy1003: Scap cancelled without rolling back. * 17:25 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 17:24 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 17:24 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 17:24 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 17:24 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 17:23 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 17:23 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 17:23 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 17:23 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 17:23 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 17:23 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 17:20 blake@deploy1003: blake: apache config update ([[phab:T428772|T428772]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:20 blake@deploy1003: Started scap sync-world: apache config update ([[phab:T428772|T428772]]) * 17:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 17:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2212: Migration of db2212.codfw.wmnet completed * 17:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 17:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1235: Migration of db1235.eqiad.wmnet completed * 17:08 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 16:45 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:43 dzahn@dns1005: END - running authdns-update * 16:42 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:41 dzahn@dns1005: START - running authdns-update * 16:41 mutante: releases.wikimedia.org - switching backend from codfw to eqiad - releases1003 is now the source of rsync for uploaded releases files (use releases.discovery.wmnet to not have to think about it) - [[phab:T418299|T418299]] * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts rdb2007.codfw.wmnet * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts rdb1011.eqiad.wmnet * 16:35 jiji@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) * 16:34 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb2009.codfw.wmnet * 16:34 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:34 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2009.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:33 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Migration of db2212.codfw.wmnet completed * 16:27 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2009.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:27 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1235: Migration of db1235.eqiad.wmnet completed * 16:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2212.codfw.wmnet with OS trixie * 16:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1235.eqiad.wmnet with OS trixie * 16:13 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:07 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:06 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 16:05 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 16:05 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 16:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 16:04 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2212.codfw.wmnet with reason: host reimage * 16:01 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 16:01 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:01 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 16:01 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 16:00 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 16:00 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 16:00 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 16:00 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2212.codfw.wmnet with reason: host reimage * 15:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1235.eqiad.wmnet with reason: host reimage * 15:58 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 15:58 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 15:57 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 15:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 15:57 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 15:57 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 15:56 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2009.codfw.wmnet * 15:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:55 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb1011.eqiad.wmnet * 15:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 15:55 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2007.codfw.wmnet * 15:54 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 15:54 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1235.eqiad.wmnet with reason: host reimage * 15:54 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 15:53 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 15:53 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 15:40 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 15:40 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2212.codfw.wmnet with OS trixie * 15:39 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 15:39 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1235.eqiad.wmnet with OS trixie * 15:36 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 15:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1235: Upgrading db1235.eqiad.wmnet * 15:35 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 15:35 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1235: Upgrading db1235.eqiad.wmnet * 15:35 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:32 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 15:32 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:31 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 15:30 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300822{{!}}T428849: temporarily disable noisy warnings in HandleParsoidSectionLinks (T428849 T417530)]] (duration: 11m 29s) * 15:27 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2212: Upgrading db2212.codfw.wmnet * 15:26 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2212: Upgrading db2212.codfw.wmnet * 15:26 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:26 cscott@deploy1003: cscott: Continuing with deployment * 15:26 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1235: Upgrading db1235.eqiad.wmnet * 15:25 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1235: Upgrading db1235.eqiad.wmnet * 15:25 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:21 cscott@deploy1003: cscott: Backport for [[gerrit:1300822{{!}}T428849: temporarily disable noisy warnings in HandleParsoidSectionLinks (T428849 T417530)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:19 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1300822{{!}}T428849: temporarily disable noisy warnings in HandleParsoidSectionLinks (T428849 T417530)]] * 15:18 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 15:17 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 15:13 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 15:13 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 15:13 moritzm: installing libdbi-perl security updates * 14:53 moritzm: installing Bind security updates (just client-side tools/libraries) * 14:51 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry * 14:48 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry * 14:43 moritzm: installing Poppler security updates * 14:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:33 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 14:32 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 14:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1234: Migration of db1234.eqiad.wmnet completed * 14:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin02 and group 01 * 14:24 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin02 and group 01 * 14:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 14:18 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet * 14:08 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet * 14:00 Lucas_WMDE: UTC afternoon backport+config window done * 13:58 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300733{{!}}stream: webrequest.page_view_stats.dev0 (T428725)]] (duration: 08m 12s) * 13:57 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp5024.* * 13:55 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: name=cp5024.* * 13:55 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp5020.* * 13:54 javiermonton@deploy1003: javiermonton: Continuing with deployment * 13:52 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1300733{{!}}stream: webrequest.page_view_stats.dev0 (T428725)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:51 slyngshede@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P<nowiki>{</nowiki>lvs5004*<nowiki>}</nowiki> and A:liberica * 13:50 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1300733{{!}}stream: webrequest.page_view_stats.dev0 (T428725)]] * 13:50 slyngshede@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P<nowiki>{</nowiki>lvs5004*<nowiki>}</nowiki> and A:liberica * 13:50 slyngs: reloading liberica config on lvs5004 * 13:50 moritzm: installing openssl security updates * 13:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:46 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 13:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm * 13:46 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1234: Migration of db1234.eqiad.wmnet completed * 13:46 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 13:45 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 13:45 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 13:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2202.codfw.wmnet with OS trixie * 13:43 alexsanford@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298890{{!}}Add 2FA enforcement demotion config for phase 3 groups (T423120)]] (duration: 07m 19s) * 13:39 alexsanford@deploy1003: alexsanford: Continuing with deployment * 13:38 alexsanford@deploy1003: alexsanford: Backport for [[gerrit:1298890{{!}}Add 2FA enforcement demotion config for phase 3 groups (T423120)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:36 alexsanford@deploy1003: Started scap sync-world: Backport for [[gerrit:1298890{{!}}Add 2FA enforcement demotion config for phase 3 groups (T423120)]] * 13:36 slyngshede@dns1004: END - running authdns-update * 13:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1234.eqiad.wmnet with OS trixie * 13:34 moritzm: installing dovecot security updates * 13:34 slyngshede@dns1004: START - running authdns-update * 13:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300787{{!}}hCaptcha: Enable for MobileFrontend on all group1 wikis (T425940)]] (duration: 06m 59s) * 13:29 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 13:29 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 13:29 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 13:29 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 13:28 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 13:28 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 13:28 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 13:27 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300787{{!}}hCaptcha: Enable for MobileFrontend on all group1 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:26 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2202.codfw.wmnet with reason: host reimage * 13:25 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300787{{!}}hCaptcha: Enable for MobileFrontend on all group1 wikis (T425940)]] * 13:25 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/Android_FAQ 'Wikimedia Apps/FAQ/Android' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:24 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/Android_FAQ 'Wikimedia Apps/FAQ/Android' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:22 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300736{{!}}fix: correct intake-url and payload type for NCS experiment events (T422295)]] (duration: 06m 51s) * 13:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage * 13:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1234.eqiad.wmnet with reason: host reimage * 13:18 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Continuing with deployment * 13:18 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2202.codfw.wmnet with reason: host reimage * 13:18 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Backport for [[gerrit:1300736{{!}}fix: correct intake-url and payload type for NCS experiment events (T422295)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:18 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 13:17 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 13:16 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1300736{{!}}fix: correct intake-url and payload type for NCS experiment events (T422295)]] * 13:15 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage * 13:14 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/Android_FAQ 'Wikimedia Apps/FAQ/Android' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:13 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:13 gkyziridis@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300731{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] (duration: 08m 47s) * 13:13 andrewbogott: sudo -i reprepro --noskipold --component thirdparty/openstack-trixie-flamingo-backports update trixie-wikimedia * 13:12 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1234.eqiad.wmnet with reason: host reimage * 13:12 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 13:12 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki '--reason=per [[:phab:T428900]]' Wikimedia_Apps/iOS_FAQ 'Wikimedia Apps/FAQ/iOS' 'Martin Urbanec (WMF)' # [[phab:T428900|T428900]] * 13:12 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 13:12 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 13:11 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 13:11 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 13:11 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 13:11 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 13:11 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 13:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 13:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 13:09 gkyziridis@deploy1003: gkyziridis: Continuing with deployment * 13:06 gkyziridis@deploy1003: gkyziridis: Backport for [[gerrit:1300731{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:06 claime: echo 'https://api.wikimedia.org/service/lw/specs/openapi.yaml' {{!}} mwscript-k8s --attach -- purgeList.php * 13:04 gkyziridis@deploy1003: Started scap sync-world: Backport for [[gerrit:1300731{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] * 13:02 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2202.codfw.wmnet with OS trixie * 13:00 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:57 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1234.eqiad.wmnet with OS trixie * 12:55 moritzm: installing Exim security updates on Bullseye * 12:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ganeti5006 * 12:47 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti5006 * 12:46 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti5006 * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti5006.eqsin.wmnet 9.0.132.10.in-addr.arpa 9.0.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 12:46 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache ganeti5006.eqsin.wmnet 9.0.132.10.in-addr.arpa 9.0.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5006 - jmm@cumin2002" * 12:46 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5006 - jmm@cumin2002" * 12:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1234: Upgrading db1234.eqiad.wmnet * 12:44 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1234: Upgrading db1234.eqiad.wmnet * 12:44 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2188: Migration of db2188.codfw.wmnet completed * 12:29 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "UX improvements - oblivian@cumin1003" * 12:29 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: UX improvements - oblivian@cumin1003 * 12:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1232: Migration of db1232.eqiad.wmnet completed * 12:28 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: UX improvements - oblivian@cumin1003 * 12:28 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "UX improvements - oblivian@cumin1003" * 12:27 jmm@cumin2002: START - Cookbook sre.dns.netbox * 12:26 jmm@cumin2002: START - Cookbook sre.hosts.move-vlan for host ganeti5006 * 12:26 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm * 12:21 moritzm: remove ganeti5006 from eqsin cluster for reimage [[phab:T428229|T428229]] * 12:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet * 12:10 moritzm: installing openjdk-21 security updates on Bookworm * 12:03 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300764{{!}}Remove GrowthExperiments extension from closed wikis (T428884)]] (duration: 06m 53s) * 11:59 urbanecm@deploy1003: urbanecm: Continuing with deployment * 11:58 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1300764{{!}}Remove GrowthExperiments extension from closed wikis (T428884)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:56 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1300764{{!}}Remove GrowthExperiments extension from closed wikis (T428884)]] * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb1012.eqiad.wmnet * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb2010.codfw.wmnet * 11:49 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:48 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2010.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 11:46 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:46 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rdb2008.codfw.wmnet * 11:46 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:46 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2188: Migration of db2188.codfw.wmnet completed * 11:44 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 11:43 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:43 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rdb2010.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 11:43 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1232: Migration of db1232.eqiad.wmnet completed * 11:38 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:37 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 11:37 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 11:36 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 11:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2188.codfw.wmnet with OS trixie * 11:35 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb1012.eqiad.wmnet * 11:34 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2008.codfw.wmnet * 11:34 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts rdb2010.codfw.wmnet * 11:33 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 11:32 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 11:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1232.eqiad.wmnet with OS trixie * 11:27 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2002.codfw.wmnet * 11:25 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300749{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300751{{!}}hCaptcha: Enable for DiscussionTools on all wikis (T426039)]] (duration: 08m 38s) * 11:21 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 11:19 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300749{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300751{{!}}hCaptcha: Enable for DiscussionTools on all wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2188.codfw.wmnet with reason: host reimage * 11:17 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300749{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300751{{!}}hCaptcha: Enable for DiscussionTools on all wikis (T426039)]] * 11:15 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2188.codfw.wmnet with reason: host reimage * 11:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1232.eqiad.wmnet with reason: host reimage * 11:13 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc2002.codfw.wmnet * 11:13 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 11:12 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet * 11:11 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 11:09 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2001.codfw.wmnet * 11:09 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1232.eqiad.wmnet with reason: host reimage * 11:08 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet * 11:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:04 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc2001.codfw.wmnet * 11:04 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host testreduce1002.eqiad.wmnet * 11:04 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:02 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1262.eqiad.wmnet with reason: crash * 11:00 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 11:00 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host testreduce1002.eqiad.wmnet * 10:59 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 10:59 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 10:58 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 10:55 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2188.codfw.wmnet with OS trixie * 10:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2188: Upgrading db2188.codfw.wmnet * 10:52 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2188: Upgrading db2188.codfw.wmnet * 10:52 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:52 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1232.eqiad.wmnet with OS trixie * 10:48 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1232: Upgrading db1232.eqiad.wmnet * 10:48 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1232: Upgrading db1232.eqiad.wmnet * 10:48 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:33 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:32 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:31 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300734{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300727{{!}}hCaptcha: Enable for DiscussionTools on group 1 wikis (T426039)]] (duration: 11m 01s) * 10:26 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:23 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:23 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:22 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1300734{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300727{{!}}hCaptcha: Enable for DiscussionTools on group 1 wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:20 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1300734{{!}}HCaptcha: Return 'forceshowcaptcha' error when CAPTCHA forced (T426476)]], [[gerrit:1300727{{!}}hCaptcha: Enable for DiscussionTools on group 1 wikis (T426039)]] * 10:18 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:18 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:10 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 10:10 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 10:09 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2045.codfw.wmnet with OS trixie * 10:09 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repool es2046', diff saved to https://phabricator.wikimedia.org/P94069 and previous config saved to /var/cache/conftool/dbconfig/20260611-100221-marostegui.json * 10:01 marostegui@cumin1003: dbctl commit (dc=all): 'Depool es2046', diff saved to https://phabricator.wikimedia.org/P94068 and previous config saved to /var/cache/conftool/dbconfig/20260611-100145-marostegui.json * 10:01 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:59 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300580{{!}}ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] (duration: 15m 41s) * 09:54 jiji@deploy1003: jiji: Continuing with deployment * 09:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2045.codfw.wmnet with reason: host reimage * 09:45 jiji@deploy1003: jiji: Backport for [[gerrit:1300580{{!}}ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:43 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1300580{{!}}ProductionServices.php: switch filebackend.php back to rdb1013 (T291916 T419976)]] * 09:42 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2045.codfw.wmnet with reason: host reimage * 09:37 elukey: uploaded spicerack_12.8.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia * 09:26 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS trixie * 09:26 marostegui@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host es2045.codfw.wmnet with OS bookworm * 09:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2176: Migration of db2176.codfw.wmnet completed * 09:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1219: Migration of db1219.eqiad.wmnet completed * 09:11 claime: cumin -x 'A:swift-fe' "disable-puppet 'Disabling puppet for ratelimit deploy - cgoubert'" * 08:57 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS bookworm * 08:39 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2176: Migration of db2176.codfw.wmnet completed * 08:34 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94055) * 08:34 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1219: Migration of db1219.eqiad.wmnet completed * 08:33 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94053) * 08:30 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T428823|T428823]] (duration: 01m 18s) * 08:29 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T428823|T428823]] * 08:27 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2176.codfw.wmnet with OS trixie * 08:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Migration to 10.11.17 * 08:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 08:25 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 08:25 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool pc1021: Migration to 10.11.17 * 08:25 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94052) * 08:24 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): Testing upgrade for [[phab:T428823|T428823]] (duration: 01m 17s) * 08:23 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): Testing upgrade for [[phab:T428823|T428823]] * 08:22 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94051) * 08:22 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1219.eqiad.wmnet with OS trixie * 08:17 moritzm: installing PHP 8.2 security updates * 08:15 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 08:14 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 08:11 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 08:11 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 08:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2176.codfw.wmnet with reason: host reimage * 08:08 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1013.eqiad.wmnet with OS trixie * 08:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5004.eqsin.wmnet to cluster eqsin02 and group 01 * 08:06 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 08:06 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 08:05 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2021.codfw.wmnet,pc1021.eqiad.wmnet with reason: upgrade * 08:05 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1219.eqiad.wmnet with reason: host reimage * 08:05 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5004.eqsin.wmnet to cluster eqsin02 and group 01 * 08:05 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 08:05 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 08:04 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2176.codfw.wmnet with reason: host reimage * 08:04 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 08:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 08:03 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 08:03 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1021: Migration to 10.11.17 [[phab:T427345|T427345]] * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5004.eqsin.wmnet * 07:58 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1219.eqiad.wmnet with reason: host reimage * 07:56 marostegui: install mariadb 10.11.17 on pc1 [[phab:T427345|T427345]] * 07:54 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1013.eqiad.wmnet with reason: host reimage * 07:50 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1013.eqiad.wmnet with reason: host reimage * 07:49 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 07:49 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 07:49 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5004.eqsin.wmnet * 07:47 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply * 07:47 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply * 07:46 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2176.codfw.wmnet with OS trixie * 07:43 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1219.eqiad.wmnet with OS trixie * 07:43 moritzm: imported Jenkins 2.541.3 for thirdparty/ci (Bullseye) and thirdparty/jenkins (Bookworm, Trixie) * 07:42 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 07:35 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1013.eqiad.wmnet with OS trixie * 07:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2176: Upgrading db2176.codfw.wmnet * 07:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1219: Upgrading db1219.eqiad.wmnet * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2176: Upgrading db2176.codfw.wmnet * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1219: Upgrading db1219.eqiad.wmnet * 07:31 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 07:31 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:30 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 07:29 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1163: Repooling * 07:19 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 06:51 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS trixie * 06:50 marostegui@cumin1003: dbctl commit (dc=all): 'Repool es2042', diff saved to https://phabricator.wikimedia.org/P94044 and previous config saved to /var/cache/conftool/dbconfig/20260611-065049-marostegui.json * 06:50 marostegui@cumin1003: dbctl commit (dc=all): 'Depool es2042', diff saved to https://phabricator.wikimedia.org/P94043 and previous config saved to /var/cache/conftool/dbconfig/20260611-065027-marostegui.json * 06:44 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1163: Repooling * 06:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1163 [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94041 and previous config saved to /var/cache/conftool/dbconfig/20260611-064319-fceratto.json * 06:42 fceratto@dns1005: END - running authdns-update * 06:40 fceratto@dns1005: START - running authdns-update * 06:33 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:33 fceratto@cumin1003: MariaDB change: Setting sections s1 as read-write for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:33 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:33 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1184 to s1 primary and set section read-write [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94040 and previous config saved to /var/cache/conftool/dbconfig/20260611-063323-fceratto.json * 06:32 fceratto@cumin1003: dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94039 and previous config saved to /var/cache/conftool/dbconfig/20260611-063251-fceratto.json * 06:32 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:32 fceratto@cumin1003: Dbctl change: Setting sections s1 as read-write for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:32 fceratto@cumin1003: MariaDB change: Setting sections s1 as read-write for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:31 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:31 fceratto@cumin1003: dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94037 and previous config saved to /var/cache/conftool/dbconfig/20260611-063100-fceratto.json * 06:30 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:30 fceratto@cumin1003: MariaDB change: Setting sections s1 as read-only for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:30 fceratto@cumin1003: Dbctl change: Setting sections s1 as read-only for [[phab:T426083|T426083]]: 'Maintenance until 06:15 UTC' * 06:29 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:29 federico3: Starting s1 eqiad failover from db1163 to db1184 - [[phab:T426083|T426083]] * 06:22 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1184 with weight 0 [[phab:T426083|T426083]]', diff saved to https://phabricator.wikimedia.org/P94035 and previous config saved to /var/cache/conftool/dbconfig/20260611-062224-fceratto.json * 06:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 30 hosts with reason: Primary switchover s1 [[phab:T426083|T426083]] * 05:37 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 05:28 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab * 05:27 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 05:18 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab * 05:17 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2045.codfw.wmnet with OS trixie * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2045: Upgrading es2045.codfw.wmnet * 05:16 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2045: Upgrading es2045.codfw.wmnet * 05:16 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 44s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:23 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2046.* * 01:19 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync * 01:18 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync * 01:18 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1009.eqiad.wmnet with OS trixie * 01:12 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:12 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 01:12 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:12 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:11 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:11 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 01:11 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:10 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:10 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 01:09 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 01:09 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 01:08 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 01:08 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 01:08 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 01:07 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 01:07 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 01:06 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 01:06 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 01:06 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 01:05 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 01:05 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 01:05 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 01:02 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1009.eqiad.wmnet with reason: host reimage * 00:58 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1009.eqiad.wmnet with reason: host reimage * 00:54 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:53 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 00:52 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 00:51 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main1009 * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main1009 * 00:41 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main1009 * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main1009.eqiad.wmnet 37.48.64.10.in-addr.arpa 7.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:41 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main1009.eqiad.wmnet 37.48.64.10.in-addr.arpa 7.3.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 00:41 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1009 - jasmine@cumin2002" * 00:40 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1009 - jasmine@cumin2002" * 00:39 cdanis@cumin1003: dbctl commit (dc=all): 'depool db1262', diff saved to https://phabricator.wikimedia.org/P94032 and previous config saved to /var/cache/conftool/dbconfig/20260611-003950-cdanis.json * 00:36 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 00:34 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5020.* * 00:30 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main1009 * 00:30 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1009.eqiad.wmnet with OS trixie * 00:03 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5024.* == 2026-06-10 == * 23:53 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5024.* * 23:15 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300154{{!}}Disable ShortUrl on bdwikimedia, bhwiki, bnwiki, bnwikisource, eswikibooks, gomwiki (T107188)]] (duration: 11m 37s) * 23:11 krinkle@deploy1003: krinkle: Continuing with deployment * 23:06 krinkle@deploy1003: krinkle: Backport for [[gerrit:1300154{{!}}Disable ShortUrl on bdwikimedia, bhwiki, bnwiki, bnwikisource, eswikibooks, gomwiki (T107188)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:04 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1300154{{!}}Disable ShortUrl on bdwikimedia, bhwiki, bnwiki, bnwikisource, eswikibooks, gomwiki (T107188)]] * 22:57 ladsgroup@dns1004: END - running authdns-update * 22:55 ladsgroup@dns1004: START - running authdns-update * 22:13 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5024.eqsin.wmnet with OS trixie * 22:13 mutante: gerrit - restarting service for logging change * 22:11 dzahn@cumin2002: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 0:10:00 on gerrit.wikimedia.org with reason: service restart * 22:09 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on gerrit2003.wikimedia.org with reason: service restart * 22:06 mutante: gerrit-spare: restarting gerrit * 22:06 mutante: gerrit-replica: restarting gerrit * 21:44 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage * 21:37 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage * 21:22 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300250{{!}}ExecuteTestAndCacheJob: Fix stdClasses serialised wrongly by JobQueue (T428801)]], [[gerrit:1300248{{!}}tests: Fix StandaloneHooksTest ordering, now broken by DB upgrade]] (duration: 08m 23s) * 21:17 jforrester@deploy1003: jforrester: Continuing with deployment * 21:15 jforrester@deploy1003: jforrester: Backport for [[gerrit:1300250{{!}}ExecuteTestAndCacheJob: Fix stdClasses serialised wrongly by JobQueue (T428801)]], [[gerrit:1300248{{!}}tests: Fix StandaloneHooksTest ordering, now broken by DB upgrade]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:13 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1300250{{!}}ExecuteTestAndCacheJob: Fix stdClasses serialised wrongly by JobQueue (T428801)]], [[gerrit:1300248{{!}}tests: Fix StandaloneHooksTest ordering, now broken by DB upgrade]] * 21:03 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5024 * 21:02 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5024 * 21:02 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300247{{!}}Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] (duration: 06m 51s) * 21:00 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp5024 * 21:00 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5024.eqsin.wmnet 35.0.132.10.in-addr.arpa 5.3.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 21:00 brett@cumin2002: START - Cookbook sre.dns.wipe-cache cp5024.eqsin.wmnet 35.0.132.10.in-addr.arpa 5.3.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 21:00 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:00 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5024 - brett@cumin2002" * 20:59 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5024 - brett@cumin2002" * 20:57 catrope@deploy1003: catrope: Continuing with deployment * 20:57 catrope@deploy1003: catrope: Backport for [[gerrit:1300247{{!}}Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:55 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1300247{{!}}Revert "wgRestSandboxSpecs: Add Lift Wing API to documentation wikis" (T427902)]] * 20:54 brett@cumin2002: START - Cookbook sre.dns.netbox * 20:50 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5024 * 20:49 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5024.eqsin.wmnet with OS trixie * 20:48 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5020.* * 20:44 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300073{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] (duration: 11m 55s) * 20:40 catrope@deploy1003: catrope, gkyziridis: Continuing with deployment * 20:34 catrope@deploy1003: catrope, gkyziridis: Backport for [[gerrit:1300073{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:32 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1300073{{!}}wgRestSandboxSpecs: Add Lift Wing API to documentation wikis (T427902)]] * 20:30 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5020.eqsin.wmnet with OS trixie * 20:30 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300226{{!}}[arzwiki] Change the wordmark (T427720)]] (duration: 09m 49s) * 20:25 catrope@deploy1003: gergesshamon, catrope: Continuing with deployment * 20:22 catrope@deploy1003: gergesshamon, catrope: Backport for [[gerrit:1300226{{!}}[arzwiki] Change the wordmark (T427720)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1300226{{!}}[arzwiki] Change the wordmark (T427720)]] * 19:59 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5020.eqsin.wmnet with reason: host reimage * 19:53 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5020.eqsin.wmnet with reason: host reimage * 19:30 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 19:27 bblack@cumin1003: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 19:23 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2046.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:19 brett@cumin2002: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2046.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5020 * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5020 * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2044.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:18 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp5020 * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5020.eqsin.wmnet 24.0.132.10.in-addr.arpa 4.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:18 brett@cumin2002: START - Cookbook sre.dns.wipe-cache cp5020.eqsin.wmnet 24.0.132.10.in-addr.arpa 4.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:17 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5020 - brett@cumin2002" * 19:17 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5020 - brett@cumin2002" * 19:14 brett@cumin2002: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp2044.codfw.wmnet<nowiki>}</nowiki> and A:cp - testing {{Gerrit|1300236}} () * 19:11 brett@cumin2002: START - Cookbook sre.dns.netbox * 19:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 19:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2174: Migration of db2174.codfw.wmnet completed * 19:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 19:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1218: Migration of db1218.eqiad.wmnet completed * 18:24 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5020 * 18:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5020.eqsin.wmnet with OS trixie * 18:22 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2174: Migration of db2174.codfw.wmnet completed * 18:20 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 18:17 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1218: Migration of db1218.eqiad.wmnet completed * 18:16 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5018.* * 18:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2174.codfw.wmnet with OS trixie * 18:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1218.eqiad.wmnet with OS trixie * 17:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2174.codfw.wmnet with reason: host reimage * 17:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1218.eqiad.wmnet with reason: host reimage * 17:46 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2010.codfw.wmnet with OS trixie * 17:45 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync * 17:44 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync * 17:44 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2174.codfw.wmnet with reason: host reimage * 17:42 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1218.eqiad.wmnet with reason: host reimage * 17:33 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94021) * 17:29 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2010.codfw.wmnet with reason: host reimage * 17:26 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1218.eqiad.wmnet with OS trixie * 17:26 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2174.codfw.wmnet with OS trixie * 17:25 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:24 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 17:24 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1218: Upgrading db1218.eqiad.wmnet * 17:24 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:24 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:24 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1218: Upgrading db1218.eqiad.wmnet * 17:23 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 17:23 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2174: Upgrading db2174.codfw.wmnet * 17:23 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 17:23 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2010.codfw.wmnet with reason: host reimage * 17:23 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:22 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2174: Upgrading db2174.codfw.wmnet * 17:22 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:22 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload and not P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 17:22 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 17:22 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 17:22 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text and not P<nowiki>{</nowiki>cp7008*<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 17:21 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 17:21 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 17:20 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 17:19 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 17:19 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 17:18 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 17:18 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 17:17 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 17:17 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 17:17 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 17:15 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 17:14 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 17:13 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 17:12 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox) * 17:03 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 17:03 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1206: Migration of db1206.eqiad.wmnet completed * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2010 * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2010 * 17:02 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2010 * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2010.codfw.wmnet 35.48.192.10.in-addr.arpa 5.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:02 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2010.codfw.wmnet 35.48.192.10.in-addr.arpa 5.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:02 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2010 - jasmine@cumin2002" * 17:01 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2010 - jasmine@cumin2002" * 16:57 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 16:50 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2010 * 16:50 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2010.codfw.wmnet with OS trixie * 16:41 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 16:39 bblack@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 16:39 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 16:34 bblack@cumin1003: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7008.magru.wmnet<nowiki>}</nowiki> and A:cp - Upgrade wmfuniq to 0.3.0 () * 16:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5018.eqsin.wmnet with OS trixie * 16:22 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 16:20 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 16:17 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1206: Migration of db1206.eqiad.wmnet completed * 16:15 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 16:15 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 16:14 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 16:12 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 16:12 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 16:11 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 16:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1206.eqiad.wmnet with OS trixie * 16:01 blblack: apt: uploaded libvmod-wmfuniq 0.3.0 for trixie * 15:59 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5018.eqsin.wmnet with reason: host reimage * 15:53 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:52 vriley@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:51 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5018.eqsin.wmnet with reason: host reimage * 15:50 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1206.eqiad.wmnet with reason: host reimage * 15:45 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1206.eqiad.wmnet with reason: host reimage * 15:43 sukhe@cumin1003: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool drmrs [reason: no reason specified, no task ID specified] * 15:42 sukhe@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: no reason specified, no task ID specified] * 15:38 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 15:38 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2173: Migration of db2173.codfw.wmnet completed * 15:34 topranks: drain traffic through cr2-drmrs to reset pic0 * 15:33 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release (dblist: https://phabricator.wikimedia.org/P94013) * 15:30 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1206.eqiad.wmnet with OS trixie * 15:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1206: Upgrading db1206.eqiad.wmnet * 15:28 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1206: Upgrading db1206.eqiad.wmnet * 15:27 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 15:25 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:24 vriley@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 15:24 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1009 * 15:24 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Harroyo-wmf out of all services on: 2436 hosts * 15:23 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1009 * 15:21 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:20 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist translate extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release * 15:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5018 * 15:19 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5018 * 15:18 vriley@cumin1003: START - Cookbook sre.dns.netbox * 15:18 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cp5018 * 15:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5018.eqsin.wmnet 18.0.132.10.in-addr.arpa 8.1.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 15:18 brett@cumin2002: START - Cookbook sre.dns.wipe-cache cp5018.eqsin.wmnet 18.0.132.10.in-addr.arpa 8.1.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 15:18 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:15 brett@cumin2002: START - Cookbook sre.dns.netbox * 15:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 15:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1195: Migration of db1195.eqiad.wmnet completed * 15:12 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:11 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:11 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin1003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:11 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin1003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 15:08 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300169{{!}}Fix snak value display for rtl languages (T360854)]], [[gerrit:1300168{{!}}Fix snak value display for rtl languages (T360854)]] (duration: 08m 39s) * 15:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 15:01 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1300169{{!}}Fix snak value display for rtl languages (T360854)]], [[gerrit:1300168{{!}}Fix snak value display for rtl languages (T360854)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:59 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:59 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1300169{{!}}Fix snak value display for rtl languages (T360854)]], [[gerrit:1300168{{!}}Fix snak value display for rtl languages (T360854)]] * 14:58 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:55 Lucas_WMDE: lucaswerkmeister-wmde@deploy1003 $ printf 'https://www.mediawiki.org/keys/%s\n' '' 'keys.txt' 'keys.html' {{!}} mwscript-k8s --attach --comment=[[phab:T423267|T423267]] purgeList mediawikiwiki * 14:54 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist translate extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release, now with correct schema * 14:53 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2173: Migration of db2173.codfw.wmnet completed * 14:50 ayounsi@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin2003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:50 ayounsi@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2003.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:49 ayounsi@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:48 ayounsi@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - ayounsi@cumin1003 * 14:47 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299614{{!}}Add my public key to mediawiki.org/keys (T423267)]] (duration: 08m 33s) * 14:46 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:42 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, matmarex: Continuing with deployment * 14:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2173.codfw.wmnet with OS trixie * 14:40 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, matmarex: Backport for [[gerrit:1299614{{!}}Add my public key to mediawiki.org/keys (T423267)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:40 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:40 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:38 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1299614{{!}}Add my public key to mediawiki.org/keys (T423267)]] * 14:38 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox and (A:dnsbox) * 14:34 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:34 cmooney@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:33 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:29 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1195: Migration of db1195.eqiad.wmnet completed * 14:28 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: add new eqsin vlans as legacy temp workaround in wmf-plugin.py - cmooney@cumin1003 * 14:27 vriley@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:26 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 14:26 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 14:24 atsuko@deploy1003: mwscript-k8s job started: foreachwikiindblist translate extensions/Translate/scripts/ttmserver-export.php --ttmserver eqiad-test # [[phab:T425377|T425377]] populating ttmserver index on test cluster to estimate time required for the release, now with dblist translate * 14:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2173.codfw.wmnet with reason: host reimage * 14:23 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 14:22 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 14:22 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply * 14:21 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply * 14:20 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox) * 14:20 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2173.codfw.wmnet with reason: host reimage * 14:20 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:19 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:19 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:18 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:18 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:18 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:18 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1195.eqiad.wmnet with OS trixie * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 14:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 14:15 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 14:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 14:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 14:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 14:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:13 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:13 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 14:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 14:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 14:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 14:09 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 14:08 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:08 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 14:07 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 14:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 14:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 14:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 14:02 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2173.codfw.wmnet with OS trixie * 14:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 14:00 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1195.eqiad.wmnet with reason: host reimage * 14:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 13:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2173: Upgrading db2173.codfw.wmnet * 13:59 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2173: Upgrading db2173.codfw.wmnet * 13:58 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:58 atsuko@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/ttmserver-export.php --wiki=default --ttmserver eqiad-test # [[phab:T425377|T425377]] populating production index on test cluster to estimate time required for the release * 13:56 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1195.eqiad.wmnet with reason: host reimage * 13:54 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Atieno out of all services on: 2436 hosts * 13:42 Lucas_WMDE: UTC afternoon backport+config window done * 13:42 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1195.eqiad.wmnet with OS trixie * 13:36 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297237{{!}}wmf-config: Update private subnets to include additions (T427393)]] (duration: 07m 20s) * 13:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1195: Upgrading db1195.eqiad.wmnet * 13:33 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling restart_daemons on A:hcaptcha-proxy and A:hcaptcha-proxy * 13:33 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum and A:durum * 13:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2170: Migration of db2170.codfw.wmnet completed * 13:33 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1195: Upgrading db1195.eqiad.wmnet * 13:32 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:32 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, brett: Continuing with deployment * 13:32 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough * 13:31 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply * 13:31 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, brett: Backport for [[gerrit:1297237{{!}}wmf-config: Update private subnets to include additions (T427393)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:31 eevans@deploy1003: helmfile [staging] START helmfile.d/services/data-gateway: apply * 13:29 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1297237{{!}}wmf-config: Update private subnets to include additions (T427393)]] * 13:28 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp5018.eqsin.wmnet with reason: host down * 13:28 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy (exit_code=0) rolling restart_daemons on A:tcpproxy and A:tcpproxy * 13:25 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5018.eqsin.wmnet,service=(cdn{{!}}ats-be) * 13:22 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox) * 13:20 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum and A:durum * 13:20 sukhe@cumin1003: START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling restart_daemons on A:hcaptcha-proxy and A:hcaptcha-proxy * 13:19 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299676{{!}}Enable ULS v2 on group0 wikis]] (duration: 17m 00s) * 13:19 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough * 13:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1186: Migration of db1186.eqiad.wmnet completed * 13:18 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-test: apply * 13:18 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-test: apply * 13:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply * 13:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply * 13:15 sbisson@deploy1003: sbisson, abi: Continuing with deployment * 13:10 sukhe@cumin1003: START - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy rolling restart_daemons on A:tcpproxy and A:tcpproxy * 13:05 sbisson@deploy1003: sbisson, abi: Backport for [[gerrit:1299676{{!}}Enable ULS v2 on group0 wikis]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:03 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1014.eqiad.wmnet with OS trixie * 13:02 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299676{{!}}Enable ULS v2 on group0 wikis]] * 12:47 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2170: Migration of db2170.codfw.wmnet completed * 12:46 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5004.eqsin.wmnet with OS bookworm * 12:46 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:46 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:46 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply * 12:45 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1014.eqiad.wmnet with reason: host reimage * 12:42 topranks: re-map DSCP AF41 from 'low' to 'normal' priority qos class on network [[phab:T424640|T424640]] * 12:41 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1014.eqiad.wmnet with reason: host reimage * 12:36 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2170.codfw.wmnet with OS trixie * 12:33 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1186: Migration of db1186.eqiad.wmnet completed * 12:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5004.eqsin.wmnet with reason: host reimage * 12:24 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1014 * 12:24 jiji@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host rdb1014 * 12:23 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1186.eqiad.wmnet with OS trixie * 12:21 jiji@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host rdb1014 * 12:21 jiji@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb1014.eqiad.wmnet 42.48.64.10.in-addr.arpa 2.4.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 12:21 jiji@cumin1003: START - Cookbook sre.dns.wipe-cache rdb1014.eqiad.wmnet 42.48.64.10.in-addr.arpa 2.4.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 12:21 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:21 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host rdb1014 - jiji@cumin1003" * 12:21 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host rdb1014 - jiji@cumin1003" * 12:20 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5004.eqsin.wmnet with reason: host reimage * 12:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2170.codfw.wmnet with reason: host reimage * 12:16 jiji@cumin1003: START - Cookbook sre.dns.netbox * 12:13 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1014 * 12:12 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1014.eqiad.wmnet with OS trixie * 12:12 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2170.codfw.wmnet with reason: host reimage * 12:08 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300104{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643{{!}}wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] (duration: 11m 06s) * 12:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1186.eqiad.wmnet with reason: host reimage * 12:03 reedy@deploy1003: reedy: Continuing with deployment * 12:02 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1186.eqiad.wmnet with reason: host reimage * 11:59 reedy@deploy1003: reedy: Backport for [[gerrit:1300104{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643{{!}}wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes c * 11:57 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1300104{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1300102{{!}}Mandatory2FAChecker: Allow getGroupsRequiring2FA() to work on implicit groups (T420792)]], [[gerrit:1299643{{!}}wmf-config: Add $wmgOATHAuthRequire2FAForAll config (T420792)]] * 11:53 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2170.codfw.wmnet with OS trixie * 11:51 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ganeti5004 * 11:51 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti5004 * 11:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2170: Upgrading db2170.codfw.wmnet * 11:49 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2170: Upgrading db2170.codfw.wmnet * 11:49 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti5004 * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti5004.eqsin.wmnet 40.0.132.10.in-addr.arpa 0.4.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 11:49 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache ganeti5004.eqsin.wmnet 40.0.132.10.in-addr.arpa 0.4.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5004 - jmm@cumin2002" * 11:49 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti5004 - jmm@cumin2002" * 11:49 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:48 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1186.eqiad.wmnet with OS trixie * 11:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1186: Upgrading db1186.eqiad.wmnet * 11:45 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1186: Upgrading db1186.eqiad.wmnet * 11:45 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:38 jmm@cumin2002: START - Cookbook sre.dns.netbox * 11:35 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 11:34 jmm@cumin2002: START - Cookbook sre.hosts.move-vlan for host ganeti5004 * 11:34 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 11:34 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5004.eqsin.wmnet with OS bookworm * 11:34 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 11:33 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1151: Security updates * 11:33 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 11:33 root@cumin1003: START - Cookbook sre.mysql.parsercache * 11:33 root@cumin1003: START - Cookbook sre.mysql.pool pool db1151: Security updates * 11:31 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply * 11:30 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply * 11:30 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply * 11:30 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:23 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 11:23 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:23 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 11:23 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 11:16 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 11:15 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 11:15 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 11:15 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:09 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1151: Security updates * 11:09 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 11:09 root@cumin1003: START - Cookbook sre.mysql.parsercache * 11:09 root@cumin1003: START - Cookbook sre.mysql.depool depool db1151: Security updates * 11:08 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300092{{!}}ProductionServices: re-add poolcounter2006 (T426736)]] (duration: 06m 55s) * 11:04 blake@deploy1003: blake: Continuing with deployment * 11:04 blake@deploy1003: blake: Backport for [[gerrit:1300092{{!}}ProductionServices: re-add poolcounter2006 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:03 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:01 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300092{{!}}ProductionServices: re-add poolcounter2006 (T426736)]] * 10:59 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet * 10:57 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 10:57 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 10:57 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 10:56 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 10:56 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply * 10:56 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply * 10:56 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet * 10:56 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300087{{!}}ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] (duration: 06m 42s) * 10:51 blake@deploy1003: blake: Continuing with deployment * 10:51 moritzm: remove ganeti5004 from eqsin cluster for reimage [[phab:T428229|T428229]] * 10:51 blake@deploy1003: blake: Backport for [[gerrit:1300087{{!}}ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:49 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300087{{!}}ProductionServices: reboot poolcounter2006, re-add poolcounter 2005 (T426736)]] * 10:47 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet * 10:47 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 10:46 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 10:46 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 10:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 10:43 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet * 10:43 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300082{{!}}ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] (duration: 07m 38s) * 10:41 moritzm: installing nginx security updates * 10:38 blake@deploy1003: blake: Continuing with deployment * 10:38 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1152: Security updates * 10:38 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:38 root@cumin1003: START - Cookbook sre.mysql.parsercache * 10:38 root@cumin1003: START - Cookbook sre.mysql.pool pool db1152: Security updates * 10:38 blake@deploy1003: blake: Backport for [[gerrit:1300082{{!}}ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:37 moritzm: failover Ganeti master in eqsin to ganeti5007 [[phab:T428229|T428229]] * 10:35 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300082{{!}}ProductionServices: reboot poolcounter2005, re-add poolcounter 1007 (T426736)]] * 10:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 10:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 10:33 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet * 10:29 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet * 10:29 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300072{{!}}ProductionServices: reboot poolcounter1007 (T426736)]] (duration: 07m 45s) * 10:27 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 10:27 jmm@cumin2002: DONE (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for sretest2009.codfw.wmnet: Renew puppet certificate - jmm@cumin2002 * 10:24 blake@deploy1003: blake: Continuing with deployment * 10:23 blake@deploy1003: blake: Backport for [[gerrit:1300072{{!}}ProductionServices: reboot poolcounter1007 (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:22 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 10:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 10:21 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:21 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300072{{!}}ProductionServices: reboot poolcounter1007 (T426736)]] * 10:21 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:21 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:21 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:21 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:20 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet * 10:14 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1152: Security updates * 10:14 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:14 root@cumin1003: START - Cookbook sre.mysql.parsercache * 10:14 root@cumin1003: START - Cookbook sre.mysql.depool depool db1152: Security updates * 10:13 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet * 10:12 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300064{{!}}ProductionServices: reboot poolcounter1006.eqiad (T426736)]] (duration: 07m 46s) * 10:07 blake@deploy1003: blake: Continuing with deployment * 10:06 blake@deploy1003: blake: Backport for [[gerrit:1300064{{!}}ProductionServices: reboot poolcounter1006.eqiad (T426736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:04 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1300064{{!}}ProductionServices: reboot poolcounter1006.eqiad (T426736)]] * 09:57 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1300058{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] (duration: 09m 32s) * 09:52 kharlan@deploy1003: kharlan: Continuing with deployment * 09:49 kharlan@deploy1003: kharlan: Backport for [[gerrit:1300058{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:47 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1300058{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]], [[gerrit:1300059{{!}}SourceEditorOverlay: Show CAPTCHA panel when AF challenge closed (T425929)]] * 09:35 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox * 09:34 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 09:32 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 09:32 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 09:26 moritzm: upgrade routinator in eqiad to 0.15.2 [[phab:T428456|T428456]] * 09:23 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 09:23 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 09:22 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 09:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of prometheus5003.eqsin.wmnet to plain * 09:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of prometheus5003.eqsin.wmnet to plain * 09:15 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:04 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 09:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 09:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:54 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:20 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:09 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:09 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:07 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 08:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:05 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:04 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:01 fceratto@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host db1215.eqiad.wmnet with OS trixie * 07:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:53 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:48 javiermonton@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply * 07:48 javiermonton@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply * 07:44 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1215.eqiad.wmnet with reason: host reimage * 07:41 javiermonton@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply * 07:40 javiermonton@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply * 07:40 moritzm: installing openssl security updates * 07:39 fceratto@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1215.eqiad.wmnet with reason: host reimage * 07:38 javiermonton@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply * 07:37 javiermonton@deploy1003: helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply * 07:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:29 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299556{{!}}ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561{{!}}ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529{{!}}translate: adding separate read/write endpoints (T425377)]] (duration: 14m 03s) * 07:25 atsuko@deploy1003: atsuko: Continuing with deployment * 07:23 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db1215.eqiad.wmnet with OS trixie * 07:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1215.eqiad.wmnet with reason: Reimage * 07:21 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:20 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:20 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:17 atsuko@deploy1003: atsuko: Backport for [[gerrit:1299556{{!}}ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561{{!}}ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529{{!}}translate: adding separate read/write endpoints (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be veri * 07:16 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:15 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1299556{{!}}ElasticSearchTtmServer: drop include_type_name and support int replicas (T428168)]], [[gerrit:1299561{{!}}ElasticSearchTtmServer: clean stale _doc usage and version error output (T428168)]], [[gerrit:1299529{{!}}translate: adding separate read/write endpoints (T425377)]] * 07:14 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:12 atsukoito: backporting extensions/Translate to wmf/1.47.0-wmf.5 and applying the config * 07:12 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 07:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 07:11 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 06:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 06:45 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 05:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 05:43 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 05:42 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet * 05:41 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 47s) * 02:07 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1008.eqiad.wmnet with OS trixie * 02:03 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync * 02:02 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:52 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:51 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 01:51 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:50 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:50 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 01:49 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1008.eqiad.wmnet with reason: host reimage * 01:49 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 01:49 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 01:49 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 01:49 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 01:48 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 01:48 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 01:47 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 01:47 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 01:46 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 01:46 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 01:45 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 01:44 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 01:44 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 01:43 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 01:43 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1008.eqiad.wmnet with reason: host reimage * 01:25 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main1008 * 01:24 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main1008 * 01:24 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main1008 * 01:24 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main1008.eqiad.wmnet 45.32.64.10.in-addr.arpa 5.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 01:23 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main1008.eqiad.wmnet 45.32.64.10.in-addr.arpa 5.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 01:23 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 01:23 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1008 - jasmine@cumin2002" * 01:23 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main1008 - jasmine@cumin2002" * 01:19 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 01:12 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main1008 * 01:11 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1008.eqiad.wmnet with OS trixie * 01:00 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2009.codfw.wmnet with OS trixie * 00:54 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync * 00:53 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync * 00:43 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2009.codfw.wmnet with reason: host reimage * 00:40 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:39 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 00:38 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2009.codfw.wmnet with reason: host reimage * 00:38 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 00:38 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 00:37 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 00:37 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 00:36 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 00:36 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 00:35 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 00:34 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 00:34 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 00:33 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 00:33 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 00:32 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 00:32 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 00:32 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2009 * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2009 * 00:15 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2009 * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2009.codfw.wmnet 33.48.192.10.in-addr.arpa 3.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:15 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2009.codfw.wmnet 33.48.192.10.in-addr.arpa 3.3.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 00:15 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2009 - jasmine@cumin2002" * 00:15 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2009 - jasmine@cumin2002" * 00:10 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 00:03 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2009 * 00:03 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2009.codfw.wmnet with OS trixie == 2026-06-09 == * 22:50 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299640{{!}}HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] (duration: 08m 59s) * 22:45 cscott@deploy1003: cscott: Continuing with deployment * 22:43 cscott@deploy1003: cscott: Backport for [[gerrit:1299640{{!}}HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:41 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299640{{!}}HandleSectionLinks: add temporary fallback to identify html headings (T428677)]] * 22:15 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299639{{!}}[Bug] Donor Badge: Remove client prefs for control group (T428501)]] (duration: 20m 57s) * 22:11 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 22:07 mutante: gerrit - apache httpd log file location moved to /srv/gerrit/site_path/review_site/logs/ [[phab:T425667|T425667]] * 22:06 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on gerrit2003.wikimedia.org with reason: debug * 21:56 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1299639{{!}}[Bug] Donor Badge: Remove client prefs for control group (T428501)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:54 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299639{{!}}[Bug] Donor Badge: Remove client prefs for control group (T428501)]] * 21:52 ryankemper: [[phab:T428241|T428241]] removed retired wdqs2009 full-graph journal dump (446G x2, ~892G) from clouddumps100[1-2]:/srv/dumps/xmldatadumps/public/other/wdqs * 21:49 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299602{{!}}Revert "Create VectorComponentPageToolbar component" (T428649)]] (duration: 08m 16s) * 21:48 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 21:45 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 21:43 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1299602{{!}}Revert "Create VectorComponentPageToolbar component" (T428649)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:41 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1299602{{!}}Revert "Create VectorComponentPageToolbar component" (T428649)]] * 21:34 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gerrit1003.wikimedia.org with reason: debug * 21:27 maryum: Deployed security fix for [[phab:T428324|T428324]] * 21:24 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 21:15 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart * 21:06 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart * 20:50 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs2002.codfw.wmnet with OS trixie * 20:50 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299588{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] (duration: 11m 13s) * 20:46 cscott@deploy1003: cscott: Continuing with deployment * 20:43 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs2002.codfw.wmnet with OS trixie * 20:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:42 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:41 cscott@deploy1003: cscott: Backport for [[gerrit:1299588{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:39 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299588{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T378906 T420336 T424427 T427664 T427972 T428452 T428270)]], [[gerrit:1299589{{!}}Bump wikimedia/parsoid to 0.24.0-a8 (T428270)]] * 20:38 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:38 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:33 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:33 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:32 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299454{{!}}wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] (duration: 22m 08s) * 20:28 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:28 cscott@deploy1003: cscott, gkyziridis: Continuing with deployment * 20:24 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2004 * 20:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2004 * 20:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2003 * 20:14 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2003 * 20:14 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2002 * 20:13 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2002 * 20:13 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2001 * 20:13 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2001 * 20:12 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:12 cscott@deploy1003: cscott, gkyziridis: Backport for [[gerrit:1299454{{!}}wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:10 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1299454{{!}}wgRestSandboxSpecs: Add lift-wing spec pointing to api.wikimedia.org (T427902)]] * 20:09 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 20:04 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:59 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:54 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:53 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:48 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:47 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:47 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:46 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:46 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:45 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:45 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:28 ryankemper@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts wdqs1015.eqiad.wmnet * 19:28 ryankemper@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:28 ryankemper@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002" * 19:27 ryankemper@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002" * 19:20 ryankemper@cumin2002: START - Cookbook sre.dns.netbox * 19:15 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2008.codfw.wmnet with OS trixie * 19:15 ryankemper@cumin2002: START - Cookbook sre.hosts.decommission for hosts wdqs1015.eqiad.wmnet * 19:12 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync * 19:12 jasmine@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: sync * 19:00 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:58 jasmine@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 18:58 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2008.codfw.wmnet with reason: host reimage * 18:58 jasmine@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 18:58 jasmine@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 18:57 jasmine@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 18:57 jasmine@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 18:56 jasmine@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 18:56 jasmine@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 18:55 jasmine@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 18:55 jasmine@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 18:55 jasmine@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 18:54 jasmine@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 18:54 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:54 jasmine@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 18:53 jasmine@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 18:53 jasmine@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 18:53 jasmine@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 18:52 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:52 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2003 to codfw - jhancock@cumin2002" * 18:52 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2003 to codfw - jhancock@cumin2002" * 18:52 jasmine@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 18:52 jasmine@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 18:51 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2008.codfw.wmnet with reason: host reimage * 18:51 jasmine@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 18:51 jasmine@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 18:51 jasmine@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 18:50 jasmine@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 18:50 jasmine@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 18:47 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:47 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:47 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:46 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:46 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:42 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:42 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:31 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 18:29 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2008.codfw.wmnet with OS trixie * 18:26 jasmine@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2008.codfw.wmnet with OS trixie * 17:48 mutante: https://releases.wikimedia.org {{!}} https://releases-jenkins.wikimedia.org - down for maintenance [[phab:T418299|T418299]] * 17:48 cmooney@dns2005: END - running authdns-update * 17:47 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases2003.codfw.wmnet with reason: reimage * 17:47 cmooney@dns2005: START - running authdns-update * 17:46 sukhe: sudo cumin 'A:hcaptcha-proxy' 'run-puppet-agent': rolling out CR {{Gerrit|1299427}} [[phab:T428539|T428539]] * 17:43 jayme: kafka-main2008 is down due to hardware failure [[phab:T428654|T428654]] * 17:32 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf1002.eqiad.wmnet with OS trixie * 17:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf1002.eqiad.wmnet with reason: host reimage * 17:06 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf1002.eqiad.wmnet with reason: host reimage * 17:05 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-main2008 * 17:05 jasmine@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2008 * 17:04 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 17:04 jasmine@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2008 * 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-main2008.codfw.wmnet 4.32.192.10.in-addr.arpa 4.0.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:04 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 17:04 jasmine@cumin2002: START - Cookbook sre.dns.wipe-cache kafka-main2008.codfw.wmnet 4.32.192.10.in-addr.arpa 4.0.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:04 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2008 - jasmine@cumin2002" * 17:04 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host cp5018 * 17:04 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-main2008 - jasmine@cumin2002" * 17:03 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5018.eqsin.wmnet with OS trixie * 16:58 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 16:58 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 16:57 jasmine@cumin2002: START - Cookbook sre.dns.netbox * 16:57 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 16:57 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 16:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 16:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 16:50 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf1002.eqiad.wmnet with OS trixie * 16:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 16:47 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf1001.eqiad.wmnet with OS trixie * 16:47 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply * 16:47 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply * 16:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 16:41 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 16:41 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 16:35 jasmine@cumin2002: START - Cookbook sre.hosts.move-vlan for host kafka-main2008 * 16:34 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2008.codfw.wmnet with OS trixie * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:31 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply * 16:30 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply * 16:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf1001.eqiad.wmnet with reason: host reimage * 16:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:26 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf1001.eqiad.wmnet with reason: host reimage * 16:23 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply * 16:22 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: apply * 16:20 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:16 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:15 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:13 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:13 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:12 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf1001.eqiad.wmnet with OS trixie * 16:10 jiji@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. * 16:09 jiji@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. * 16:07 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf2002.codfw.wmnet with OS trixie * 16:02 jiji@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 16:02 jiji@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 16:00 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. * 15:59 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 15:59 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. * 15:59 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 15:59 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. * 15:59 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 15:58 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] DONE helmfile.d/services/termbox: apply * 15:58 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] START helmfile.d/services/termbox: apply * 15:57 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 15:57 jiji@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 15:57 lucaswerkmeister-wmde@deploy1003: helmfile [staging] DONE helmfile.d/services/termbox: apply * 15:56 lucaswerkmeister-wmde@deploy1003: helmfile [staging] START helmfile.d/services/termbox: apply * 15:54 jiji@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 15:53 jiji@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 15:51 jiji@deploy1003: Finished scap sync-world: redeploy {{Gerrit|1299468}} (duration: 07m 23s) * 15:49 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf2002.codfw.wmnet with reason: host reimage * 15:47 jiji@deploy1003: jiji: Continuing with deployment * 15:46 jiji@deploy1003: jiji: redeploy {{Gerrit|1299468}} synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:46 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf2002.codfw.wmnet with reason: host reimage * 15:45 jiji@deploy1003: Started scap sync-world: redeploy {{Gerrit|1299468}} * 15:43 brouberol@cumin1003: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-eqiad * 15:34 brennen@deploy1003: Finished deploy [phabricator/deployment@73e57ce]: deploy phab1004 for [[phab:T410849|T410849]] (followup for robots.txt) (duration: 00m 40s) * 15:33 brennen@deploy1003: Started deploy [phabricator/deployment@73e57ce]: deploy phab1004 for [[phab:T410849|T410849]] (followup for robots.txt) * 15:33 brennen@deploy1003: Finished deploy [phabricator/deployment@73e57ce]: deploy phab2002 for [[phab:T410849|T410849]] (followup for robots.txt) (duration: 00m 45s) * 15:32 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299468{{!}}ProductionServices.php: switch filebackend.php to rdb2015:6381 #2 (T418918 T291916)]] (duration: 07m 21s) * 15:32 brennen@deploy1003: Started deploy [phabricator/deployment@73e57ce]: deploy phab2002 for [[phab:T410849|T410849]] (followup for robots.txt) * 15:28 jiji@deploy1003: Rolling back deployment * 15:27 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf2002.codfw.wmnet with OS trixie * 15:27 jiji@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 15:26 jiji@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 15:25 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1299468{{!}}ProductionServices.php: switch filebackend.php to rdb2015:6381 #2 (T418918 T291916)]] * 15:22 urbanecm: Remove `migrateMentorStatusAwayToCommunityConfiguration` from updatelog on all wikis ([[phab:T409170|T409170]]; the script was only ever run as a dry-run) * 15:21 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 15:21 jiji@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 15:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-wf2001.codfw.wmnet with OS trixie * 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@d244a3e]: deploy phab1004 for [[phab:T410849|T410849]] (duration: 00m 42s) * 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@d244a3e]: deploy phab1004 for [[phab:T410849|T410849]] * 15:02 brennen@deploy1003: Finished deploy [phabricator/deployment@d244a3e]: deploy phab2002 for [[phab:T410849|T410849]] (duration: 00m 45s) * 15:01 brennen@deploy1003: Started deploy [phabricator/deployment@d244a3e]: deploy phab2002 for [[phab:T410849|T410849]] * 14:58 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf2001.codfw.wmnet with reason: host reimage * 14:52 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf2001.codfw.wmnet with reason: host reimage * 14:52 arnaudb@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phab[2002-2003].codfw.wmnet,phab[1004-1006].eqiad.wmnet with reason: [[phab:T410849|T410849]] * 14:47 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 14:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 14:40 moritzm: upgrade routinator in codfw to 0.15.2 [[phab:T428456|T428456]] * 14:35 brouberol@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad * 14:33 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc-wf2001.codfw.wmnet with OS trixie * 14:26 brouberol@cumin1003: END (ERROR) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=97) rolling reboot on A:cephosd-eqiad * 14:26 brouberol@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad * 14:20 btullis@cumin1003: END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-codfw * 14:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host parsoidtest1001.eqiad.wmnet * 14:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2153: Migration of db2153.codfw.wmnet completed * 14:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of rpki2003.codfw.wmnet to drbd * 14:14 moritzm: imported routinator 0.15.2-1bookworm to thirdparty/routinator for bookworm-wikimedia [[phab:T428456|T428456]] * 14:12 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:12 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1184: Migration of db1184.eqiad.wmnet completed * 14:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host parsoidtest1001.eqiad.wmnet * 14:07 Dreamy_Jazz: Afternoon UTC backport window done * 14:07 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 14:06 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299495{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502{{!}}SecurePollLogPager: Cast user IDs to ints before use (T428599)]] (duration: 06m 53s) * 14:06 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 14:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: rack depool * 14:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of rpki2003.codfw.wmnet to drbd * 14:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow2004.codfw.wmnet to drbd * 14:02 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:02 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1299495{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502{{!}}SecurePollLogPager: Cast user IDs to ints before use (T428599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:59 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1299495{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]], [[gerrit:1299502{{!}}SecurePollLogPager: Cast user IDs to ints before use (T428599)]] * 13:58 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:58 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply * 13:56 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 13:56 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 13:55 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:55 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * {{safesubst:SAL entry|1=13:55 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298929{{!}}Simplify fragment processing (T423700)]], [[gerrit:1298926{{!}}Move ::getFragmentsToTransform() to Content<nowiki>{</nowiki>Text,DOM<nowiki>}</nowiki>TransformStage]], [[gerrit:1298927{{!}}OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925{{!}}Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497}} * 13:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:51 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow2004.codfw.wmnet to drbd * 13:50 cscott@deploy1003: cscott: Continuing with deployment * 13:50 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2045.codfw.wmnet to cluster codfw and group A * 13:48 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2045.codfw.wmnet to cluster codfw and group A * 13:48 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2027.codfw.wmnet to cluster codfw and group A * 13:47 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2027.codfw.wmnet to cluster codfw and group A * 13:46 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 13:45 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 13:44 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * {{safesubst:SAL entry|1=13:42 cscott@deploy1003: cscott: Backport for [[gerrit:1298929{{!}}Simplify fragment processing (T423700)]], [[gerrit:1298926{{!}}Move ::getFragmentsToTransform() to Content<nowiki>{</nowiki>Text,DOM<nowiki>}</nowiki>TransformStage]], [[gerrit:1298927{{!}}OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925{{!}}Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497{{!}}Store indicators}} * 13:41 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * {{safesubst:SAL entry|1=13:40 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1298929{{!}}Simplify fragment processing (T423700)]], [[gerrit:1298926{{!}}Move ::getFragmentsToTransform() to Content<nowiki>{</nowiki>Text,DOM<nowiki>}</nowiki>TransformStage]], [[gerrit:1298927{{!}}OutputTransform: Rename DeduplicateStyles and ExpandToAbsoluteUrls stages]], [[gerrit:1298925{{!}}Reset DeduplicateStyles state between different pipeline executions (T428336 T428215)]], [[gerrit:1299497{{!}}}} * 13:40 btullis@cumin1003: START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-codfw * 13:39 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 13:37 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 13:35 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 13:33 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 13:32 ayounsi@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 13:32 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298834{{!}}config: Disable EmailConfirmationBanner on all wikis (T428291)]] (duration: 07m 01s) * 13:30 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2153: Migration of db2153.codfw.wmnet completed * 13:28 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 13:28 lucaswerkmeister-wmde@deploy1003: mmartorana, lucaswerkmeister-wmde: Continuing with deployment * 13:27 lucaswerkmeister-wmde@deploy1003: mmartorana, lucaswerkmeister-wmde: Backport for [[gerrit:1298834{{!}}config: Disable EmailConfirmationBanner on all wikis (T428291)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:26 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1184: Migration of db1184.eqiad.wmnet completed * 13:25 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298834{{!}}config: Disable EmailConfirmationBanner on all wikis (T428291)]] * 13:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 13:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 13:23 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:21 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 13:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2153.codfw.wmnet with OS trixie * 13:20 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2241: rack depool * 13:20 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1237: repool after maintenance db1237 * 13:19 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298654{{!}}Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] (duration: 09m 40s) * 13:17 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2006.codfw.wmnet * 13:17 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2006.codfw.wmnet * 13:16 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2251-2253].codfw.wmnet * 13:16 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2251-2253].codfw.wmnet * 13:16 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2005.codfw.wmnet * 13:16 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2005.codfw.wmnet * 13:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1184.eqiad.wmnet with OS trixie * 13:14 lucaswerkmeister-wmde@deploy1003: neriah, lucaswerkmeister-wmde: Continuing with deployment * 13:11 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 * 13:11 lucaswerkmeister-wmde@deploy1003: neriah, lucaswerkmeister-wmde: Backport for [[gerrit:1298654{{!}}Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:09 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298654{{!}}Enable wgNewUserMessageOnFirstEdit on commonswiki (T426206)]] * 13:04 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:04 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:04 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2153.codfw.wmnet with reason: host reimage * 13:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 13:03 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1015.eqiad.wmnet with OS trixie * 12:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage * 12:58 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2153.codfw.wmnet with reason: host reimage * 12:57 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb1016.eqiad.wmnet with OS trixie * 12:57 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 * 12:56 XioNoX: lsw1-a4-codfw> request system reboot - [[phab:T427357|T427357]] * 12:55 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:53 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage * 12:50 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299477{{!}}hCaptcha: Roll out to all wikis for api account creation. (T426050)]] (duration: 07m 21s) * 12:46 kharlan@deploy1003: kharlan, dbrant: Continuing with deployment * 12:46 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 * 12:45 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1015.eqiad.wmnet with reason: host reimage * 12:45 kharlan@deploy1003: kharlan, dbrant: Backport for [[gerrit:1299477{{!}}hCaptcha: Roll out to all wikis for api account creation. (T426050)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:45 topranks: shut sub-interfaces for row A/B legacy vlans on cr1-codfw [[phab:T427357|T427357]] * 12:45 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 * 12:43 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1299477{{!}}hCaptcha: Roll out to all wikis for api account creation. (T426050)]] * 12:42 topranks: increase OSPF cost on ssw1-a1-codfw link to lsw1-a4-codfw to force traffic via alternate spine [[phab:T427357|T427357]] * 12:41 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299478{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]] (duration: 07m 02s) * 12:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1016.eqiad.wmnet with reason: host reimage * 12:40 moritzm: installing wireshark security updates * 12:40 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2153.codfw.wmnet with OS trixie * 12:38 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1184.eqiad.wmnet with OS trixie * 12:37 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 12:36 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1299478{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:34 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2153: Upgrading db2153.codfw.wmnet * 12:34 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1237: repool after maintenance db1237 * 12:34 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1299478{{!}}STVFormatter: Cast strings to float before passing to round (T428584)]] * 12:34 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2153: Upgrading db2153.codfw.wmnet * 12:34 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1184: Upgrading db1184.eqiad.wmnet * 12:33 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1184: Upgrading db1184.eqiad.wmnet * 12:33 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:32 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1237.eqiad.wmnet with OS trixie * 12:32 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1015.eqiad.wmnet with reason: host reimage * 12:32 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1016.eqiad.wmnet with reason: host reimage * 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 12:27 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2005.codfw.wmnet * 12:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2046: repool after maintenance * 12:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2006.codfw.wmnet * 12:23 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298829{{!}}wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] (duration: 16m 04s) * 12:23 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2006.codfw.wmnet * 12:22 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2251-2253].codfw.wmnet * 12:22 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2005.codfw.wmnet * 12:20 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2251-2253].codfw.wmnet * 12:20 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply * 12:20 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: rack depool * 12:20 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply * 12:20 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2241: rack depool * 12:19 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1016 * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1016 * 12:19 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host rdb1015 * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.move-vlan for host rdb1015 * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1016.eqiad.wmnet with OS trixie * 12:19 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host rdb1015.eqiad.wmnet with OS trixie * 12:17 ayounsi@cumin1003: END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 * 12:17 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 24 hosts with reason: Rack A4 depool * 12:16 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Continuing with deployment * 12:15 topranks: drain traffic on ssw1-a1-codfw - add gshut community in evpn underlay - [[phab:T427357|T427357]] * 12:14 ayounsi@cumin1003: START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 * 12:13 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Backport for [[gerrit:1298829{{!}}wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:10 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1237.eqiad.wmnet with reason: host reimage * 12:07 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1298829{{!}}wmf-config: Enable hCaptcha on UploadWizard publish for testwiki (T426126)]] * 12:05 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1237.eqiad.wmnet with reason: host reimage * 12:00 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Dmaza out of all services on: 2435 hosts * 11:51 atsuko@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 11:51 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie * 11:49 atsuko@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 11:48 atsuko@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 11:47 atsuko@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 11:45 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 11:44 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 11:43 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:43 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2046: repool after maintenance * 11:38 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 11:36 fceratto@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:36 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2046.codfw.wmnet with OS trixie * 11:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2185.codfw.wmnet with reason: Reimage * 11:31 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging HMonroy out of all services on: 2435 hosts * 11:28 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging KSiebert out of all services on: 2435 hosts * 11:26 slyngs: CAS-SSO upgrade to version 7.3.7.2 * 11:26 slyngshede@dns1004: END - running authdns-update * 11:24 slyngshede@dns1004: START - running authdns-update * 11:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2046.codfw.wmnet with reason: host reimage * 11:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1043: repool after upgrade * 11:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2046.codfw.wmnet with reason: host reimage * 10:55 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2046.codfw.wmnet with OS trixie * 10:53 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2046: Upgrading es2046.codfw.wmnet * 10:53 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2046: Upgrading es2046.codfw.wmnet * 10:52 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply * 10:52 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:52 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply * 10:52 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply * 10:52 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:52 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply * 10:51 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:32 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1043: repool after upgrade * 10:31 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:28 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1160: Repooling * 10:26 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1043.eqiad.wmnet with OS trixie * 10:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:17 elukey: complete rollout of apache2 upgrades * 10:16 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:15 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:13 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:13 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:12 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:12 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:08 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1043.eqiad.wmnet with reason: host reimage * 10:04 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:04 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1043.eqiad.wmnet with reason: host reimage * 10:04 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:04 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:04 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:04 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:04 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:57 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1160: Repooling * 09:51 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 09:51 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 09:50 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 09:50 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 09:49 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1043.eqiad.wmnet with OS trixie * 09:48 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool es1043: Upgrading es1043.eqiad.wmnet * 09:48 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 09:47 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:45 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 09:41 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 09:36 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=5 --verbose --last-checked="20260603"` (after stopping previous scan run) * 09:34 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=5 --verbose` (after stopping previous scan run) * 09:27 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 09:26 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 09:17 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 09:17 fceratto@cumin1003: MariaDB change: Setting sections s5 as read-write * 09:17 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 09:14 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1043: Upgrading es1043.eqiad.wmnet * 09:14 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:12 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1042 to es4 eqiad primary [[phab:T428386|T428386]]', diff saved to https://phabricator.wikimedia.org/P93943 and previous config saved to /var/cache/conftool/dbconfig/20260609-091215-marostegui.json * 09:11 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1043 to es4 eqiad primary [[phab:T428386|T428386]]', diff saved to https://phabricator.wikimedia.org/P93942 and previous config saved to /var/cache/conftool/dbconfig/20260609-091147-marostegui.json * 09:03 jiji@cumin1003: conftool action : set/pooled=yes; selector: service=docker-registry,name=registry2005.codfw.wmnet * 08:59 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:59 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:57 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1237.eqiad.wmnet with OS trixie * 08:55 jiji@cumin1003: conftool action : set/pooled=no; selector: service=docker-registry,name=registry2005.codfw.wmnet * 08:55 jiji@cumin1003: conftool action : set/pooled=yes; selector: service=docker-registry,name=registry2004.codfw.wmnet * 08:50 jiji@cumin1003: conftool action : set/pooled=no; selector: service=docker-registry,name=registry2004.codfw.wmnet * 08:22 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=docker-registry,name=codfw * 08:22 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=docker-registry,name=eqiad * 08:08 jiji@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=docker-registry,name=eqiad * 08:08 jiji@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=docker-registry,name=codfw * 07:59 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:59 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix typoes - ayounsi@cumin1003" * 07:59 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix typoes - ayounsi@cumin1003" * 07:52 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 07:47 brouberol@dns1004: END - running authdns-update * 07:46 brouberol@dns1004: START - running authdns-update * 07:44 brouberol@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/kafka-ui: apply * 07:43 brouberol@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/kafka-ui: apply * 07:43 brouberol@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/kafka-ui: apply * 07:42 brouberol@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/kafka-ui: apply * 07:41 brouberol@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/kafka-ui: apply * 07:39 brouberol@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/kafka-ui: apply * 07:38 brouberol@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:37 brouberol@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 07:37 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie * 07:36 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.major-upgrade (exit_code=97) * 07:36 brouberol@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:36 brouberol@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:26 fceratto@dns1004: END - running authdns-update * 07:24 fceratto@dns1004: START - running authdns-update * 07:22 marostegui@dns1004: END - running authdns-update * 07:21 marostegui@dns1004: START - running authdns-update * 07:19 elukey@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:19 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Fix dse-k8s-wdqs2002 duplicate ipv6 address - elukey@cumin1003" * 07:19 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Fix dse-k8s-wdqs2002 duplicate ipv6 address - elukey@cumin1003" * 07:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1160.eqiad.wmnet with reason: Maintenance * 07:12 elukey@cumin1003: START - Cookbook sre.dns.netbox * 07:11 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1160: Repooling * 07:11 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1160: Repooling * 07:11 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1160: Repooling * 07:11 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1160: Repooling * 07:00 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:00 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1237.eqiad.wmnet with OS trixie * 06:24 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1160 [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93940 and previous config saved to /var/cache/conftool/dbconfig/20260609-062412-fceratto.json * 06:17 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 06:16 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 06:16 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 06:16 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 06:15 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 06:15 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 06:15 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 06:14 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 06:12 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1244 to s4 primary and set section read-write [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93939 and previous config saved to /var/cache/conftool/dbconfig/20260609-061222-fceratto.json * 06:11 fceratto@cumin1003: dbctl commit (dc=all): 'Set s4 eqiad as read-only for maintenance - [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93938 and previous config saved to /var/cache/conftool/dbconfig/20260609-061131-fceratto.json * 06:10 federico3: Starting s4 eqiad failover from db1160 to db1244 - [[phab:T426086|T426086]] * 06:01 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1244 with weight 0 [[phab:T426086|T426086]]', diff saved to https://phabricator.wikimedia.org/P93937 and previous config saved to /var/cache/conftool/dbconfig/20260609-060121-fceratto.json * 06:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 40 hosts with reason: Primary switchover s4 [[phab:T426086|T426086]] * 05:40 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie * 05:37 marostegui@dns1004: START - running authdns-update * 05:27 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1237: Upgrading db1237.eqiad.wmnet * 05:27 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1237: Upgrading db1237.eqiad.wmnet * 05:27 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:24 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1237 [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93935 and previous config saved to /var/cache/conftool/dbconfig/20260609-052420-marostegui.json * 05:23 marostegui@dns1004: START - running authdns-update * 05:23 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db1220 to x1 primary and set section read-write [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93934 and previous config saved to /var/cache/conftool/dbconfig/20260609-052311-marostegui.json * 05:22 marostegui@cumin1003: dbctl commit (dc=all): 'Set x1 eqiad as read-only for maintenance - [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93933 and previous config saved to /var/cache/conftool/dbconfig/20260609-052253-marostegui.json * 05:22 marostegui: Starting x1 eqiad failover from db1237 to db1220 - [[phab:T428158|T428158]] * 05:19 marostegui@cumin1003: dbctl commit (dc=all): 'Set db1220 with weight 0 [[phab:T428158|T428158]]', diff saved to https://phabricator.wikimedia.org/P93932 and previous config saved to /var/cache/conftool/dbconfig/20260609-051859-marostegui.json * 05:18 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: Primary switchover x1 [[phab:T428158|T428158]] * 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.3 (duration: 02m 43s) * 03:40 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] (duration: 37m 16s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.6 refs [[phab:T423915|T423915]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 38s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-06-08 == * 22:00 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298915{{!}}CommonSettings: Set $wgScoreSafeMode = false (T428484)]] (duration: 07m 42s) * 21:56 reedy@deploy1003: reedy: Continuing with deployment * 21:54 reedy@deploy1003: reedy: Backport for [[gerrit:1298915{{!}}CommonSettings: Set $wgScoreSafeMode = false (T428484)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:53 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1298915{{!}}CommonSettings: Set $wgScoreSafeMode = false (T428484)]] * 21:12 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298891{{!}}OOUIHTMLForm: Avoid treating form header as a clickable label (T428359)]] (duration: 08m 10s) * 21:07 mlitn@deploy1003: mlitn, neriah: Continuing with deployment * 21:05 mlitn@deploy1003: mlitn, neriah: Backport for [[gerrit:1298891{{!}}OOUIHTMLForm: Avoid treating form header as a clickable label (T428359)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:03 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1298891{{!}}OOUIHTMLForm: Avoid treating form header as a clickable label (T428359)]] * 20:43 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297162{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias]], [[gerrit:1298841{{!}}Squashed diff to master]] (duration: 07m 05s) * 20:39 mlitn@deploy1003: mlitn: Continuing with deployment * 20:38 mlitn@deploy1003: mlitn: Backport for [[gerrit:1297162{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias]], [[gerrit:1298841{{!}}Squashed diff to master]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:36 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1297162{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias]], [[gerrit:1298841{{!}}Squashed diff to master]] * 20:29 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298390{{!}}English Wikibooks: update FlaggedRevs configuration (T428329)]], [[gerrit:1298328{{!}}English Wikiversity: Add new user group "autopatrolled" (T428269)]] (duration: 08m 58s) * 20:25 mlitn@deploy1003: mlitn, vadymts1: Continuing with deployment * 20:22 mlitn@deploy1003: mlitn, vadymts1: Backport for [[gerrit:1298390{{!}}English Wikibooks: update FlaggedRevs configuration (T428329)]], [[gerrit:1298328{{!}}English Wikiversity: Add new user group "autopatrolled" (T428269)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1298390{{!}}English Wikibooks: update FlaggedRevs configuration (T428329)]], [[gerrit:1298328{{!}}English Wikiversity: Add new user group "autopatrolled" (T428269)]] * 20:03 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298879{{!}}SimpleCaptcha: Re-render captcha when edit form is redisplayed (T428437)]] (duration: 37m 43s) * 19:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:31 kharlan@deploy1003: kharlan: Continuing with deployment * 19:30 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:30 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:29 kharlan@deploy1003: kharlan: Backport for [[gerrit:1298879{{!}}SimpleCaptcha: Re-render captcha when edit form is redisplayed (T428437)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 19:28 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:27 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:25 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1298879{{!}}SimpleCaptcha: Re-render captcha when edit form is redisplayed (T428437)]] * 19:24 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab (duration: 01m 32s) * 19:23 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:22 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab * 19:20 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab (duration: 01m 40s) * 19:19 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab * 19:16 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:14 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:06 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:59 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 18:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2004 * 18:52 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2004 * 18:52 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2003 * 18:52 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2003 * 18:51 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:51 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2004 to codfw - jhancock@cumin2002" * 18:51 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2004 to codfw - jhancock@cumin2002" * 18:44 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:42 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:42 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2030 to codfw - jhancock@cumin2002" * 18:42 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2030 to codfw - jhancock@cumin2002" * 18:37 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:33 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2002 * 18:32 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2002 * 18:31 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:31 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2002 to codfw - jhancock@cumin2002" * 18:31 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs2002 to codfw - jhancock@cumin2002" * 18:25 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:22 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-wdqs2001 * 18:22 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-wdqs2001 * 18:21 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:21 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: updating dse-k8s-wdqs2001 to codfw - jhancock@cumin2002" * 18:21 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: updating dse-k8s-wdqs2001 to codfw - jhancock@cumin2002" * 18:17 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 18:02 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T427286|T427286]] (duration: 00m 12s) * 18:02 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T427286|T427286]] * 17:37 jnuche@deploy1003: Installation of scap version "4.268.0" completed for 2 hosts * 17:35 jnuche@deploy1003: Installing scap version "4.268.0" for 2 host(s) * 17:21 claime: restarting varnish-frontend service on cp6012 * 17:21 claime: restarting varnish-frontend service on cp6011 * 17:21 claime: restarted varnish-frontend service on cp6009 * 17:13 taavi: bounce sirenbot to get it to re-join a channel * 17:05 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 17:05 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:58 urbanecm@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 16:57 urbanecm@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 16:55 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 16:53 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 16:53 urbanecm@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 16:52 urbanecm@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 16:30 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 16:29 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 16:29 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 16:28 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 16:28 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 16:28 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 16:28 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 16:27 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 16:27 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 16:26 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 16:26 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 16:25 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 16:18 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 16:17 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 16:17 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 16:16 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 16:16 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 16:16 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 16:16 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 16:15 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 16:14 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 16:14 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 16:14 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 16:14 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 16:13 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 16:13 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 16:13 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 16:12 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 16:12 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 16:10 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 16:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 16:10 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 16:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 16:10 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 16:10 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 16:09 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 16:08 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 16:08 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 16:07 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 16:06 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 15:57 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2042: repool after upgrade * 15:45 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db[2183-2184].codfw.wmnet * 15:45 jynus@cumin2002: START - Cookbook sre.hosts.remove-downtime for db[2183-2184].codfw.wmnet * 15:18 jynus: dbmaint on backup1-codfw@codfw ([[phab:T428467|T428467]]) * 15:12 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2042: repool after upgrade * 15:12 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 15:09 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 15:09 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 15:09 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 15:08 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 15:07 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2042.codfw.wmnet with OS trixie * 15:04 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 15:04 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 15:03 jynus@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db[2183-2184].codfw.wmnet with reason: Switchover db * 15:03 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 15:03 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 15:02 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 15:01 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply * 15:00 eevans@deploy1003: helmfile [staging] START helmfile.d/services/data-gateway: apply * 14:59 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:55 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:55 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:54 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:50 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:50 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:50 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:49 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:49 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2042.codfw.wmnet with reason: host reimage * 14:42 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2042.codfw.wmnet with reason: host reimage * 14:32 Lucas_WMDE: UTC afternoon backport+config window done * 14:32 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298709{{!}}Add translatable messages for WikiProject names (T427804)]], [[gerrit:1298710{{!}}Use translatable messages for WikiProject links (T427804)]], [[gerrit:1297644{{!}}WikiProject links - remove 'text' config (T427804)]] (duration: 31m 57s) * 14:27 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 14:26 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2042.codfw.wmnet with OS trixie * 14:26 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 14:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2042: Upgrading es2042.codfw.wmnet * 14:25 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2042: Upgrading es2042.codfw.wmnet * 14:25 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:24 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2043 to es4 codfw primary [[phab:T428386|T428386]]', diff saved to https://phabricator.wikimedia.org/P93926 and previous config saved to /var/cache/conftool/dbconfig/20260608-142423-marostegui.json * 14:23 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 14:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1041: repool after maintenance * 14:19 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Continuing with deployment * 14:18 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Backport for [[gerrit:1298709{{!}}Add translatable messages for WikiProject names (T427804)]], [[gerrit:1298710{{!}}Use translatable messages for WikiProject links (T427804)]], [[gerrit:1297644{{!}}WikiProject links - remove 'text' config (T427804)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:11 cgoubert@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=liftwing-openapi-server.* * 14:10 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp6013.* * 14:10 cgoubert@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:05 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 14:05 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 13:54 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 13:52 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 13:50 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 13:50 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 13:50 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296550{{!}}hCaptcha: Don't show AbuseFilter CAPTCHA for wbsetclaim API (T427608)]] (duration: 08m 31s) * 13:48 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 13:46 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 13:43 cgoubert@dns1004: END - running authdns-update * 13:43 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296550{{!}}hCaptcha: Don't show AbuseFilter CAPTCHA for wbsetclaim API (T427608)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:41 cgoubert@dns1004: START - running authdns-update * 13:41 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296550{{!}}hCaptcha: Don't show AbuseFilter CAPTCHA for wbsetclaim API (T427608)]] * 13:39 urbanecm@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * {{safesubst:SAL entry|1=13:38 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298758{{!}}feat(V2): toggle experiment features based on custom url override (T424646)]], [[gerrit:1298762{{!}}specialCreateAccount: use GECreateAccountExperimentV2 instead of hook (T424646)]], [[gerrit:1298764{{!}}fix: correctly read experiments param on Special:UserLogin]], [[gerrit:1298765{{!}}signup.js: use JS var instead of TestKitchen to show exp}} * 13:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1041: repool after maintenance * 13:38 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 13:38 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:37 urbanecm@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 13:36 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 13:35 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1041.eqiad.wmnet with OS trixie * 13:34 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 13:34 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2041: repool after upgrade * 13:34 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Continuing with deployment * 13:34 urbanecm@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 13:32 urbanecm@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * {{safesubst:SAL entry|1=13:30 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Backport for [[gerrit:1298758{{!}}feat(V2): toggle experiment features based on custom url override (T424646)]], [[gerrit:1298762{{!}}specialCreateAccount: use GECreateAccountExperimentV2 instead of hook (T424646)]], [[gerrit:1298764{{!}}fix: correctly read experiments param on Special:UserLogin]], [[gerrit:1298765{{!}}signup.js: use JS var instead of TestKitchen to show}} * {{safesubst:SAL entry|1=13:29 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298758{{!}}feat(V2): toggle experiment features based on custom url override (T424646)]], [[gerrit:1298762{{!}}specialCreateAccount: use GECreateAccountExperimentV2 instead of hook (T424646)]], [[gerrit:1298764{{!}}fix: correctly read experiments param on Special:UserLogin]], [[gerrit:1298765{{!}}signup.js: use JS var instead of TestKitchen to show expe}} * 13:21 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298418{{!}}NewUserMessage: Add $wgNewUserMessageOnAutoCreateFirstEdit (T426206)]], [[gerrit:1298717{{!}}Replace NewUserMessageOnAutoCreateFirstEdit with wgNewUserMessageOnFirstEdit (T426206)]], [[gerrit:1298734{{!}}Enable wgNewUserMessageOnFirstEdit on incubatorwiki (T426206)]] (duration: 11m 06s) * 13:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1041.eqiad.wmnet with reason: host reimage * 13:17 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Continuing with deployment * 13:12 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 13:12 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Backport for [[gerrit:1298418{{!}}NewUserMessage: Add $wgNewUserMessageOnAutoCreateFirstEdit (T426206)]], [[gerrit:1298717{{!}}Replace NewUserMessageOnAutoCreateFirstEdit with wgNewUserMessageOnFirstEdit (T426206)]], [[gerrit:1298734{{!}}Enable wgNewUserMessageOnFirstEdit on incubatorwiki (T426206)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki * 13:12 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 13:12 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1041.eqiad.wmnet with reason: host reimage * 13:11 kamila@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 13:11 kamila@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 13:10 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1298418{{!}}NewUserMessage: Add $wgNewUserMessageOnAutoCreateFirstEdit (T426206)]], [[gerrit:1298717{{!}}Replace NewUserMessageOnAutoCreateFirstEdit with wgNewUserMessageOnFirstEdit (T426206)]], [[gerrit:1298734{{!}}Enable wgNewUserMessageOnFirstEdit on incubatorwiki (T426206)]] * 12:57 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298767{{!}}Follow-up: Allow CaptchaConsequence to be skipped via hook (T427608)]] (duration: 06m 20s) * 12:57 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1041.eqiad.wmnet with OS trixie * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:56 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1041: Upgrading es1041.eqiad.wmnet * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:55 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1041: Upgrading es1041.eqiad.wmnet * 12:55 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:54 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:53 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 12:53 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1298767{{!}}Follow-up: Allow CaptchaConsequence to be skipped via hook (T427608)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:51 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:51 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1298767{{!}}Follow-up: Allow CaptchaConsequence to be skipped via hook (T427608)]] * 12:49 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:49 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2041: repool after upgrade * 12:49 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:47 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:46 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 12:44 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 12:43 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 12:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 12:41 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 12:40 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2063.codfw.wmnet with OS bullseye * 12:32 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2062.codfw.wmnet with OS bullseye * 12:27 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2041.codfw.wmnet with OS trixie * 12:21 joal@deploy1003: Finished deploy [analytics/refinery@d67c584] (thin): Regular analytics weekly train THIN [analytics/refinery@d67c584f] (duration: 02m 00s) * 12:19 joal@deploy1003: Started deploy [analytics/refinery@d67c584] (thin): Regular analytics weekly train THIN [analytics/refinery@d67c584f] * 12:19 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2063.codfw.wmnet with reason: host reimage * 12:18 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 12:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 12:16 joal@deploy1003: Finished deploy [analytics/refinery@d67c584]: Regular analytics weekly train [analytics/refinery@d67c584f] (duration: 07m 52s) * 12:15 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2063.codfw.wmnet with reason: host reimage * 12:13 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2062.codfw.wmnet with reason: host reimage * 12:09 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2041.codfw.wmnet with reason: host reimage * 12:08 joal@deploy1003: Started deploy [analytics/refinery@d67c584]: Regular analytics weekly train [analytics/refinery@d67c584f] * 12:08 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2062.codfw.wmnet with reason: host reimage * 12:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:06 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add eqiad e8 public vlans - ayounsi@cumin1003" * 12:06 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add eqiad e8 public vlans - ayounsi@cumin1003" * 12:03 joal@deploy1003: Finished deploy [analytics/refinery@d67c584] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@d67c584f] (duration: 02m 00s) * 12:03 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2041.codfw.wmnet with reason: host reimage * 12:01 joal@deploy1003: Started deploy [analytics/refinery@d67c584] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@d67c584f] * 12:01 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 12:00 ayounsi@cumin1003: END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) * 12:00 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 12:00 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 12:00 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:57 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2063 * 11:57 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2063 * 11:57 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2063 * 11:57 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2063.codfw.wmnet 52.16.192.10.in-addr.arpa 2.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:56 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2063.codfw.wmnet 52.16.192.10.in-addr.arpa 2.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:56 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:56 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2063 - mvernon@cumin2002" * 11:56 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2063 - mvernon@cumin2002" * 11:51 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:51 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2063 * 11:50 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2063.codfw.wmnet with OS bullseye * 11:50 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2062 * 11:50 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2062 * 11:49 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2062 * 11:49 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2062.codfw.wmnet 123.0.192.10.in-addr.arpa 3.2.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:49 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2062.codfw.wmnet 123.0.192.10.in-addr.arpa 3.2.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:49 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:49 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2062 - mvernon@cumin2002" * 11:49 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2062 - mvernon@cumin2002" * 11:47 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2041.codfw.wmnet with OS trixie * 11:45 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2041: Upgrading es2041.codfw.wmnet * 11:45 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2041: Upgrading es2041.codfw.wmnet * 11:44 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:44 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.major-upgrade (exit_code=97) * 11:44 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:44 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1042: repool after maintenance * 11:43 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:43 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2062 * 11:42 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2062.codfw.wmnet with OS bullseye * 11:30 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298728{{!}}SpecialMediaSearch: Prefer thumb steps over thumb limits (T424032)]] (duration: 17m 39s) * 11:25 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 11:18 Raine: progressively switching shellbox to bookworm (start) * 11:15 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 11:14 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 11:14 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1298728{{!}}SpecialMediaSearch: Prefer thumb steps over thumb limits (T424032)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:13 kamila@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 11:12 kamila@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 11:12 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1298728{{!}}SpecialMediaSearch: Prefer thumb steps over thumb limits (T424032)]] * 11:02 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be2062 * 11:02 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be2063 * 10:58 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1042: repool after maintenance * 10:58 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:56 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1042.eqiad.wmnet with OS trixie * 10:47 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298721{{!}}GuessedThumbnailInfo: Also allow showing webp originals (T428202)]] (duration: 16m 41s) * 10:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1042.eqiad.wmnet with reason: host reimage * 10:39 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 10:39 kamila@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 10:38 kamila@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 10:36 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2160.codfw.wmnet * 10:36 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2160.codfw.wmnet * 10:35 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2043: repool after upgrade * 10:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2160.codfw.wmnet with reason: Reboot * 10:34 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1298721{{!}}GuessedThumbnailInfo: Also allow showing webp originals (T428202)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:34 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1042.eqiad.wmnet with reason: host reimage * 10:30 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1298721{{!}}GuessedThumbnailInfo: Also allow showing webp originals (T428202)]] * 10:18 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1042.eqiad.wmnet with OS trixie * 10:18 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:18 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:18 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:18 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:16 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:15 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1042: Upgrading es1042.eqiad.wmnet * 10:14 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:14 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:14 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:14 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:13 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1042: Upgrading es1042.eqiad.wmnet * 10:13 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:12 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be2063 * 10:09 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be2062 * 10:07 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:07 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:07 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:06 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 09:52 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply * 09:52 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply * 09:50 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply * 09:49 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply * 09:49 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2043: repool after upgrade * 09:49 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2043.codfw.wmnet with OS trixie * 09:44 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 09:44 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 09:42 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: sync * 09:42 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: sync * 09:41 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: sync * 09:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2043.codfw.wmnet with reason: host reimage * 09:27 jelto@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org * 09:23 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2043.codfw.wmnet with reason: host reimage * 09:17 jelto@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org * 09:15 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: sync * 09:15 ozge@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: sync * 09:07 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2043.codfw.wmnet with OS trixie * 09:06 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2043: Upgrading es2043.codfw.wmnet * 09:06 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2043: Upgrading es2043.codfw.wmnet * 09:05 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:41 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1217.eqiad.wmnet with OS trixie * 08:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1217.eqiad.wmnet with reason: host reimage * 08:15 taavi@cumin1003: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database urwikisource ([[phab:T415977|T415977]]) * 08:14 taavi@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database urwikisource ([[phab:T415977|T415977]]) * 08:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1217.eqiad.wmnet with reason: host reimage * 08:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2052: repool after upgrade * 08:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1051: repool after maintenance * 08:03 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Managing sanitization for wikis urwikisource in section s5 * 07:55 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1217.eqiad.wmnet with OS trixie * 07:53 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1217.eqiad.wmnet with reason: reimage * 07:53 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis urwikisource in section s5 * 07:52 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Checking sanitization for wikis urwikisource in section s5 * 07:50 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis urwikisource in section s5 * 07:50 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.sanitize-wiki (exit_code=97) Managing sanitization for wikis urwikisource in section s5 * 07:50 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis urwikisource in section s5 * 07:44 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297681{{!}}Global rollout - Sub-ref deployments to Group 0, Group 1 and frwiki (T425662)]] (duration: 32m 51s) * 07:32 wmde-fisch@deploy1003: wmde-fisch, lilients: Continuing with deployment * 07:29 wmde-fisch@deploy1003: wmde-fisch, lilients: Backport for [[gerrit:1297681{{!}}Global rollout - Sub-ref deployments to Group 0, Group 1 and frwiki (T425662)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:21 elukey: upgrade sudo package on an-* hosts for [[phab:T428384|T428384]] * 07:18 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2052: repool after upgrade * 07:18 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1051: repool after maintenance * 07:17 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:17 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:12 taavi@cumin1003: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database urwikisource ([[phab:T415977|T415977]]) * 07:12 elukey: upgrade exim4 packages on seaborgium for security upgrades * 07:11 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1297681{{!}}Global rollout - Sub-ref deployments to Group 0, Group 1 and frwiki (T425662)]] * 06:36 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1051.eqiad.wmnet with OS trixie * 06:20 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1051.eqiad.wmnet with reason: host reimage * 06:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1051.eqiad.wmnet with reason: host reimage * 06:15 taavi@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database urwikisource ([[phab:T415977|T415977]]) * 05:58 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1051.eqiad.wmnet with OS trixie * 05:54 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2052.codfw.wmnet with OS trixie * 05:44 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool es1051: Upgrading es1051.eqiad.wmnet * 05:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2052.codfw.wmnet with reason: host reimage * 05:35 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2052.codfw.wmnet with reason: host reimage * 05:35 marostegui@dns1004: END - running authdns-update * 05:34 marostegui@dns1004: START - running authdns-update * 05:33 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1051: Upgrading es1051.eqiad.wmnet * 05:33 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:31 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1054 to es3 eqiad primary [[phab:T428050|T428050]]', diff saved to https://phabricator.wikimedia.org/P93895 and previous config saved to /var/cache/conftool/dbconfig/20260608-053156-marostegui.json * 05:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2052.codfw.wmnet with OS trixie * 05:18 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2052: Upgrading es2052.codfw.wmnet * 05:18 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2052: Upgrading es2052.codfw.wmnet * 05:18 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade == 2026-06-07 == * 16:32 elukey: `elukey@cumin1003:~$ sudo cumin 'cp6* and not cp6014* and not cp6010*' "varnish-frontend-restart" -b 1` * 16:29 elukey: restart varnish-frontend on cp6014 == 2026-06-06 == * 09:07 ammarpad@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=hewiki --logwiki=metawiki W.Mechelke Tungsten_Mechelke # [[phab:T428182|T428182]] == 2026-06-05 == * 22:16 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 22:15 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 21:01 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=10 --verbose` (after stopping the other commons scan) * 20:56 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=30 --verbose` (after stopping the other commons scan) * 20:20 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290093{{!}}Enable wmgUseUrlShortenerLegacy on test2wiki (T107188)]] (duration: 10m 02s) * 20:16 krinkle@deploy1003: krinkle: Continuing with deployment * 20:12 krinkle@deploy1003: krinkle: Backport for [[gerrit:1290093{{!}}Enable wmgUseUrlShortenerLegacy on test2wiki (T107188)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:10 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1290093{{!}}Enable wmgUseUrlShortenerLegacy on test2wiki (T107188)]] * 16:45 jgreen@dns1004: END - running authdns-update * 16:44 jgreen@dns1004: START - running authdns-update * 16:17 dzahn@dns1005: END - running authdns-update * 16:17 mutante: DNS - adding new project language "mag" - Magahi - a language spoken in India and Nepal by about 12 million native speakers ([[phab:T428266|T428266]]) * 16:16 dzahn@dns1005: START - running authdns-update * 14:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:38 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:37 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 12:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:30 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:30 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 12:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2202.codfw.wmnet with reason: Reboot * 12:28 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:28 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 12:08 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:07 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 12:07 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 12:06 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 11:29 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 11:28 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:55 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:54 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:31 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1054: repool after upgrade * 08:08 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 08:07 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 08:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 08:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/kafka-ui: apply * 07:39 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1054: repool after upgrade * 07:38 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 07:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 07:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/kafka-ui: apply * 07:07 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 06:01 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1054.eqiad.wmnet with OS trixie * 05:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1054.eqiad.wmnet with reason: host reimage * 05:37 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1054.eqiad.wmnet with reason: host reimage * 05:22 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1054.eqiad.wmnet with OS trixie * 05:21 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1054: Upgrading es1054.eqiad.wmnet * 05:21 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1054: Upgrading es1054.eqiad.wmnet * 05:20 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 01:55 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1010.eqiad.wmnet with OS trixie * 01:39 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: host reimage * 01:32 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: host reimage * 01:16 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1010.eqiad.wmnet with OS trixie * 00:56 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1007.eqiad.wmnet with OS trixie * 00:40 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1007.eqiad.wmnet with reason: host reimage * 00:33 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1007.eqiad.wmnet with reason: host reimage * 00:17 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1007.eqiad.wmnet with OS trixie * 00:02 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297268{{!}}Redirect unknown wikinews languages to portal (T427126)]] (duration: 07m 02s) == 2026-06-04 == * 23:57 ladsgroup@deploy1003: ladsgroup, pppery: Continuing with deployment * 23:57 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1006.eqiad.wmnet with OS trixie * 23:57 ladsgroup@deploy1003: ladsgroup, pppery: Backport for [[gerrit:1297268{{!}}Redirect unknown wikinews languages to portal (T427126)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:55 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1297268{{!}}Redirect unknown wikinews languages to portal (T427126)]] * 23:40 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1006.eqiad.wmnet with reason: host reimage * 23:36 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1006.eqiad.wmnet with reason: host reimage * 23:20 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main1006.eqiad.wmnet with OS trixie * 21:28 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host releases1003.eqiad.wmnet with OS trixie * 21:04 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on releases1003.eqiad.wmnet with reason: host reimage * 20:58 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on releases1003.eqiad.wmnet with reason: host reimage * 20:50 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5030.* * 20:42 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host releases1003.eqiad.wmnet with OS trixie * 20:27 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1100.eqiad.wmnet,service=(cdn{{!}}ats-be) * 20:26 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6013.drmrs.wmnet,service=(cdn{{!}}ats-be) * 20:20 brett@dns1006: END - running authdns-update * 20:19 brett@dns1006: START - running authdns-update * 20:18 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5030.eqsin.wmnet with OS trixie * 20:10 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296015{{!}}Deploy PRV to 6 wikis (T427851)]] (duration: 07m 39s) * 20:08 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist group2.dblist extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` * 20:06 arlolra@deploy1003: arlolra: Continuing with deployment * 20:04 arlolra@deploy1003: arlolra: Backport for [[gerrit:1296015{{!}}Deploy PRV to 6 wikis (T427851)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:02 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1296015{{!}}Deploy PRV to 6 wikis (T427851)]] * 19:49 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5030.eqsin.wmnet with reason: host reimage * 19:43 cmooney@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5030.eqsin.wmnet with reason: host reimage * 19:15 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5030 * 19:15 cmooney@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp5030 * 19:14 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cp5030 * 19:14 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5030.eqsin.wmnet 27.0.132.10.in-addr.arpa 7.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:14 cmooney@cumin1003: START - Cookbook sre.dns.wipe-cache cp5030.eqsin.wmnet 27.0.132.10.in-addr.arpa 7.2.0.0.0.0.0.0.2.3.1.0.0.1.0.0.1.0.1.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors * 19:14 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:14 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5030 - cmooney@cumin1003" * 19:13 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cp5030 - cmooney@cumin1003" * 19:09 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 19:08 cmooney@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5030 * 19:08 cmooney@cumin1003: START - Cookbook sre.hosts.reimage for host cp5030.eqsin.wmnet with OS trixie * 18:51 cmooney@dns2005: END - running authdns-update * 18:50 cmooney@dns2005: START - running authdns-update * 18:43 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:42 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove IPs that had been used for eqsin cr links - cmooney@cumin1003" * 18:40 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove IPs that had been used for eqsin cr links - cmooney@cumin1003" * 18:37 sukhe: sukhe@cp6013:~$ sudo traffic_server -C clear_cache * 18:36 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 18:08 dancy@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.5 refs [[phab:T423914|T423914]] * 17:17 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297751{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297752{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] (duration: 06m 40s) * 17:13 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 17:13 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297751{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297752{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:11 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297751{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297752{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] * 16:55 topranks: shift traffic off cr1-esams et-1/0/1 link to asw1-by27-esams [[phab:T427056|T427056]] * 16:45 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297741{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297742{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] (duration: 13m 58s) * 16:41 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 16:33 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297741{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297742{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:31 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297741{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]], [[gerrit:1297742{{!}}hCaptcha: Update MF interface name for instrumentation (T428178)]] * 16:17 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 16:03 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297740{{!}}hCaptcha: Move ConfirmEditCaptchaClass hook inside hCaptcha block (T428183)]] (duration: 10m 21s) * 16:03 elukey: uploaded spicerack_12.7.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia * 15:59 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 15:55 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297740{{!}}hCaptcha: Move ConfirmEditCaptchaClass hook inside hCaptcha block (T428183)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:53 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297740{{!}}hCaptcha: Move ConfirmEditCaptchaClass hook inside hCaptcha block (T428183)]] * 15:44 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5030.* * 15:41 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2007.codfw.wmnet with OS trixie * 15:39 ladsgroup@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0) * 15:28 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views * 15:24 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297730{{!}}ptwiki: Disable Article Guidance experiment (T426871)]] (duration: 07m 26s) * 15:24 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2007.codfw.wmnet with reason: host reimage * 15:20 sbisson@deploy1003: sbisson: Continuing with deployment * 15:19 sbisson@deploy1003: sbisson: Backport for [[gerrit:1297730{{!}}ptwiki: Disable Article Guidance experiment (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:19 jayme@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2007.codfw.wmnet with reason: host reimage * 15:17 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1297730{{!}}ptwiki: Disable Article Guidance experiment (T426871)]] * 15:13 ladsgroup@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0) * 15:06 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297724{{!}}Revert "Start reading from new file tables on commons"]] (duration: 07m 00s) * 15:05 ladsgroup@cumin1003: START - Cookbook sre.wikireplicas.update-views * 15:02 zabe@deploy1003: zabe: Continuing with deployment * 15:01 zabe@deploy1003: zabe: Backport for [[gerrit:1297724{{!}}Revert "Start reading from new file tables on commons"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:59 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1297724{{!}}Revert "Start reading from new file tables on commons"]] * 14:57 zabe@deploy1003: Finished scap sync-world: [[phab:T416548|T416548]] (duration: 05m 10s) * 14:56 jayme@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-main2007.codfw.wmnet with OS trixie * 14:52 zabe@deploy1003: Started scap sync-world: [[phab:T416548|T416548]] * 14:50 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:49 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:43 zabe@deploy1003: sync-world aborted: Backport for [[gerrit:1270513{{!}}Start reading from new file tables on commons (T416548)]] (duration: 03m 58s) * 14:43 zabe@deploy1003: zabe: Continuing with deployment * 14:41 zabe@deploy1003: zabe: Backport for [[gerrit:1270513{{!}}Start reading from new file tables on commons (T416548)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:40 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f1-codfw * 14:40 ayounsi@cumin1003: START - Cookbook sre.network.tls for network device lsw1-f1-codfw * 14:39 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1270513{{!}}Start reading from new file tables on commons (T416548)]] * 14:36 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297711{{!}}hCaptcha: Enable for MobileFrontend in some Group 2 wikis (T425940)]] (duration: 08m 20s) * 14:32 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:30 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297711{{!}}hCaptcha: Enable for MobileFrontend in some Group 2 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:29 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1057: repool after upgrade * 14:28 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297711{{!}}hCaptcha: Enable for MobileFrontend in some Group 2 wikis (T425940)]] * 14:20 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 14:16 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply * 14:16 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:15 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply * 14:15 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:15 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply * 14:13 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297704{{!}}Use the globalblock-local-status right over globalblock-whitelist (T277942)]], [[gerrit:1296620{{!}}core-Permissions: Stop assigning unused globalblock-whitelist right (T277942)]] (duration: 06m 46s) * 14:10 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 14:08 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:08 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297704{{!}}Use the globalblock-local-status right over globalblock-whitelist (T277942)]], [[gerrit:1296620{{!}}core-Permissions: Stop assigning unused globalblock-whitelist right (T277942)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:07 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:06 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply * 14:06 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297704{{!}}Use the globalblock-local-status right over globalblock-whitelist (T277942)]], [[gerrit:1296620{{!}}core-Permissions: Stop assigning unused globalblock-whitelist right (T277942)]] * 14:06 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:06 tappof: bump space for prometheus k8s-aux in eqiad * 14:05 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply * 14:05 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply * 14:04 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply * 13:56 _joe_: transferred requestctl api tokens for all ops to the db ([[phab:T428119|T428119]]) * 13:56 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2050 to es3 codfw primary [[phab:T428050|T428050]]', diff saved to https://phabricator.wikimedia.org/P93878 and previous config saved to /var/cache/conftool/dbconfig/20260604-135631-marostegui.json * 13:56 Dreamy_Jazz: Afternoon UTC backport window done * 13:54 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297700{{!}}Revert "hCaptcha: Provide always challenge sitekey for account creation"]] (duration: 13m 38s) * 13:51 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 13:50 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 13:47 sukhe: sukhe@cp6011:~$ sudo -i varnish-frontend-restart * 13:44 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1057: repool after upgrade * 13:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:43 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297700{{!}}Revert "hCaptcha: Provide always challenge sitekey for account creation"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:41 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1057.eqiad.wmnet with OS trixie * 13:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297700{{!}}Revert "hCaptcha: Provide always challenge sitekey for account creation"]] * 13:38 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297692{{!}}hCaptcha: Provide always challenge sitekey for account creation (T421041)]] (duration: 05m 27s) * 13:38 dreamyjazz@deploy1003: dreamyjazz: Rolling back deployment * 13:36 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: down * 13:35 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297692{{!}}hCaptcha: Provide always challenge sitekey for account creation (T421041)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:33 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297692{{!}}hCaptcha: Provide always challenge sitekey for account creation (T421041)]] * 13:31 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295978{{!}}Update config for WikiProjects linking prototype (T427804)]] (duration: 17m 13s) * 13:26 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Continuing with deployment * 13:25 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1057.eqiad.wmnet with reason: host reimage * 13:17 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1057.eqiad.wmnet with reason: host reimage * 13:16 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, audreypenven: Backport for [[gerrit:1295978{{!}}Update config for WikiProjects linking prototype (T427804)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:14 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1295978{{!}}Update config for WikiProjects linking prototype (T427804)]] * 13:13 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:13 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1220: Migration of db1220.eqiad.wmnet completed * 13:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: down * 13:12 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1224', diff saved to https://phabricator.wikimedia.org/P93875 and previous config saved to /var/cache/conftool/dbconfig/20260604-131219-marostegui.json * 13:00 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1057.eqiad.wmnet with OS trixie * 13:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1057: Upgrading es1057.eqiad.wmnet * 12:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1057: Upgrading es1057.eqiad.wmnet * 12:59 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:56 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296557{{!}}wmf-config: Skip CAPTCHA for action=mcrundo (T427612)]] (duration: 08m 30s) * 12:52 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Continuing with deployment * 12:50 dreamyjazz@deploy1003: mpostoronca, dreamyjazz: Backport for [[gerrit:1296557{{!}}wmf-config: Skip CAPTCHA for action=mcrundo (T427612)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2050: repool after upgrade * 12:48 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296557{{!}}wmf-config: Skip CAPTCHA for action=mcrundo (T427612)]] * 12:37 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 12:37 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 12:28 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1220: Migration of db1220.eqiad.wmnet completed * 12:20 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1220.eqiad.wmnet with OS trixie * 12:04 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2050: repool after upgrade * 12:04 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 12:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1220.eqiad.wmnet with reason: host reimage * 11:59 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1220.eqiad.wmnet with reason: host reimage * 11:42 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1220.eqiad.wmnet with OS trixie * 11:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2050.codfw.wmnet with OS trixie * 11:40 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1220: Upgrading db1220.eqiad.wmnet * 11:37 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1220: Upgrading db1220.eqiad.wmnet * 11:36 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1179: Migration of db1179.eqiad.wmnet completed * 11:23 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2050.codfw.wmnet with reason: host reimage * 11:16 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2050.codfw.wmnet with reason: host reimage * 11:00 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2050.codfw.wmnet with OS trixie * 11:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2050: Upgrading es2050.codfw.wmnet * 10:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2050: Upgrading es2050.codfw.wmnet * 10:59 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:59 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2057: repool after upgrade * 10:58 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:55 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 10:46 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1179: Migration of db1179.eqiad.wmnet completed * 10:38 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1179.eqiad.wmnet with OS trixie * 10:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1179.eqiad.wmnet with reason: host reimage * 10:16 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply * 10:15 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply * 10:15 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply * 10:15 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/kartotherian: apply * 10:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1179.eqiad.wmnet with reason: host reimage * 10:13 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2057: repool after upgrade * 10:13 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2057.codfw.wmnet with OS trixie * 09:59 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1179.eqiad.wmnet with OS trixie * 09:58 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1179: Upgrading db1179.eqiad.wmnet * 09:58 jynus: redoing m2 backups after grant change [[phab:T411111|T411111]] * 09:57 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1179: Upgrading db1179.eqiad.wmnet * 09:56 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:54 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2057.codfw.wmnet with reason: host reimage * 09:53 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 09:49 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2057.codfw.wmnet with reason: host reimage * 09:39 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:39 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Migration of db1224.eqiad.wmnet completed * 09:38 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 09:37 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/kafka-ui: apply * 09:36 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/kafka-ui: apply * 09:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/kafka-ui: apply * 09:33 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2057.codfw.wmnet with OS trixie * 09:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2057: Upgrading es2057.codfw.wmnet * 09:32 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2057: Upgrading es2057.codfw.wmnet * 09:31 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:26 Dreamy_Jazz: Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=30 --sleep=60 --verbose` * 09:25 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist "group0.dblist + group1.dblist - mediamoderation-continuous-scan.dblist" extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` * 08:54 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Introduce pluggable authentication - oblivian@cumin1003" * 08:54 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Introduce pluggable authentication - oblivian@cumin1003 * 08:53 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Migration of db1224.eqiad.wmnet completed * 08:53 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Introduce pluggable authentication - oblivian@cumin1003 * 08:53 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Introduce pluggable authentication - oblivian@cumin1003" * 08:29 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 08:29 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 08:24 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 08:24 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 08:21 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 08:21 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1224.eqiad.wmnet with OS trixie * 08:21 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 08:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1224.eqiad.wmnet with reason: host reimage * 08:02 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2249.codfw.wmnet with reason: upgrade * 08:00 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1224.eqiad.wmnet with reason: host reimage * 07:53 marostegui: Install mariadb 10.11.17 on db2249 [[phab:T427345|T427345]] * 07:43 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1224.eqiad.wmnet with OS trixie * 07:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1224: Upgrading db1224.eqiad.wmnet * 07:41 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1224: Upgrading db1224.eqiad.wmnet * 07:41 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:39 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:39 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1255: Migration of db1255.eqiad.wmnet completed * 07:34 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297536{{!}}hCaptcha risk scores: VE plugin to collect risk scores for block notices (T426943)]], [[gerrit:1297200{{!}}hCaptcha: Render a fresh mobile widget for each captcha attempt (T425929)]], [[gerrit:1297173{{!}}hCaptcha: Enable risk-score collection for users blocked by IP blocks (T424629)]] (duration: 08m 56s) * 07:29 kharlan@deploy1003: kharlan, harroyo-wmf: Continuing with deployment * 07:27 kharlan@deploy1003: kharlan, harroyo-wmf: Backport for [[gerrit:1297536{{!}}hCaptcha risk scores: VE plugin to collect risk scores for block notices (T426943)]], [[gerrit:1297200{{!}}hCaptcha: Render a fresh mobile widget for each captcha attempt (T425929)]], [[gerrit:1297173{{!}}hCaptcha: Enable risk-score collection for users blocked by IP blocks (T424629)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwd * 07:25 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297536{{!}}hCaptcha risk scores: VE plugin to collect risk scores for block notices (T426943)]], [[gerrit:1297200{{!}}hCaptcha: Render a fresh mobile widget for each captcha attempt (T425929)]], [[gerrit:1297173{{!}}hCaptcha: Enable risk-score collection for users blocked by IP blocks (T424629)]] * 07:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2191: Migration of db2191.codfw.wmnet completed * 07:12 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297550{{!}}Revert "EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion"]] (duration: 06m 45s) * 07:08 kharlan@deploy1003: kharlan: Continuing with deployment * 07:08 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297550{{!}}Revert "EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:06 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297550{{!}}Revert "EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion"]] * 07:04 otto@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297260{{!}}EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] (duration: 399m 30s) * 07:03 otto@deploy1003: otto: Rolling back deployment * 06:53 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1255: Migration of db1255.eqiad.wmnet completed * 06:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1255.eqiad.wmnet with OS trixie * 06:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2191: Migration of db2191.codfw.wmnet completed * 06:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1255.eqiad.wmnet with reason: host reimage * 06:32 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2191.codfw.wmnet with OS trixie * 06:31 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1255.eqiad.wmnet with reason: host reimage * 06:16 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1255.eqiad.wmnet with OS trixie * 06:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2191.codfw.wmnet with reason: host reimage * 06:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1255: Upgrading db1255.eqiad.wmnet * 06:12 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1255: Upgrading db1255.eqiad.wmnet * 06:12 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:11 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2191.codfw.wmnet with reason: host reimage * 06:04 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db1255 [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93836 and previous config saved to /var/cache/conftool/dbconfig/20260604-060428-cwilliams.json * 06:03 cwilliams@dns1004: END - running authdns-update * 06:02 cwilliams@dns1004: START - running authdns-update * 05:54 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db1258 to x3 primary and set section read-write [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93835 and previous config saved to /var/cache/conftool/dbconfig/20260604-055429-cwilliams.json * 05:53 cwilliams@cumin1003: dbctl commit (dc=all): 'Set x3 eqiad as read-only for maintenance - [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93834 and previous config saved to /var/cache/conftool/dbconfig/20260604-055346-cwilliams.json * 05:53 cezmunsta: Starting x3 eqiad failover from db1255 to db1258 - [[phab:T427895|T427895]] * 05:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2191.codfw.wmnet with OS trixie * 05:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2191: Upgrading db2191.codfw.wmnet * 05:50 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2191: Upgrading db2191.codfw.wmnet * 05:50 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db1258 with weight 0 [[phab:T427895|T427895]]', diff saved to https://phabricator.wikimedia.org/P93833 and previous config saved to /var/cache/conftool/dbconfig/20260604-055021-cwilliams.json * 05:50 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:50 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Primary switchover x3 [[phab:T427895|T427895]] * 05:48 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 05:46 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2191 [[phab:T428120|T428120]]', diff saved to https://phabricator.wikimedia.org/P93832 and previous config saved to /var/cache/conftool/dbconfig/20260604-054614-marostegui.json * 05:45 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db2215 to x1 primary [[phab:T428120|T428120]]', diff saved to https://phabricator.wikimedia.org/P93831 and previous config saved to /var/cache/conftool/dbconfig/20260604-054528-marostegui.json * 05:44 marostegui: Starting x1 codfw failover from db2191 to db2215 - [[phab:T428120|T428120]] * 05:27 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: Primary switchover x1 [[phab:T428120|T428120]] * 05:27 marostegui@cumin1003: dbctl commit (dc=all): 'Set db2215 with weight 0 [[phab:T428120|T428120]]', diff saved to https://phabricator.wikimedia.org/P93830 and previous config saved to /var/cache/conftool/dbconfig/20260604-052722-marostegui.json * 05:19 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 03:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93829 and previous config saved to /var/cache/conftool/dbconfig/20260604-034546-fceratto.json * 03:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P93828 and previous config saved to /var/cache/conftool/dbconfig/20260604-033538-fceratto.json * 03:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P93827 and previous config saved to /var/cache/conftool/dbconfig/20260604-032531-fceratto.json * 03:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93826 and previous config saved to /var/cache/conftool/dbconfig/20260604-031523-fceratto.json * 03:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1263 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93825 and previous config saved to /var/cache/conftool/dbconfig/20260604-030710-fceratto.json * 03:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1263.eqiad.wmnet with reason: Maintenance * 03:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93824 and previous config saved to /var/cache/conftool/dbconfig/20260604-030642-fceratto.json * 02:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P93823 and previous config saved to /var/cache/conftool/dbconfig/20260604-025634-fceratto.json * 02:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P93822 and previous config saved to /var/cache/conftool/dbconfig/20260604-024627-fceratto.json * 02:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93821 and previous config saved to /var/cache/conftool/dbconfig/20260604-023619-fceratto.json * 02:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1262 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93820 and previous config saved to /var/cache/conftool/dbconfig/20260604-022809-fceratto.json * 02:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Maintenance * 02:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93819 and previous config saved to /var/cache/conftool/dbconfig/20260604-022742-fceratto.json * 02:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P93818 and previous config saved to /var/cache/conftool/dbconfig/20260604-021734-fceratto.json * 02:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P93817 and previous config saved to /var/cache/conftool/dbconfig/20260604-020726-fceratto.json * 01:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93816 and previous config saved to /var/cache/conftool/dbconfig/20260604-015718-fceratto.json * 01:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1261 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93815 and previous config saved to /var/cache/conftool/dbconfig/20260604-014909-fceratto.json * 01:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1261.eqiad.wmnet with reason: Maintenance * 01:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93814 and previous config saved to /var/cache/conftool/dbconfig/20260604-014841-fceratto.json * 01:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P93813 and previous config saved to /var/cache/conftool/dbconfig/20260604-013833-fceratto.json * 01:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P93812 and previous config saved to /var/cache/conftool/dbconfig/20260604-012826-fceratto.json * 01:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93811 and previous config saved to /var/cache/conftool/dbconfig/20260604-011818-fceratto.json * 01:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1260 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93810 and previous config saved to /var/cache/conftool/dbconfig/20260604-011005-fceratto.json * 01:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1260.eqiad.wmnet with reason: Maintenance * 01:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93809 and previous config saved to /var/cache/conftool/dbconfig/20260604-010937-fceratto.json * 00:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P93808 and previous config saved to /var/cache/conftool/dbconfig/20260604-005929-fceratto.json * 00:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P93807 and previous config saved to /var/cache/conftool/dbconfig/20260604-004922-fceratto.json * 00:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93806 and previous config saved to /var/cache/conftool/dbconfig/20260604-003914-fceratto.json * 00:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1252 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93805 and previous config saved to /var/cache/conftool/dbconfig/20260604-002851-fceratto.json * 00:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1252.eqiad.wmnet with reason: Maintenance * 00:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93804 and previous config saved to /var/cache/conftool/dbconfig/20260604-002821-fceratto.json * 00:26 otto@deploy1003: otto: Backport for [[gerrit:1297260{{!}}EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 00:24 otto@deploy1003: Started scap sync-world: Backport for [[gerrit:1297260{{!}}EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] * 00:18 Amir1: mwscript-k8s --follow --dblist=all -- extensions/timeline/maintenance/DeleteOldTimelineFiles.php --date {{Gerrit|20210101000000}} * 00:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P93803 and previous config saved to /var/cache/conftool/dbconfig/20260604-001813-fceratto.json * 00:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P93802 and previous config saved to /var/cache/conftool/dbconfig/20260604-000805-fceratto.json == 2026-06-03 == * 23:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93801 and previous config saved to /var/cache/conftool/dbconfig/20260603-235758-fceratto.json * 23:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93800 and previous config saved to /var/cache/conftool/dbconfig/20260603-234935-fceratto.json * 23:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1248.eqiad.wmnet with reason: Maintenance * 23:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93799 and previous config saved to /var/cache/conftool/dbconfig/20260603-234907-fceratto.json * 23:42 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296561{{!}}Add a maintenance script to delete old files]], [[gerrit:1296560{{!}}Add a maintenance script to delete old files]] (duration: 07m 09s) * 23:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P93798 and previous config saved to /var/cache/conftool/dbconfig/20260603-233859-fceratto.json * 23:37 ladsgroup@deploy1003: ladsgroup, reedy: Continuing with deployment * 23:36 ladsgroup@deploy1003: ladsgroup, reedy: Backport for [[gerrit:1296561{{!}}Add a maintenance script to delete old files]], [[gerrit:1296560{{!}}Add a maintenance script to delete old files]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:34 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1296561{{!}}Add a maintenance script to delete old files]], [[gerrit:1296560{{!}}Add a maintenance script to delete old files]] * 23:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P93797 and previous config saved to /var/cache/conftool/dbconfig/20260603-232852-fceratto.json * 23:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 23:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 23:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93796 and previous config saved to /var/cache/conftool/dbconfig/20260603-231844-fceratto.json * 23:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93795 and previous config saved to /var/cache/conftool/dbconfig/20260603-231031-fceratto.json * 23:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1247.eqiad.wmnet with reason: Maintenance * 23:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93794 and previous config saved to /var/cache/conftool/dbconfig/20260603-231001-fceratto.json * 22:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P93793 and previous config saved to /var/cache/conftool/dbconfig/20260603-225953-fceratto.json * 22:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P93792 and previous config saved to /var/cache/conftool/dbconfig/20260603-224945-fceratto.json * 22:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93791 and previous config saved to /var/cache/conftool/dbconfig/20260603-223937-fceratto.json * 22:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1244 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93790 and previous config saved to /var/cache/conftool/dbconfig/20260603-223116-fceratto.json * 22:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1244.eqiad.wmnet with reason: Maintenance * 22:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93789 and previous config saved to /var/cache/conftool/dbconfig/20260603-223048-fceratto.json * 22:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P93788 and previous config saved to /var/cache/conftool/dbconfig/20260603-222041-fceratto.json * 22:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P93787 and previous config saved to /var/cache/conftool/dbconfig/20260603-221034-fceratto.json * 22:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93786 and previous config saved to /var/cache/conftool/dbconfig/20260603-220026-fceratto.json * 21:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1243 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93785 and previous config saved to /var/cache/conftool/dbconfig/20260603-215110-fceratto.json * 21:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1243.eqiad.wmnet with reason: Maintenance * 21:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93784 and previous config saved to /var/cache/conftool/dbconfig/20260603-215053-fceratto.json * 21:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P93783 and previous config saved to /var/cache/conftool/dbconfig/20260603-214046-fceratto.json * 21:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P93782 and previous config saved to /var/cache/conftool/dbconfig/20260603-213038-fceratto.json * 21:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93781 and previous config saved to /var/cache/conftool/dbconfig/20260603-212030-fceratto.json * 21:12 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1242 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93779 and previous config saved to /var/cache/conftool/dbconfig/20260603-211206-fceratto.json * 21:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1242.eqiad.wmnet with reason: Maintenance * 21:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93778 and previous config saved to /var/cache/conftool/dbconfig/20260603-211138-fceratto.json * 21:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P93774 and previous config saved to /var/cache/conftool/dbconfig/20260603-210130-fceratto.json * 20:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P93773 and previous config saved to /var/cache/conftool/dbconfig/20260603-205122-fceratto.json * 20:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93772 and previous config saved to /var/cache/conftool/dbconfig/20260603-204115-fceratto.json * 20:33 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297228{{!}}Attribution research don't use testKitchen compatibility layer (T417050)]] (duration: 06m 41s) * 20:32 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1241 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93771 and previous config saved to /var/cache/conftool/dbconfig/20260603-203254-fceratto.json * 20:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1241.eqiad.wmnet with reason: Maintenance * 20:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93770 and previous config saved to /var/cache/conftool/dbconfig/20260603-203227-fceratto.json * 20:29 cjming@deploy1003: cjming: Continuing with deployment * 20:29 cjming@deploy1003: cjming: Backport for [[gerrit:1297228{{!}}Attribution research don't use testKitchen compatibility layer (T417050)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:26 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1297228{{!}}Attribution research don't use testKitchen compatibility layer (T417050)]] * 20:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P93769 and previous config saved to /var/cache/conftool/dbconfig/20260603-202219-fceratto.json * 20:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P93766 and previous config saved to /var/cache/conftool/dbconfig/20260603-201211-fceratto.json * 20:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93765 and previous config saved to /var/cache/conftool/dbconfig/20260603-200203-fceratto.json * 19:59 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 19:59 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 19:59 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 19:59 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 19:53 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93764 and previous config saved to /var/cache/conftool/dbconfig/20260603-195341-fceratto.json * 19:53 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1238.eqiad.wmnet with reason: Maintenance * 19:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93763 and previous config saved to /var/cache/conftool/dbconfig/20260603-195313-fceratto.json * 19:47 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5032.* * 19:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P93762 and previous config saved to /var/cache/conftool/dbconfig/20260603-194306-fceratto.json * 19:39 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5032.* * 19:37 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5032.* * 19:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P93761 and previous config saved to /var/cache/conftool/dbconfig/20260603-193258-fceratto.json * 19:26 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 19:25 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 19:25 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 19:25 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 19:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93760 and previous config saved to /var/cache/conftool/dbconfig/20260603-192250-fceratto.json * 19:22 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 19:22 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 19:14 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93759 and previous config saved to /var/cache/conftool/dbconfig/20260603-191437-fceratto.json * 19:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1024-1025].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 19:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1221.eqiad.wmnet with reason: Maintenance * 19:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93758 and previous config saved to /var/cache/conftool/dbconfig/20260603-191348-fceratto.json * 19:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P93757 and previous config saved to /var/cache/conftool/dbconfig/20260603-190340-fceratto.json * 18:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P93756 and previous config saved to /var/cache/conftool/dbconfig/20260603-185331-fceratto.json * 18:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93755 and previous config saved to /var/cache/conftool/dbconfig/20260603-184324-fceratto.json * 18:34 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1199 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93754 and previous config saved to /var/cache/conftool/dbconfig/20260603-183455-fceratto.json * 18:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance * 18:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93753 and previous config saved to /var/cache/conftool/dbconfig/20260603-183427-fceratto.json * 18:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P93752 and previous config saved to /var/cache/conftool/dbconfig/20260603-182420-fceratto.json * 18:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P93751 and previous config saved to /var/cache/conftool/dbconfig/20260603-181412-fceratto.json * 18:10 dancy@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.5 refs [[phab:T423914|T423914]] * 18:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93750 and previous config saved to /var/cache/conftool/dbconfig/20260603-180404-fceratto.json * 17:57 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5032.* * 17:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93749 and previous config saved to /var/cache/conftool/dbconfig/20260603-175544-fceratto.json * 17:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance * 17:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93748 and previous config saved to /var/cache/conftool/dbconfig/20260603-175342-fceratto.json * 17:52 hashar: contint1003: sudo puppet agent --disable "Prevent Jenkins from coming back" * 17:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P93747 and previous config saved to /var/cache/conftool/dbconfig/20260603-174334-fceratto.json * 17:38 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:37 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 17:37 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:36 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:36 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:35 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:35 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:35 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:34 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:34 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:33 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P93746 and previous config saved to /var/cache/conftool/dbconfig/20260603-173327-fceratto.json * 17:33 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:32 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:29 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5032.* * 17:26 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host sretest2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 17:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93745 and previous config saved to /var/cache/conftool/dbconfig/20260603-172319-fceratto.json * 17:18 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: Stopping before sync operations * 17:17 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: Started scap sync-world: No-deploy scap run to verify scap config change * 17:17 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:15 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:15 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1253 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93744 and previous config saved to /var/cache/conftool/dbconfig/20260603-171521-fceratto.json * 17:15 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:15 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance * 17:14 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93743 and previous config saved to /var/cache/conftool/dbconfig/20260603-171452-fceratto.json * 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:13 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:12 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:10 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:09 ayounsi@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2012.wikimedia.org with OS trixie * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P93742 and previous config saved to /var/cache/conftool/dbconfig/20260603-170444-fceratto.json * 17:04 swfrench@deploy1003: Stopping before sync operations * 17:03 swfrench@deploy1003: Started scap sync-world: No-deploy scap run to verify clean state before config change * 16:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P93741 and previous config saved to /var/cache/conftool/dbconfig/20260603-165436-fceratto.json * 16:53 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:53 hashar: Restarting CI Jenkins one last time # [[phab:T418521|T418521]] * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:44 btullis@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295922{{!}}Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] (duration: 07m 16s) * 16:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93740 and previous config saved to /var/cache/conftool/dbconfig/20260603-164428-fceratto.json * 16:43 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:43 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:42 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:41 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:40 btullis@deploy1003: btullis: Continuing with deployment * 16:39 btullis@deploy1003: btullis: Backport for [[gerrit:1295922{{!}}Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93739 and previous config saved to /var/cache/conftool/dbconfig/20260603-163726-fceratto.json * 16:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance * 16:37 btullis@deploy1003: Started scap sync-world: Backport for [[gerrit:1295922{{!}}Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] * 16:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93738 and previous config saved to /var/cache/conftool/dbconfig/20260603-163658-fceratto.json * 16:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P93737 and previous config saved to /var/cache/conftool/dbconfig/20260603-162650-fceratto.json * 16:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:19 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P93736 and previous config saved to /var/cache/conftool/dbconfig/20260603-161643-fceratto.json * 16:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93735 and previous config saved to /var/cache/conftool/dbconfig/20260603-160635-fceratto.json * 16:04 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:59 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93734 and previous config saved to /var/cache/conftool/dbconfig/20260603-155928-fceratto.json * 15:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance * 15:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93733 and previous config saved to /var/cache/conftool/dbconfig/20260603-155859-fceratto.json * 15:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 15:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P93732 and previous config saved to /var/cache/conftool/dbconfig/20260603-154852-fceratto.json * 15:46 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:46 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2012.wikimedia.org with OS trixie * 15:40 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:40 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 15:40 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 15:40 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 15:39 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 15:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P93731 and previous config saved to /var/cache/conftool/dbconfig/20260603-153844-fceratto.json * 15:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93729 and previous config saved to /var/cache/conftool/dbconfig/20260603-152836-fceratto.json * 15:25 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host sretest2012 * 15:25 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2012 * 15:25 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host sretest2012 * 15:25 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2012 * 15:24 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:23 mutante: disabling jenkins on CI servers for maintenance * 15:23 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host sretest2012 * 15:23 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2012 * 15:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 15:21 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1202 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93728 and previous config saved to /var/cache/conftool/dbconfig/20260603-152129-fceratto.json * 15:21 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance * 15:21 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:21 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding sretest2012 to codfw - jhancock@cumin2002" * 15:21 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 15:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93727 and previous config saved to /var/cache/conftool/dbconfig/20260603-152102-fceratto.json * 15:20 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding sretest2012 to codfw - jhancock@cumin2002" * 15:18 brouberol@dns1004: END - running authdns-update * 15:18 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:16 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 15:16 brouberol@dns1004: START - running authdns-update * 15:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P93726 and previous config saved to /var/cache/conftool/dbconfig/20260603-151055-fceratto.json * 15:01 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P93725 and previous config saved to /var/cache/conftool/dbconfig/20260603-150047-fceratto.json * 14:57 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 14:52 cmooney@cumin1003: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox * 14:51 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93723 and previous config saved to /var/cache/conftool/dbconfig/20260603-145039-fceratto.json * 14:48 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297137{{!}}Revert "MultimediaViewer: enable image carousel as a beta feature on Wikipedias"]] (duration: 06m 46s) * 14:47 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 14:46 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:46 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:43 mlitn@deploy1003: mlitn: Continuing with deployment * 14:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93722 and previous config saved to /var/cache/conftool/dbconfig/20260603-144334-fceratto.json * 14:43 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:43 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance * 14:43 mlitn@deploy1003: mlitn: Backport for [[gerrit:1297137{{!}}Revert "MultimediaViewer: enable image carousel as a beta feature on Wikipedias"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93721 and previous config saved to /var/cache/conftool/dbconfig/20260603-144306-fceratto.json * 14:41 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:41 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:41 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1297137{{!}}Revert "MultimediaViewer: enable image carousel as a beta feature on Wikipedias"]] * 14:39 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:39 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:39 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:39 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:38 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:35 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 14:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 14:34 sgimeno@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297130{{!}}editor: make redesigned anon warning the default experience (T424595)]] (duration: 10m 45s) * 14:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P93719 and previous config saved to /var/cache/conftool/dbconfig/20260603-143259-fceratto.json * 14:30 vriley@cumin1003: START - Cookbook sre.hosts.provision for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:28 sgimeno@deploy1003: sgimeno: Continuing with deployment * 14:25 sgimeno@deploy1003: sgimeno: Backport for [[gerrit:1297130{{!}}editor: make redesigned anon warning the default experience (T424595)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:24 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:24 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:23 sgimeno@deploy1003: Started scap sync-world: Backport for [[gerrit:1297130{{!}}editor: make redesigned anon warning the default experience (T424595)]] * 14:23 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P93717 and previous config saved to /var/cache/conftool/dbconfig/20260603-142251-fceratto.json * 14:22 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:22 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:21 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:21 cmooney@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:21 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:20 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:20 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:20 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:20 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:19 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:19 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:16 vriley@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:16 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:16 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:13 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:12 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93716 and previous config saved to /var/cache/conftool/dbconfig/20260603-141242-fceratto.json * 14:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:11 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:11 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:10 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mc2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:10 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host mc2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:10 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:09 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:08 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host mc2055 * 14:07 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2055 * 14:05 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296631{{!}}translate: adding separate read/write endpoints (T425377)]] (duration: 13m 06s) * 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1191 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93715 and previous config saved to /var/cache/conftool/dbconfig/20260603-140537-fceratto.json * 14:05 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance * 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93714 and previous config saved to /var/cache/conftool/dbconfig/20260603-140507-fceratto.json * 14:01 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:58 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:58 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:58 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 13:56 dcausse@deploy1003: atsuko, dcausse: Rolling back deployment * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 13:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T426633|T426633]])', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20260603-133440-fceratto.json * 13:29 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:29 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2186: Migration of db2186.codfw.wmnet completed * 13:28 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295910{{!}}hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]] (duration: 07m 36s) * 13:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1174 ([[phab:T426633|T426633]])', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20260603-132638-fceratto.json * 13:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance * 13:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93710 and previous config saved to /var/cache/conftool/dbconfig/20260603-132605-fceratto.json * 13:25 sukhe: sudo cumin 'A:lvs or A:liberica' 'disable-puppet "merging CR 1282764"' * 13:23 kharlan@deploy1003: kharlan: Continuing with deployment * 13:22 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295910{{!}}hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:20 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295910{{!}}hCaptcha: Roll out self-hosted secure-api.js to all wikis (T403829)]] * 13:18 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296649{{!}}hCaptcha: Roll out to all except enwiki for mobile apps. (T426048)]] (duration: 07m 46s) * 13:16 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20260603-131556-fceratto.json * 13:15 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:13 kharlan@deploy1003: dbrant, kharlan: Continuing with deployment * 13:12 kharlan@deploy1003: dbrant, kharlan: Backport for [[gerrit:1296649{{!}}hCaptcha: Roll out to all except enwiki for mobile apps. (T426048)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296649{{!}}hCaptcha: Roll out to all except enwiki for mobile apps. (T426048)]] * 13:09 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:09 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add codfw d3 and e5 public vlans - ayounsi@cumin1003" * 13:09 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add codfw d3 and e5 public vlans - ayounsi@cumin1003" * 13:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P93708 and previous config saved to /var/cache/conftool/dbconfig/20260603-130548-fceratto.json * 13:05 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 12:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93706 and previous config saved to /var/cache/conftool/dbconfig/20260603-125540-fceratto.json * 12:51 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297110{{!}}ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] (duration: 07m 44s) * 12:49 jgreen@dns1004: END - running authdns-update * 12:47 jgreen@dns1004: START - running authdns-update * 12:46 jiji@deploy1003: jiji: Continuing with deployment * 12:46 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93705 and previous config saved to /var/cache/conftool/dbconfig/20260603-124624-fceratto.json * 12:46 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance * 12:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93704 and previous config saved to /var/cache/conftool/dbconfig/20260603-124556-fceratto.json * 12:45 jiji@deploy1003: jiji: Backport for [[gerrit:1297110{{!}}ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:43 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2186: Migration of db2186.codfw.wmnet completed * 12:43 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1297110{{!}}ProductionServices.php: switch filebackend.php to rdb2013:6381 (T418261 T419976)]] * 12:41 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1067.eqiad.wmnet with OS bullseye * 12:38 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1292364{{!}}Update hCaptcha checks to retrieve API parameters from $_REQUEST (T427105)]] (duration: 11m 15s) * 12:36 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2186.codfw.wmnet with OS trixie * 12:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P93702 and previous config saved to /var/cache/conftool/dbconfig/20260603-123548-fceratto.json * 12:34 dreamyjazz@deploy1003: somerandomdeveloper, dreamyjazz: Continuing with deployment * 12:31 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1066.eqiad.wmnet with OS bullseye * 12:29 dreamyjazz@deploy1003: somerandomdeveloper, dreamyjazz: Backport for [[gerrit:1292364{{!}}Update hCaptcha checks to retrieve API parameters from $_REQUEST (T427105)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:27 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1292364{{!}}Update hCaptcha checks to retrieve API parameters from $_REQUEST (T427105)]] * 12:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P93701 and previous config saved to /var/cache/conftool/dbconfig/20260603-122541-fceratto.json * 12:22 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1067.eqiad.wmnet with reason: host reimage * 12:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2186.codfw.wmnet with reason: host reimage * 12:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93700 and previous config saved to /var/cache/conftool/dbconfig/20260603-121533-fceratto.json * 12:13 mvernon@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ms-be1066.eqiad.wmnet with reason: host reimage * 12:13 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2186.codfw.wmnet with reason: host reimage * 12:11 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1067.eqiad.wmnet with reason: host reimage * 12:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93699 and previous config saved to /var/cache/conftool/dbconfig/20260603-120732-fceratto.json * 12:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 12:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance * 12:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93698 and previous config saved to /var/cache/conftool/dbconfig/20260603-120634-fceratto.json * 12:03 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1066.eqiad.wmnet with reason: host reimage * 11:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P93697 and previous config saved to /var/cache/conftool/dbconfig/20260603-115626-fceratto.json * 11:54 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2186.codfw.wmnet with OS trixie * 11:54 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be1067 * 11:54 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1067 * 11:52 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1067 * 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be1067.eqiad.wmnet 96.48.64.10.in-addr.arpa 6.9.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:52 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be1067.eqiad.wmnet 96.48.64.10.in-addr.arpa 6.9.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1067 - mvernon@cumin2002" * 11:52 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1067 - mvernon@cumin2002" * 11:48 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2186: Upgrading db2186.codfw.wmnet * 11:48 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2186: Upgrading db2186.codfw.wmnet * 11:48 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:47 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:46 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be1067 * 11:46 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be1067.eqiad.wmnet with OS bullseye * 11:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P93695 and previous config saved to /var/cache/conftool/dbconfig/20260603-114618-fceratto.json * 11:46 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be1066 * 11:46 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1066 * 11:45 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1066 * 11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be1066.eqiad.wmnet 117.32.64.10.in-addr.arpa 7.1.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:45 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be1066.eqiad.wmnet 117.32.64.10.in-addr.arpa 7.1.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:45 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1066 - mvernon@cumin2002" * 11:45 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be1066 - mvernon@cumin2002" * 11:43 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply * 11:42 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply * 11:41 mvernon@cumin2002: START - Cookbook sre.dns.netbox * 11:40 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be1066 * 11:40 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be1066.eqiad.wmnet with OS bullseye * 11:39 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be1067 * 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93693 and previous config saved to /var/cache/conftool/dbconfig/20260603-113611-fceratto.json * 11:33 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:33 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:32 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2196: Migration of db2196.codfw.wmnet completed * 11:29 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93691 and previous config saved to /var/cache/conftool/dbconfig/20260603-112909-fceratto.json * 11:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance * 11:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance * 11:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93690 and previous config saved to /var/cache/conftool/dbconfig/20260603-112838-fceratto.json * 11:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:20 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:20 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:20 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P93689 and previous config saved to /var/cache/conftool/dbconfig/20260603-111831-fceratto.json * 11:14 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:09 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: apply * 11:09 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: apply * 11:08 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P93687 and previous config saved to /var/cache/conftool/dbconfig/20260603-110823-fceratto.json * 11:07 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be1066 * 11:07 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply * 11:06 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply * 11:05 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply * 11:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:01 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:01 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 11:00 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289895{{!}}Update UserInfoCard to be enabled by default for certain user groups (T426021)]] (duration: 07m 37s) * 11:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:59 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply * 10:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:59 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply * 10:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:58 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply * 10:58 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93685 and previous config saved to /var/cache/conftool/dbconfig/20260603-105815-fceratto.json * 10:58 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 10:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:56 mszwarc@deploy1003: mszwarc: Continuing with deployment * 10:55 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1289895{{!}}Update UserInfoCard to be enabled by default for certain user groups (T426021)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:54 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply * 10:54 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply * 10:53 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply * 10:53 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1289895{{!}}Update UserInfoCard to be enabled by default for certain user groups (T426021)]] * 10:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:50 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1198 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93684 and previous config saved to /var/cache/conftool/dbconfig/20260603-105006-fceratto.json * 10:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance * 10:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93683 and previous config saved to /var/cache/conftool/dbconfig/20260603-104939-fceratto.json * 10:45 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:45 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:44 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2196: Migration of db2196.codfw.wmnet completed * 10:44 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply * 10:41 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 10:40 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 10:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P93681 and previous config saved to /var/cache/conftool/dbconfig/20260603-103931-fceratto.json * 10:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1053: repool after upgrade * 10:37 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2196.codfw.wmnet with OS trixie * 10:36 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297090{{!}}hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] (duration: 12m 03s) * 10:32 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 10:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 10:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P93679 and previous config saved to /var/cache/conftool/dbconfig/20260603-102924-fceratto.json * 10:26 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1297090{{!}}hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:24 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1297090{{!}}hCaptcha: Enable for MobileFrontend on most group1 wikis (T425940)]] * 10:22 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be1067 * 10:21 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be1066 * 10:19 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2196.codfw.wmnet with reason: host reimage * 10:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93677 and previous config saved to /var/cache/conftool/dbconfig/20260603-101916-fceratto.json * 10:15 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2013.codfw.wmnet * 10:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2196.codfw.wmnet with reason: host reimage * 10:11 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93676 and previous config saved to /var/cache/conftool/dbconfig/20260603-101105-fceratto.json * 10:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1189.eqiad.wmnet with reason: Maintenance * 10:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93675 and previous config saved to /var/cache/conftool/dbconfig/20260603-101037-fceratto.json * 10:10 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2013.codfw.wmnet * 10:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P93673 and previous config saved to /var/cache/conftool/dbconfig/20260603-100029-fceratto.json * 09:59 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2196.codfw.wmnet with OS trixie * 09:57 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2196: Upgrading db2196.codfw.wmnet * 09:57 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2196: Upgrading db2196.codfw.wmnet * 09:57 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:52 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1053: repool after upgrade * 09:52 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:52 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:52 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:52 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:51 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:51 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:51 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 09:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P93670 and previous config saved to /var/cache/conftool/dbconfig/20260603-095022-fceratto.json * 09:49 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:49 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:48 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1053.eqiad.wmnet with OS trixie * 09:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply * 09:43 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2013.codfw.wmnet * 09:41 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on es1053.eqiad.wmnet with reason: host reimage * 09:41 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1053.eqiad.wmnet with reason: host reimage * 09:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93669 and previous config saved to /var/cache/conftool/dbconfig/20260603-094014-fceratto.json * 09:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2215: Migration of db2215.codfw.wmnet completed * 09:38 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2013.codfw.wmnet * 09:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93667 and previous config saved to /var/cache/conftool/dbconfig/20260603-093146-fceratto.json * 09:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance * 09:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93666 and previous config saved to /var/cache/conftool/dbconfig/20260603-093119-fceratto.json * 09:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 09:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1211: Migration of db1211.eqiad.wmnet completed * 09:27 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297069{{!}}hCaptcha: Collect risk score for blocked account creations (T427784)]] (duration: 07m 26s) * 09:25 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1053.eqiad.wmnet with OS trixie * 09:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add public1-b3-codfw gateway IPs - ayounsi@cumin1003" * 09:24 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add public1-b3-codfw gateway IPs - ayounsi@cumin1003" * 09:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1053: Upgrading es1053.eqiad.wmnet * 09:23 kharlan@deploy1003: kharlan: Continuing with deployment * 09:22 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1053: Upgrading es1053.eqiad.wmnet * 09:22 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:21 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297069{{!}}hCaptcha: Collect risk score for blocked account creations (T427784)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:21 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply * 09:21 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2054: repool after upgrade * 09:21 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply * 09:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P93661 and previous config saved to /var/cache/conftool/dbconfig/20260603-092111-fceratto.json * 09:20 ayounsi@cumin1003: START - Cookbook sre.dns.netbox * 09:20 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297069{{!}}hCaptcha: Collect risk score for blocked account creations (T427784)]] * 09:14 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297065{{!}}Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] (duration: 07m 06s) * 09:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P93659 and previous config saved to /var/cache/conftool/dbconfig/20260603-091104-fceratto.json * 09:10 kharlan@deploy1003: kharlan: Continuing with deployment * 09:09 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297065{{!}}Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:07 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297065{{!}}Revert^4 "hCaptcha: Load self-hosted secure-api.js on group0 wikis"]] * 09:06 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply * 09:06 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1297064{{!}}Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 10m 54s) * 09:05 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply * 09:04 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 09:01 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003 - [[phab:T422043|T422043]]" * 09:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93656 and previous config saved to /var/cache/conftool/dbconfig/20260603-090056-fceratto.json * 09:00 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003 - [[phab:T422043|T422043]]" * 09:00 ayounsi@cumin1003: END (ERROR) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=97) generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003" * 09:00 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "new eqiad/codfw public vlans - ayounsi@cumin1003" * 08:59 kharlan@deploy1003: kharlan: Continuing with deployment * 08:59 kharlan@deploy1003: kharlan: Backport for [[gerrit:1297064{{!}}Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:55 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1297064{{!}}Revert^3 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] * 08:53 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296635{{!}}Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 11m 43s) * 08:52 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2215: Migration of db2215.codfw.wmnet completed * 08:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet * 08:52 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet * 08:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb[1022-1023].eqiad.wmnet * 08:51 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb[1022-1023].eqiad.wmnet * 08:50 kharlan@deploy1003: kharlan: Rolling back deployment * 08:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93652 and previous config saved to /var/cache/conftool/dbconfig/20260603-084846-fceratto.json * 08:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance * 08:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93651 and previous config saved to /var/cache/conftool/dbconfig/20260603-084819-fceratto.json * 08:47 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296635{{!}}Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2215.codfw.wmnet with OS trixie * 08:45 jiji@cumin1003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check docker-registry: maintenance * 08:45 jiji@cumin1003: START - Cookbook sre.discovery.service-route check docker-registry: maintenance * 08:43 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1211: Migration of db1211.eqiad.wmnet completed * 08:41 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296635{{!}}Revert^2 "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] * 08:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1211.eqiad.wmnet with OS trixie * 08:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93649 and previous config saved to /var/cache/conftool/dbconfig/20260603-083811-fceratto.json * 08:37 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296632{{!}}Image Browsing: add accessible labels to carousel elements (T407793)]] (duration: 32m 11s) * 08:36 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2054: repool after upgrade * 08:35 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool es2054.codfw.wmnet: After reimage * 08:35 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2054.codfw.wmnet: After reimage * 08:35 jiji@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:34 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 08:34 jiji@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:33 jiji@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:33 jiji@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:31 jiji@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:31 jiji@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:31 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2054.codfw.wmnet with OS trixie * 08:30 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:29 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 08:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2215.codfw.wmnet with reason: host reimage * 08:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93647 and previous config saved to /var/cache/conftool/dbconfig/20260603-082804-fceratto.json * 08:25 mszwarc@deploy1003: mlitn, mszwarc: Continuing with deployment * 08:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1211.eqiad.wmnet with reason: host reimage * 08:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1049: repool after upgrade * 08:22 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2215.codfw.wmnet with reason: host reimage * 08:22 mszwarc@deploy1003: mlitn, mszwarc: Backport for [[gerrit:1296632{{!}}Image Browsing: add accessible labels to carousel elements (T407793)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:18 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1211.eqiad.wmnet with reason: host reimage * 08:18 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 08:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93645 and previous config saved to /var/cache/conftool/dbconfig/20260603-081756-fceratto.json * 08:17 jiji@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 08:17 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 08:16 jiji@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 08:14 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2054.codfw.wmnet with reason: host reimage * 08:08 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2054.codfw.wmnet with reason: host reimage * 08:05 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296632{{!}}Image Browsing: add accessible labels to carousel elements (T407793)]] * {{safesubst:SAL entry|1=08:04 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296580{{!}}Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703{{!}}jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713{{!}}conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627{{!}}Add missing lazy img to carousel (T427821)]], [[gerrit:1295968{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T426799)]}} * 08:03 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93643 and previous config saved to /var/cache/conftool/dbconfig/20260603-080346-fceratto.json * 08:03 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1211.eqiad.wmnet with OS trixie * 08:03 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance * 08:03 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2215.codfw.wmnet with OS trixie * 08:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1211: Upgrading db1211.eqiad.wmnet * 08:02 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2215: Upgrading db2215.codfw.wmnet * 08:01 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:01 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1211: Upgrading db1211.eqiad.wmnet * 08:01 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2215: Upgrading db2215.codfw.wmnet * 08:01 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:01 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:01 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1157: Repooling * 08:01 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1157: Repooling * 08:00 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 07:57 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1022-1023].eqiad.wmnet with reason: Reimaging upstream server * 07:57 mszwarc@deploy1003: anzx, mlitn, mfossati, mszwarc: Continuing with deployment * 07:56 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Reimaging upstream server * {{safesubst:SAL entry|1=07:54 mszwarc@deploy1003: anzx, mlitn, mfossati, mszwarc: Backport for [[gerrit:1296580{{!}}Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703{{!}}jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713{{!}}conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627{{!}}Add missing lazy img to carousel (T427821)]], [[gerrit:1295968{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T42}} * 07:52 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2231: repool after maintenance * 07:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2054.codfw.wmnet with OS trixie * 07:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2054: Upgrading es2054.codfw.wmnet * 07:50 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2054: Upgrading es2054.codfw.wmnet * 07:50 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:50 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296580{{!}}Add kha to wmgExtraLanguageNames (T427917)]], [[gerrit:1296703{{!}}jawiki: lift IP caps for workshop (T427912)]], [[gerrit:1296713{{!}}conductwiki: add sitename and logo (T426984 T427541)]], [[gerrit:1296627{{!}}Add missing lazy img to carousel (T427821)]], [[gerrit:1295968{{!}}MultimediaViewer: enable image carousel as a beta feature on Wikipedias (T426799)]] * 07:48 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296516{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] (duration: 32m 13s) * 07:44 marostegui@dns1004: END - running authdns-update * 07:43 marostegui@dns1004: START - running authdns-update * 07:42 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1056 to es2 eqiad primary [[phab:T427875|T427875]]', diff saved to https://phabricator.wikimedia.org/P93637 and previous config saved to /var/cache/conftool/dbconfig/20260603-074250-marostegui.json * 07:37 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1049: repool after upgrade * 07:37 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:35 mszwarc@deploy1003: mszwarc, stran: Continuing with deployment * 07:35 mszwarc@deploy1003: mszwarc, stran: Backport for [[gerrit:1296516{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:32 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1049.eqiad.wmnet with OS trixie * 07:16 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1296516{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]], [[gerrit:1296517{{!}}Add a reply-to to Direct Reporting emails (T427788 T427791 T427829)]] * 07:14 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1049.eqiad.wmnet with reason: host reimage * 07:07 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1049.eqiad.wmnet with reason: host reimage * 07:07 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2231: repool after maintenance * 07:04 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 06:57 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2231.codfw.wmnet with OS trixie * 06:52 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1049.eqiad.wmnet with OS trixie * 06:46 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1049: Upgrading es1049.eqiad.wmnet * 06:46 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2056 to es2 codfw primary [[phab:T427875|T427875]]', diff saved to https://phabricator.wikimedia.org/P93632 and previous config saved to /var/cache/conftool/dbconfig/20260603-064623-marostegui.json * 06:45 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1049: Upgrading es1049.eqiad.wmnet * 06:45 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:44 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1056: repool after upgrade * 06:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2231.codfw.wmnet with reason: host reimage * 06:36 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2231.codfw.wmnet with reason: host reimage * 06:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2231.codfw.wmnet with OS trixie * 06:09 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2231: Upgrading db2231.codfw.wmnet * 06:09 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2231: Upgrading db2231.codfw.wmnet * 06:09 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:59 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1056: repool after upgrade * 05:59 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 05:55 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1056.eqiad.wmnet with OS trixie * 05:39 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1056.eqiad.wmnet with reason: host reimage * 05:33 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1056.eqiad.wmnet with reason: host reimage * 05:18 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1056.eqiad.wmnet with OS trixie * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1056: Upgrading es1056.eqiad.wmnet * 05:17 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1056: Upgrading es1056.eqiad.wmnet * 05:16 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade == 2026-06-02 == * 22:21 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296689{{!}}hCaptcha: Correct inaccurate comment]] (duration: 06m 27s) * 22:18 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 22:18 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 22:17 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 22:17 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296689{{!}}hCaptcha: Correct inaccurate comment]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:15 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296689{{!}}hCaptcha: Correct inaccurate comment]] * 22:13 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296551{{!}}hCaptcha: Enable for badlogin on group0 wikis (T426875)]] (duration: 08m 31s) * 22:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 22:10 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 22:09 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 22:07 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296551{{!}}hCaptcha: Enable for badlogin on group0 wikis (T426875)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:05 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296551{{!}}hCaptcha: Enable for badlogin on group0 wikis (T426875)]] * 20:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93621 and previous config saved to /var/cache/conftool/dbconfig/20260602-203945-fceratto.json * 20:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93620 and previous config saved to /var/cache/conftool/dbconfig/20260602-202937-fceratto.json * 20:27 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1054.eqiad.wmnet * 20:27 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:27 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1054.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:26 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1054.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:20 jiji@cumin1003: START - Cookbook sre.dns.netbox * 20:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P93619 and previous config saved to /var/cache/conftool/dbconfig/20260602-201929-fceratto.json * 20:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93618 and previous config saved to /var/cache/conftool/dbconfig/20260602-200922-fceratto.json * 20:03 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1054.eqiad.wmnet * 19:48 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1053.eqiad.wmnet * 19:48 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:48 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1053.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 19:37 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1053.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 19:09 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93617 and previous config saved to /var/cache/conftool/dbconfig/20260602-190907-fceratto.json * 19:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance * 19:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93616 and previous config saved to /var/cache/conftool/dbconfig/20260602-190811-fceratto.json * 19:05 dancy@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.5 refs [[phab:T423914|T423914]] * 18:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P93615 and previous config saved to /var/cache/conftool/dbconfig/20260602-185804-fceratto.json * 18:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P93614 and previous config saved to /var/cache/conftool/dbconfig/20260602-184757-fceratto.json * 18:38 jiji@cumin1003: START - Cookbook sre.dns.netbox * 18:38 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:38 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93612 and previous config saved to /var/cache/conftool/dbconfig/20260602-183749-fceratto.json * 18:37 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:37 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:33 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1053.eqiad.wmnet * 18:30 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1259 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93611 and previous config saved to /var/cache/conftool/dbconfig/20260602-183023-fceratto.json * 18:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance * 18:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93610 and previous config saved to /var/cache/conftool/dbconfig/20260602-182956-fceratto.json * 18:27 mutante: gerrit delete unused plugin projects: barricade, WikimediaBlocks and WikimediaWebSessions * 18:26 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1052.eqiad.wmnet * 18:26 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:26 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1052.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 18:25 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1052.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 18:25 dancy: Train is blocked at testwikis on https://phabricator.wikimedia.org/T427935 * 18:21 Daimona: Running query from [[phab:T427962|T427962]]#11978299 in x1.wikishared * 18:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P93609 and previous config saved to /var/cache/conftool/dbconfig/20260602-181949-fceratto.json * 18:16 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296615{{!}}feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614{{!}}feat(cleanMentorList): Add a feature flag (T427386)]] (duration: 34m 09s) * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 18:13 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 18:13 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 18:12 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 18:12 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 18:12 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 18:10 jiji@cumin1003: START - Cookbook sre.dns.netbox * 18:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P93608 and previous config saved to /var/cache/conftool/dbconfig/20260602-180941-fceratto.json * 18:08 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 18:07 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 18:06 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 18:06 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 18:05 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:05 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:05 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 18:05 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 18:04 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 18:02 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 18:02 swfrench-wmf: reverting shellbox to 2026-05-20-192555 due to errors in shellbox-syntaxhighlight * 18:02 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 18:01 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 18:01 urbanecm@deploy1003: urbanecm: Continuing with deployment * 18:01 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1296615{{!}}feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614{{!}}feat(cleanMentorList): Add a feature flag (T427386)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:00 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1052.eqiad.wmnet * 17:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93607 and previous config saved to /var/cache/conftool/dbconfig/20260602-175933-fceratto.json * 17:58 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:57 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:56 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1051.eqiad.wmnet * 17:56 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:56 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1051.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:55 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1051.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:53 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93605 and previous config saved to /var/cache/conftool/dbconfig/20260602-175227-fceratto.json * 17:52 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance * 17:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93604 and previous config saved to /var/cache/conftool/dbconfig/20260602-175157-fceratto.json * 17:51 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:51 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:50 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:50 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:50 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:49 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:49 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:48 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:48 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:47 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:44 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:43 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:43 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:42 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:42 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:42 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P93603 and previous config saved to /var/cache/conftool/dbconfig/20260602-174150-fceratto.json * 17:41 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1296615{{!}}feat(cleanMentorList): Add a feature flag (T427386)]], [[gerrit:1296614{{!}}feat(cleanMentorList): Add a feature flag (T427386)]] * 17:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P93602 and previous config saved to /var/cache/conftool/dbconfig/20260602-173143-fceratto.json * 17:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93601 and previous config saved to /var/cache/conftool/dbconfig/20260602-172135-fceratto.json * 17:14 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93600 and previous config saved to /var/cache/conftool/dbconfig/20260602-171422-fceratto.json * 17:14 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance * 17:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93599 and previous config saved to /var/cache/conftool/dbconfig/20260602-171354-fceratto.json * 17:04 jiji@cumin1003: START - Cookbook sre.dns.netbox * 17:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P93598 and previous config saved to /var/cache/conftool/dbconfig/20260602-170344-fceratto.json * 16:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P93597 and previous config saved to /var/cache/conftool/dbconfig/20260602-165336-fceratto.json * 16:49 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1051.eqiad.wmnet * 16:48 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1050.eqiad.wmnet * 16:48 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:48 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1050.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:47 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1050.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93596 and previous config saved to /var/cache/conftool/dbconfig/20260602-164328-fceratto.json * 16:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93595 and previous config saved to /var/cache/conftool/dbconfig/20260602-163622-fceratto.json * 16:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance * 16:36 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:35 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93594 and previous config saved to /var/cache/conftool/dbconfig/20260602-163550-fceratto.json * 16:34 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:34 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1072.eqiad.wmnet with OS trixie * 16:30 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:29 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 16:27 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2006.codfw.wmnet with OS trixie * 16:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P93593 and previous config saved to /var/cache/conftool/dbconfig/20260602-162542-fceratto.json * 16:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P93591 and previous config saved to /var/cache/conftool/dbconfig/20260602-161534-fceratto.json * 16:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1072.eqiad.wmnet with reason: host reimage * 16:10 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1071.eqiad.wmnet with OS trixie * 16:10 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296624{{!}}Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] (duration: 06m 40s) * 16:09 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2006.codfw.wmnet with reason: host reimage * 16:05 kharlan@deploy1003: kharlan: Continuing with deployment * 16:05 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1072.eqiad.wmnet with reason: host reimage * 16:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93590 and previous config saved to /var/cache/conftool/dbconfig/20260602-160527-fceratto.json * 16:05 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main2006.codfw.wmnet with reason: host reimage * 16:05 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296624{{!}}Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:03 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296624{{!}}Revert "hCaptcha: Load self-hosted secure-api.js on group0 wikis" (T403829)]] * 15:59 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295909{{!}}hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] (duration: 09m 48s) * 15:59 kharlan@deploy1003: kharlan: Rolling back deployment * 15:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93589 and previous config saved to /var/cache/conftool/dbconfig/20260602-155817-fceratto.json * 15:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance * 15:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93588 and previous config saved to /var/cache/conftool/dbconfig/20260602-155749-fceratto.json * 15:54 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1071.eqiad.wmnet with reason: host reimage * 15:53 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1072.eqiad.wmnet with OS trixie * 15:51 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1070.eqiad.wmnet with OS trixie * 15:51 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295909{{!}}hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:50 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1071.eqiad.wmnet with reason: host reimage * 15:49 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295909{{!}}hCaptcha: Load self-hosted secure-api.js on group0 wikis (T403829)]] * 15:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P93587 and previous config saved to /var/cache/conftool/dbconfig/20260602-154742-fceratto.json * 15:47 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296558{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] (duration: 07m 24s) * 15:43 kharlan@deploy1003: kharlan: Continuing with deployment * 15:42 kharlan@deploy1003: kharlan: Backport for [[gerrit:1296558{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:40 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1296558{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]], [[gerrit:1296568{{!}}hCaptcha: Remove apiUrl health check and APCu layer from health checker (T421464)]] * 15:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P93586 and previous config saved to /var/cache/conftool/dbconfig/20260602-153734-fceratto.json * 15:37 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1071.eqiad.wmnet with OS trixie * 15:36 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1069.eqiad.wmnet with OS trixie * 15:35 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1070.eqiad.wmnet with reason: host reimage * 15:32 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:32 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:31 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1070.eqiad.wmnet with reason: host reimage * 15:30 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:29 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93585 and previous config saved to /var/cache/conftool/dbconfig/20260602-152726-fceratto.json * 15:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2158: Repooling * {{safesubst:SAL entry|1=15:22 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295502{{!}}Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106{{!}}Remove unused 'writeapi' right]], [[gerrit:1296566{{!}}Clean up bot password configuration]], [[gerrit:1296563{{!}}Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574{{!}}cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582{{!}}U}} * 15:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1069.eqiad.wmnet with reason: host reimage * 15:20 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93583 and previous config saved to /var/cache/conftool/dbconfig/20260602-152026-fceratto.json * 15:20 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance * 15:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93582 and previous config saved to /var/cache/conftool/dbconfig/20260602-151958-fceratto.json * 15:19 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:19 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 15:18 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1070.eqiad.wmnet with OS trixie * 15:18 dreamyjazz@deploy1003: matmarex, anzx, dreamyjazz: Continuing with deployment * 15:18 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 15:17 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:17 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply * 15:15 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1069.eqiad.wmnet with reason: host reimage * {{safesubst:SAL entry|1=15:15 dreamyjazz@deploy1003: matmarex, anzx, dreamyjazz: Backport for [[gerrit:1295502{{!}}Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106{{!}}Remove unused 'writeapi' right]], [[gerrit:1296566{{!}}Clean up bot password configuration]], [[gerrit:1296563{{!}}Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574{{!}}cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582}} * 15:14 jiji@cumin1003: START - Cookbook sre.dns.netbox * {{safesubst:SAL entry|1=15:13 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1295502{{!}}Revert "labswiki: Disallow account autocreation"]], [[gerrit:1283106{{!}}Remove unused 'writeapi' right]], [[gerrit:1296566{{!}}Clean up bot password configuration]], [[gerrit:1296563{{!}}Remove workaround for stuck session cookies on Wikitech (T389433)]], [[gerrit:1295574{{!}}cswiki: lift IP cap for workshop on 08-June-2026 (T427678)]], [[gerrit:1296582{{!}}Us}} * 15:12 jayme@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-main2006.codfw.wmnet with OS trixie * 15:12 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1068.eqiad.wmnet with OS trixie * 15:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P93580 and previous config saved to /var/cache/conftool/dbconfig/20260602-150951-fceratto.json * 15:09 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296514{{!}}[Growth] Set wgGEMentorshipCleanupEnabled to false on all wikis (T427386)]] (duration: 06m 22s) * 15:06 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1167: Repooling after Icing wait-for-green timeout * 15:06 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1050.eqiad.wmnet * 15:06 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1049.eqiad.wmnet * 15:06 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:06 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1049.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:05 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1049.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:02 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1296514{{!}}[Growth] Set wgGEMentorshipCleanupEnabled to false on all wikis (T427386)]] * 15:02 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1069.eqiad.wmnet with OS trixie * 15:01 jiji@cumin1003: START - Cookbook sre.dns.netbox * 14:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P93578 and previous config saved to /var/cache/conftool/dbconfig/20260602-145943-fceratto.json * 14:54 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage * 14:52 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:52 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:52 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1049.eqiad.wmnet * 14:51 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1067.eqiad.wmnet with OS trixie * 14:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:50 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage * 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93575 and previous config saved to /var/cache/conftool/dbconfig/20260602-144935-fceratto.json * 14:42 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for pc2021.codfw.wmnet * 14:42 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for pc2021.codfw.wmnet * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2250.codfw.wmnet * 14:41 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2250.codfw.wmnet * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2158.codfw.wmnet * 14:41 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2158.codfw.wmnet * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc2021: Repooling * 14:41 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 14:41 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache * 14:41 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc2021: Repooling * 14:41 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93573 and previous config saved to /var/cache/conftool/dbconfig/20260602-144110-fceratto.json * 14:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance * 14:41 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2158: Repooling * 14:40 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93571 and previous config saved to /var/cache/conftool/dbconfig/20260602-144043-fceratto.json * 14:38 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:38 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:38 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply * 14:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:37 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1048.eqiad.wmnet * 14:37 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:37 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1048.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 14:37 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1068.eqiad.wmnet with OS trixie * 14:36 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1066.eqiad.wmnet with OS trixie * 14:34 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage * 14:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P93569 and previous config saved to /var/cache/conftool/dbconfig/20260602-143035-fceratto.json * 14:30 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage * 14:25 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1048.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 14:21 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1167: Repooling after Icing wait-for-green timeout * 14:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage * 14:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P93566 and previous config saved to /var/cache/conftool/dbconfig/20260602-142027-fceratto.json * 14:17 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1067.eqiad.wmnet with OS trixie * 14:17 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 14:17 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1167.eqiad.wmnet * 14:17 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1167.eqiad.wmnet * 14:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1065.eqiad.wmnet with OS trixie * 14:15 jayme@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2006.codfw.wmnet with OS trixie * 14:14 jiji@cumin1003: START - Cookbook sre.dns.netbox * 14:13 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage * 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93564 and previous config saved to /var/cache/conftool/dbconfig/20260602-141019-fceratto.json * 14:09 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments userOptions.php --delete --nowarn growthexperiments-homepage-variant # [[phab:T417621|T417621]] * 14:09 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1048.eqiad.wmnet * 14:08 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments userOptions.php --delete growthexperiments-homepage-variant # [[phab:T417621|T417621]] * 14:05 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 14:01 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93563 and previous config saved to /var/cache/conftool/dbconfig/20260602-140140-fceratto.json * 14:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 14:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1156.eqiad.wmnet with reason: Maintenance * 14:01 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1066.eqiad.wmnet with OS trixie * 14:00 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage * 14:00 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 14:00 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 14:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93562 and previous config saved to /var/cache/conftool/dbconfig/20260602-140022-fceratto.json * 14:00 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1064.eqiad.wmnet with OS trixie * 13:56 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage * 13:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1167.eqiad.wmnet with OS trixie * 13:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:51 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P93561 and previous config saved to /var/cache/conftool/dbconfig/20260602-135015-fceratto.json * 13:47 topranks: revert all config to normal on cr1-codfw and ssw1-a1-codfw * 13:43 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1065.eqiad.wmnet with OS trixie * 13:42 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1064.eqiad.wmnet with reason: host reimage * 13:40 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1063.eqiad.wmnet with OS trixie * 13:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P93560 and previous config saved to /var/cache/conftool/dbconfig/20260602-134007-fceratto.json * 13:38 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1167.eqiad.wmnet with reason: host reimage * 13:35 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs1002.eqiad.wmnet with OS trixie * 13:35 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs1003.eqiad.wmnet with OS trixie * 13:34 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:34 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply * 13:32 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1064.eqiad.wmnet with reason: host reimage * 13:31 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1167.eqiad.wmnet with reason: host reimage * 13:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93559 and previous config saved to /var/cache/conftool/dbconfig/20260602-132959-fceratto.json * 13:27 slyngshede@dns1004: END - running authdns-update * 13:25 slyngshede@dns1004: START - running authdns-update * 13:24 topranks: increase OSPF cost on ssw1-a1-codfw et-0/0/4 towards lsw1-a5-codfw [[phab:T427301|T427301]] * 13:23 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1063.eqiad.wmnet with reason: host reimage * 13:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93558 and previous config saved to /var/cache/conftool/dbconfig/20260602-132314-fceratto.json * 13:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1210.eqiad.wmnet with reason: Maintenance * 13:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93557 and previous config saved to /var/cache/conftool/dbconfig/20260602-132246-fceratto.json * 13:20 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1064.eqiad.wmnet with OS trixie * 13:19 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 13:19 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1062.eqiad.wmnet with OS trixie * 13:18 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1063.eqiad.wmnet with reason: host reimage * 13:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2049: repool after upgrade * 13:17 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 13:16 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1167.eqiad.wmnet with OS trixie * 13:15 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 13:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1167: Upgrading db1167.eqiad.wmnet * 13:13 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1167: Upgrading db1167.eqiad.wmnet * 13:13 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:12 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 13:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P93554 and previous config saved to /var/cache/conftool/dbconfig/20260602-131238-fceratto.json * 13:12 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 13:12 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 13:11 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 13:07 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1003.eqiad.wmnet with OS trixie * 13:07 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1002.eqiad.wmnet with OS trixie * 13:06 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1063.eqiad.wmnet with OS trixie * 13:04 jayme@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-main2006.codfw.wmnet with OS trixie * 13:04 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 13:04 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:03 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1022-1023].eqiad.wmnet with reason: Reimaging upstream servers * 13:03 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1001.eqiad.wmnet with OS trixie * 13:03 topranks: increase OSPF cost on ssw1-a1-codfw et-0/0/2 towards lsw1-a3-codfw [[phab:T427301|T427301]] * 13:03 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1062.eqiad.wmnet with reason: host reimage * 13:02 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Reimaging upstream servers * 13:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P93553 and previous config saved to /var/cache/conftool/dbconfig/20260602-130230-fceratto.json * 12:59 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1062.eqiad.wmnet with reason: host reimage * 12:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:57 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:57 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply * 12:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2161: Migration of db2161.codfw.wmnet completed * 12:54 topranks: shutdown sub-interfaces on cr1-codfw et-1/1/5 for row A/B vlans [[phab:T427301|T427301]] * 12:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 12:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93550 and previous config saved to /var/cache/conftool/dbconfig/20260602-125223-fceratto.json * 12:50 topranks: enable bgp graceful-shutdown in overlay on ssw1-a1-codfw [[phab:T427301|T427301]] * 12:49 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc1061.eqiad.wmnet with OS trixie * 12:48 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt * 12:48 ayounsi@cumin1003: START - Cookbook sre.hosts.remove-downtime for lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt * 12:47 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1062.eqiad.wmnet with OS trixie * 12:45 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93548 and previous config saved to /var/cache/conftool/dbconfig/20260602-124541-fceratto.json * 12:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance * 12:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93547 and previous config saved to /var/cache/conftool/dbconfig/20260602-124512-fceratto.json * 12:43 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc1060.eqiad.wmnet with OS trixie * 12:42 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 12:42 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mc1061.eqiad.wmnet with reason: host reimage * 12:42 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1061.eqiad.wmnet with reason: host reimage * 12:41 topranks: enable bgp graceful-shutdown in underlay on ssw1-a1-codfw [[phab:T427301|T427301]] * 12:35 blake@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mc1060.eqiad.wmnet with reason: host reimage * 12:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P93545 and previous config saved to /var/cache/conftool/dbconfig/20260602-123505-fceratto.json * 12:33 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 12:33 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1060.eqiad.wmnet with reason: host reimage * 12:31 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2049: repool after upgrade * 12:31 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 12:29 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1061.eqiad.wmnet with OS trixie * 12:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2049.codfw.wmnet with OS trixie * 12:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P93542 and previous config saved to /var/cache/conftool/dbconfig/20260602-122459-fceratto.json * 12:24 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1059.eqiad.wmnet with OS trixie * 12:21 XioNoX: reboot lsw1-a3-codfw for software upgrade - [[phab:T427301|T427301]] * 12:20 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1060.eqiad.wmnet with OS trixie * 12:20 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 12:20 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1058.eqiad.wmnet with OS trixie * 12:17 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 12:16 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296532{{!}}hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533{{!}}hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] (duration: 09m 02s) * 12:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93539 and previous config saved to /var/cache/conftool/dbconfig/20260602-121451-fceratto.json * 12:11 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 12:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2049.codfw.wmnet with reason: host reimage * 12:11 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-a3-codfw,lsw1-a3-codfw IPv6,lsw1-a3-codfw.mgmt with reason: Switch maintenance * 12:10 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2161: Migration of db2161.codfw.wmnet completed * 12:09 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Switch maintenance * 12:09 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1296532{{!}}hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533{{!}}hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:08 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1200 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93537 and previous config saved to /var/cache/conftool/dbconfig/20260602-120755-fceratto.json * 12:07 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1059.eqiad.wmnet with reason: host reimage * 12:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance * 12:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93536 and previous config saved to /var/cache/conftool/dbconfig/20260602-120728-fceratto.json * 12:07 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet * 12:07 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1296532{{!}}hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533{{!}}hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] * 12:05 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2049.codfw.wmnet with reason: host reimage * 12:04 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1058.eqiad.wmnet with reason: host reimage * 12:02 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1059.eqiad.wmnet with reason: host reimage * 12:01 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2161.codfw.wmnet with OS trixie * 12:00 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1058.eqiad.wmnet with reason: host reimage * 11:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P93535 and previous config saved to /var/cache/conftool/dbconfig/20260602-115721-fceratto.json * 11:55 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 11:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:55 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 11:53 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 11:53 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 11:53 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:50 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1059.eqiad.wmnet with OS trixie * 11:49 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1057.eqiad.wmnet with OS trixie * 11:49 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2049.codfw.wmnet with OS trixie * 11:48 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2049: Upgrading es2049.codfw.wmnet * 11:48 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2049: Upgrading es2049.codfw.wmnet * 11:47 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:47 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1058.eqiad.wmnet with OS trixie * 11:47 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2056: repool after upgrade * 11:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P93532 and previous config saved to /var/cache/conftool/dbconfig/20260602-114713-fceratto.json * 11:45 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1056.eqiad.wmnet with OS trixie * 11:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2161.codfw.wmnet with reason: host reimage * 11:40 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2161.codfw.wmnet with reason: host reimage * 11:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93531 and previous config saved to /var/cache/conftool/dbconfig/20260602-113705-fceratto.json * 11:33 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1057.eqiad.wmnet with reason: host reimage * 11:30 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1185 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93529 and previous config saved to /var/cache/conftool/dbconfig/20260602-113019-fceratto.json * 11:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance * 11:29 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1056.eqiad.wmnet with reason: host reimage * 11:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1161: Repooling * 11:26 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1161: Repooling * 11:23 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2161.codfw.wmnet with OS trixie * 11:22 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1057.eqiad.wmnet with reason: host reimage * 11:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2161: Upgrading db2161.codfw.wmnet * 11:21 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2161: Upgrading db2161.codfw.wmnet * 11:21 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1056.eqiad.wmnet with reason: host reimage * 11:21 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 11:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P93527 and previous config saved to /var/cache/conftool/dbconfig/20260602-111954-fceratto.json * 11:15 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2161 [[phab:T427892|T427892]]', diff saved to https://phabricator.wikimedia.org/P93525 and previous config saved to /var/cache/conftool/dbconfig/20260602-111511-cwilliams.json * 11:12 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2165 to s8 primary [[phab:T427892|T427892]]', diff saved to https://phabricator.wikimedia.org/P93524 and previous config saved to /var/cache/conftool/dbconfig/20260602-111200-cwilliams.json * 11:10 cezmunsta: Starting s8 codfw failover from db2161 to db2165 - [[phab:T427892|T427892]] * 11:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P93523 and previous config saved to /var/cache/conftool/dbconfig/20260602-110947-fceratto.json * 11:09 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1057.eqiad.wmnet with OS trixie * 11:09 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1056.eqiad.wmnet with OS trixie * 11:04 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2165 with weight 0 [[phab:T427892|T427892]]', diff saved to https://phabricator.wikimedia.org/P93522 and previous config saved to /var/cache/conftool/dbconfig/20260602-110420-cwilliams.json * 11:03 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s8 [[phab:T427892|T427892]] * 11:02 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2056: repool after upgrade * 11:01 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93520 and previous config saved to /var/cache/conftool/dbconfig/20260602-105939-fceratto.json * 10:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93519 and previous config saved to /var/cache/conftool/dbconfig/20260602-105239-fceratto.json * 10:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 10:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance * 10:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93518 and previous config saved to /var/cache/conftool/dbconfig/20260602-105202-fceratto.json * 10:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2056.codfw.wmnet with OS trixie * 10:42 moritzm: installing busybox security updates * 10:42 claime: Enabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 10:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P93517 and previous config saved to /var/cache/conftool/dbconfig/20260602-104154-fceratto.json * 10:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P93516 and previous config saved to /var/cache/conftool/dbconfig/20260602-103146-fceratto.json * 10:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2056.codfw.wmnet with reason: host reimage * 10:27 claime: Disabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 10:25 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2056.codfw.wmnet with reason: host reimage * 10:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93515 and previous config saved to /var/cache/conftool/dbconfig/20260602-102139-fceratto.json * 10:09 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2056.codfw.wmnet with OS trixie * 10:08 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2056: Upgrading es2056.codfw.wmnet * 10:08 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2056: Upgrading es2056.codfw.wmnet * 10:08 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:06 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply * 10:06 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply * 09:56 claime: Enabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 09:46 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on cumin2003.codfw.wmnet with reason: in setup * 09:45 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1187: Pooling * 09:37 claime: Running puppet on cp6010 and cp6011 - [[phab:T422937|T422937]] * 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow2004.codfw.wmnet to plain * 09:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93511 and previous config saved to /var/cache/conftool/dbconfig/20260602-093716-fceratto.json * 09:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance * 09:35 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow2004.codfw.wmnet to plain * 09:34 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of rpki2003.codfw.wmnet to plain * 09:34 claime: Disabling puppet on A:cp-text for ATS rest-gateway cleanup - [[phab:T422937|T422937]] * 09:34 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of rpki2003.codfw.wmnet to plain * 09:32 moritzm: temporarily remove ganeti2045 from the codfw cluster [[phab:T427357|T427357]] * 09:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1055.eqiad.wmnet with OS trixie * 09:15 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1187: Pooling * 09:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1055.eqiad.wmnet with reason: host reimage * 09:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1187 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93508 and previous config saved to /var/cache/conftool/dbconfig/20260602-091126-fceratto.json * 09:09 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1055.eqiad.wmnet with reason: host reimage * 09:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1187 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93506 and previous config saved to /var/cache/conftool/dbconfig/20260602-090432-fceratto.json * 09:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1187.eqiad.wmnet with reason: Maintenance * 08:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2250.codfw.wmnet with reason: rack A3 maintenance * 08:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:56 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1055.eqiad.wmnet with OS trixie * 08:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:54 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:54 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:53 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 08:52 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 08:51 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 08:50 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 08:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 08:41 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:39 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:37 urbanecm: Reset user email of Barras@votewiki to the one of Barras@SUL * 08:30 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance * 08:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93505 and previous config saved to /var/cache/conftool/dbconfig/20260602-083033-fceratto.json * 08:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:29 slyngs: IDP, new configuration in preparation for webauthn * 08:20 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P93504 and previous config saved to /var/cache/conftool/dbconfig/20260602-082026-fceratto.json * 08:19 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 08:18 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 08:18 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:17 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296488{{!}}Revert "translate: adding separate read/write endpoints" (T425377)]] (duration: 03m 33s) * 08:16 atsuko@deploy1003: atsuko: Rolling back deployment * 08:16 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2053: repool after upgrade * 08:15 atsuko@deploy1003: atsuko: Backport for [[gerrit:1296488{{!}}Revert "translate: adding separate read/write endpoints" (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:13 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1296488{{!}}Revert "translate: adding separate read/write endpoints" (T425377)]] * 08:11 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:10 marostegui: Install mariadb 10.11.17 on es2053 [[phab:T427345|T427345]] * 08:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P93502 and previous config saved to /var/cache/conftool/dbconfig/20260602-081018-fceratto.json * 08:09 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 08:09 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: Depool for rack maintenance * 08:03 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296262{{!}}translate: fixing missed variable in credentials formatting closure (T425377)]] (duration: 14m 47s) * 08:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93499 and previous config saved to /var/cache/conftool/dbconfig/20260602-080011-fceratto.json * 07:59 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 07:59 atsuko@deploy1003: atsuko: Rolling back deployment * 07:58 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 07:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1181 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93498 and previous config saved to /var/cache/conftool/dbconfig/20260602-075759-fceratto.json * 07:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1181.eqiad.wmnet with reason: Maintenance * 07:57 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling * 07:50 atsuko@deploy1003: atsuko: Backport for [[gerrit:1296262{{!}}translate: fixing missed variable in credentials formatting closure (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:49 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1296262{{!}}translate: fixing missed variable in credentials formatting closure (T425377)]] * 07:48 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1181: Pooling * 07:47 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1181: Pooling * 07:44 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1181: Reboot * 07:43 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1181: Reboot * 07:42 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1181.eqiad.wmnet with reason: Reboot * 07:41 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 07:41 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:41 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1181: Migration of db1181.eqiad.wmnet completed * 07:40 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294949{{!}}translate: adding separate read/write endpoints (T425377)]] (duration: 21m 01s) * 07:39 atsuko@deploy1003: atsuko: Rolling back deployment * 07:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93490 and previous config saved to /var/cache/conftool/dbconfig/20260602-073904-fceratto.json * 07:32 XioNoX: pfw1-eqiad# delete protocols bgp group Production family inet6 - [[phab:T423384|T423384]] * 07:30 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2053: repool after upgrade * 07:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2158.codfw.wmnet with reason: rack A3 maintenance * 07:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93487 and previous config saved to /var/cache/conftool/dbconfig/20260602-072856-fceratto.json * 07:28 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2158: rack A3 maintenance * 07:28 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2158: rack A3 maintenance * 07:27 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on pc2021.codfw.wmnet with reason: rack A3 maintenance * 07:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc2021: rack A3 maintenance * 07:26 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 07:25 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache * 07:25 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool pc2021: rack A3 maintenance * 07:23 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2241: Depool for rack maintenance * 07:23 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2241.codfw.wmnet * 07:23 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2241.codfw.wmnet * 07:21 atsuko@deploy1003: atsuko: Backport for [[gerrit:1294949{{!}}translate: adding separate read/write endpoints (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:20 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2053.codfw.wmnet with OS trixie * 07:19 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1294949{{!}}translate: adding separate read/write endpoints (T425377)]] * 07:15 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2241.codfw.wmnet with reason: Depool for rack maintenance * 07:14 marostegui: Install mariadb 10.11.17 on db2186 [[phab:T427345|T427345]] * 07:12 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: Depool for rack maintenance * 07:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2186.codfw.wmnet with reason: upgrade * 07:12 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2241: Depool for rack maintenance * 07:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2053.codfw.wmnet with reason: host reimage * 06:59 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2053.codfw.wmnet with reason: host reimage * 06:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93478 and previous config saved to /var/cache/conftool/dbconfig/20260602-065533-fceratto.json * 06:55 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1181: Migration of db1181.eqiad.wmnet completed * 06:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 06:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1181.eqiad.wmnet with OS trixie * 06:43 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2053.codfw.wmnet with OS trixie * 06:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2053: Upgrading es2053.codfw.wmnet * 06:41 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2053: Upgrading es2053.codfw.wmnet * 06:41 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:37 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 06:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 06:36 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 06:36 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1052: repool after upgrade * 06:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1181.eqiad.wmnet with reason: host reimage * 06:24 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1181.eqiad.wmnet with reason: host reimage * 06:22 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 06:21 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 06:16 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 06:15 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 06:08 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1181.eqiad.wmnet with OS trixie * 06:05 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1181: Upgrading db1181.eqiad.wmnet * 06:05 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1181: Upgrading db1181.eqiad.wmnet * 06:04 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:02 marostegui@dns1004: END - running authdns-update * 06:01 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1181 [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93473 and previous config saved to /var/cache/conftool/dbconfig/20260602-060157-marostegui.json * 06:01 marostegui@dns1004: START - running authdns-update * 06:00 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db1236 to s7 primary and set section read-write [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93472 and previous config saved to /var/cache/conftool/dbconfig/20260602-060041-marostegui.json * 06:00 marostegui@cumin1003: dbctl commit (dc=all): 'Set s7 eqiad as read-only for maintenance - [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93471 and previous config saved to /var/cache/conftool/dbconfig/20260602-060018-marostegui.json * 06:00 marostegui: Starting s7 eqiad failover from db1181 to db1236 - [[phab:T426088|T426088]] * 05:51 marostegui@cumin1003: dbctl commit (dc=all): 'Set db1236 with weight 0 [[phab:T426088|T426088]]', diff saved to https://phabricator.wikimedia.org/P93470 and previous config saved to /var/cache/conftool/dbconfig/20260602-055153-marostegui.json * 05:51 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 [[phab:T426088|T426088]] * 05:50 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1052: repool after upgrade * 05:50 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 05:47 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:45 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1052.eqiad.wmnet with OS trixie * 05:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:29 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1052.eqiad.wmnet with reason: host reimage * 05:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:22 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1052.eqiad.wmnet with reason: host reimage * 05:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 05:07 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1052.eqiad.wmnet with OS trixie * 05:06 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1052: Upgrading es1052.eqiad.wmnet * 05:06 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1052: Upgrading es1052.eqiad.wmnet * 05:05 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 05:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 04:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 04:49 ryankemper: [[phab:T425007|T425007]] (k8s) created 4 wdqs namespaces on `dse-k8s-codfw`'s `admin_ng` ns: `wdqs-[internal,external]` & `wdqs-[internal,external]-next`; certs issued * 04:46 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 04:40 ryankemper@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 04:36 ryankemper@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 04:05 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.2 (duration: 05m 33s) == 2026-06-01 == * 23:27 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295963{{!}}Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962{{!}}Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] (duration: 07m 17s) * 23:23 jdlrobson@deploy1003: mfossati, jdlrobson: Continuing with deployment * 23:22 jdlrobson@deploy1003: mfossati, jdlrobson: Backport for [[gerrit:1295963{{!}}Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962{{!}}Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:20 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1295963{{!}}Make MultimediaViewer compatible with MobileFrontend legacy parser (T427542)]], [[gerrit:1295962{{!}}Carousel: Defer to MobileFrontend lightbox on mobile (T427679)]] * 23:15 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296022{{!}}Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028{{!}}styles: Limit selector to badge client pref (T427407)]] (duration: 09m 33s) * 23:11 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 23:07 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1296022{{!}}Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028{{!}}styles: Limit selector to badge client pref (T427407)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:06 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1296022{{!}}Donor Delight Badge: Add dependency on mw.user (T427850)]], [[gerrit:1296028{{!}}styles: Limit selector to badge client pref (T427407)]] * 23:04 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6015.* * 22:36 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296024{{!}}Add maintenance script to scrape SVG render files]] (duration: 06m 22s) * 22:32 reedy@deploy1003: reedy: Continuing with deployment * 22:31 reedy@deploy1003: reedy: Backport for [[gerrit:1296024{{!}}Add maintenance script to scrape SVG render files]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:30 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1296024{{!}}Add maintenance script to scrape SVG render files]] * 22:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 22:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 22:00 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 21:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 21:56 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 21:51 sbassett: Deployed updated mitigation for [[phab:T326691|T326691]] * 21:50 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 21:35 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 21:35 maryum: Deployed security fix for [[phab:T427611|T427611]] * 21:35 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 21:33 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 21:32 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 21:27 maryum: Deployed security fix for [[phab:T427235|T427235]] * 21:13 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1296002{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009{{!}}Redirect Special:AccountRecovery to the shared domain (T427692)]] (duration: 09m 20s) * 21:09 catrope@deploy1003: catrope, arlolra: Continuing with deployment * 21:09 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 21:09 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 21:08 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 21:07 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 21:07 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 21:06 catrope@deploy1003: catrope, arlolra: Backport for [[gerrit:1296002{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009{{!}}Redirect Special:AccountRecovery to the shared domain (T427692)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:04 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1296002{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T353697 T415591 T427565)]], [[gerrit:1296003{{!}}Bump wikimedia/parsoid to 0.24.0-a7 (T427565)]], [[gerrit:1296009{{!}}Redirect Special:AccountRecovery to the shared domain (T427692)]] * 20:53 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 20:37 ryankemper@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wdqs1015.eqiad.wmnet with reason: [[phab:T427852|T427852]] hw failure * 20:26 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1285412{{!}}Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531{{!}}Enable AbuseFilter block action on nlwiki (T427384)]] (duration: 07m 48s) * 20:22 catrope@deploy1003: sfaci, xxblackburnxx, catrope: Continuing with deployment * 20:20 catrope@deploy1003: sfaci, xxblackburnxx, catrope: Backport for [[gerrit:1285412{{!}}Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531{{!}}Enable AbuseFilter block action on nlwiki (T427384)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:18 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1285412{{!}}Remove `wgTestKitchenExperimentStreamNames` (T422358)]], [[gerrit:1295531{{!}}Enable AbuseFilter block action on nlwiki (T427384)]] * 20:12 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295504{{!}}passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] (duration: 07m 37s) * 20:08 catrope@deploy1003: catrope: Continuing with deployment * 20:07 catrope@deploy1003: catrope: Backport for [[gerrit:1295504{{!}}passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:05 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1295504{{!}}passwordlessLogin: Don't immediately error out in unsupported browsers (T427562)]] * 19:48 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 19:47 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 19:47 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 19:46 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 19:46 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 19:45 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 19:01 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: sync * 19:00 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: sync * 18:24 otto@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295950{{!}}mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] (duration: 06m 42s) * 18:20 otto@deploy1003: otto: Continuing with deployment * 18:19 otto@deploy1003: otto: Backport for [[gerrit:1295950{{!}}mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:17 otto@deploy1003: Started scap sync-world: Backport for [[gerrit:1295950{{!}}mediawiki.user_change.dev0 - key by user.wiki_id (T426198)]] * 18:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 18:05 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 18:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd2001.codfw.wmnet to plain * 18:02 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 18:02 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd2001.codfw.wmnet to plain * 18:01 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2003.codfw.wmnet to plain * 18:01 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply * 18:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2003.codfw.wmnet to plain * 17:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 17:58 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 17:53 jasmine@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2006.codfw.wmnet with OS trixie * 17:42 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295976{{!}}nlwiki: change to Wikipedia 25 logo (T424519)]] (duration: 07m 29s) * 17:37 samtar@deploy1003: chlod, samtar: Continuing with deployment * 17:36 samtar@deploy1003: chlod, samtar: Backport for [[gerrit:1295976{{!}}nlwiki: change to Wikipedia 25 logo (T424519)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:34 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1295976{{!}}nlwiki: change to Wikipedia 25 logo (T424519)]] * 17:20 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1236: Update * 17:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd2001.codfw.wmnet to drbd * 17:04 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling * 17:04 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 17:04 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1180: Pooling * 17:03 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 17:03 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pooling * 17:03 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1180: Pooling * 16:59 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd2001.codfw.wmnet to drbd * 16:58 Amir1: drop flaggedrevs tables on wikinews wikis ([[phab:T423577|T423577]]) * 16:57 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host kafka-main2006.codfw.wmnet with OS trixie * 16:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93462 and previous config saved to /var/cache/conftool/dbconfig/20260601-165717-fceratto.json * 16:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93460 and previous config saved to /var/cache/conftool/dbconfig/20260601-164709-fceratto.json * 16:42 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling * 16:37 ryankemper@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,cluster=wdqs-main,service=wdqs-main,name=wdqs1015.eqiad.wmnet * 16:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93458 and previous config saved to /var/cache/conftool/dbconfig/20260601-163701-fceratto.json * 16:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:35 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1236.eqiad.wmnet * 16:35 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1236.eqiad.wmnet * 16:35 ryankemper@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,cluster=wdqs,service=wdqs-main,name=wdqs1015.eqiad.wmnet * 16:34 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update * 16:34 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1236: Update * 16:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1236.eqiad.wmnet with reason: Kernel update [[phab:T426633|T426633]] * 16:31 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:30 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1236.eqiad.wmnet * 16:30 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1236.eqiad.wmnet * 16:30 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update * 16:29 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1236: Update * 16:29 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1236: Update * 16:29 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2003.codfw.wmnet to drbd * 16:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93455 and previous config saved to /var/cache/conftool/dbconfig/20260601-162653-fceratto.json * 16:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 16:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1209: Migration of db1209.eqiad.wmnet completed * 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1236.eqiad.wmnet with reason: Kernel update [[phab:T426633|T426633]] * 16:09 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1236: Update * 16:09 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1236: Update * 16:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 16:07 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 16:06 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 16:05 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2003.codfw.wmnet to drbd * 16:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2045.codfw.wmnet * 16:03 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2045.codfw.wmnet * 16:02 moritzm: temporarily remove ganeti2027 from the codfw cluster [[phab:T427357|T427357]] * 15:56 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:56 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.depool (exit_code=97) depool db1224: Pooling * 15:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2005.codfw.wmnet with OS bullseye * 15:53 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1224: Pooling * 15:51 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 15:49 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling * 15:49 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:48 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet * 15:45 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1224: Pooling * 15:44 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2005.codfw.wmnet with reason: host reimage * 15:40 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:40 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling * 15:40 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet * 15:40 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet * 15:40 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet * 15:40 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet * 15:39 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 15:39 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 15:39 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1209: Migration of db1209.eqiad.wmnet completed * 15:39 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:38 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:38 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling * 15:37 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2005.codfw.wmnet with reason: host reimage * 15:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 15:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1209.eqiad.wmnet with OS trixie * 15:28 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295802{{!}}hCaptcha: Raise SiteVerify error threshold to 100]] (duration: 06m 15s) * 15:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93446 and previous config saved to /var/cache/conftool/dbconfig/20260601-152638-fceratto.json * 15:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 15:26 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:25 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1224.eqiad.wmnet * 15:25 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1224.eqiad.wmnet * 15:25 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1224: Pooling * 15:25 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1224: Pooling * 15:24 kharlan@deploy1003: kharlan: Continuing with deployment * 15:24 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295802{{!}}hCaptcha: Raise SiteVerify error threshold to 100]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:22 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host testvm2005.codfw.wmnet with OS bullseye * 15:22 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295802{{!}}hCaptcha: Raise SiteVerify error threshold to 100]] * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:22 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:20 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295946{{!}}hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] (duration: 08m 24s) * 15:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:16 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 15:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1209.eqiad.wmnet with reason: host reimage * 15:14 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1295946{{!}}hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:13 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:12 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1295946{{!}}hCaptcha: Enable for VisualEditor on all WMF wikis (T425940)]] * 15:10 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1209.eqiad.wmnet with reason: host reimage * 15:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93445 and previous config saved to /var/cache/conftool/dbconfig/20260601-151024-fceratto.json * 15:08 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-reboot (exit_code=0) rolling reboot on A:sessionstore * 15:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93443 and previous config saved to /var/cache/conftool/dbconfig/20260601-150017-fceratto.json * 14:55 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1209.eqiad.wmnet with OS trixie * 14:52 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 14:52 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1209: Upgrading db1209.eqiad.wmnet * 14:52 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 14:52 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1209: Upgrading db1209.eqiad.wmnet * 14:52 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 14:51 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:51 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 14:50 atsuko@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 14:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93441 and previous config saved to /var/cache/conftool/dbconfig/20260601-145010-fceratto.json * 14:49 atsuko@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 14:49 atsuko@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 14:48 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:42 atsuko@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 14:41 atsuko@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93440 and previous config saved to /var/cache/conftool/dbconfig/20260601-144002-fceratto.json * 14:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:30 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:30 ladsgroup@deploy1003: Synchronized portals: Deploy portals ([[phab:T421797|T421797]]) (duration: 02m 43s) * 14:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:27 ladsgroup@deploy1003: Synchronized portals/wikipedia.org/assets: Deploy portals ([[phab:T421797|T421797]]) (duration: 06m 10s) * 14:25 sukhe@dns1004: END - running authdns-update * 14:23 sukhe@dns1004: START - running authdns-update * 14:22 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 14:21 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:16 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:12 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:12 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:11 Lucas_WMDE: UTC afternoon backport+config window done * 14:10 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295918{{!}}Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] (duration: 11m 06s) * 14:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:05 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, codenamenoreste: Continuing with deployment * 14:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, codenamenoreste: Backport for [[gerrit:1295918{{!}}Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:02 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:01 eevans@cumin1003: START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:sessionstore * 13:58 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1295918{{!}}Remove sfsblock-bypass from the IP block exemption user group on all wikis (T427745)]] * 13:52 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:52 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1265.eqiad.wmnet with OS trixie * 13:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93439 and previous config saved to /var/cache/conftool/dbconfig/20260601-133947-fceratto.json * 13:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 13:37 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage * 13:35 atsukoito: restarted pybal.service on lvs2013 * 13:31 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage * 13:31 atsukoito: restarted pybal.service on lvs2014 * 13:24 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs-test2001.codfw.wmnet * 13:24 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:22 atsukoito: restarted pybal.service on lvs1019 * 13:22 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in eqiad/ml-serve-eqiad: maintenance * 13:21 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in eqiad/ml-serve-eqiad: maintenance * 13:20 atsukoito: restarted pybal.service on lvs1020 * 13:20 Msz2001: UTC afternoon backpot+config window done * 13:20 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295875{{!}}Add SetGlobalPreference maintenance script (T427476)]] (duration: 06m 22s) * 13:19 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs-test2001.codfw.wmnet * 13:18 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1265.eqiad.wmnet with OS trixie * 13:18 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:16 mszwarc@deploy1003: mszwarc: Continuing with deployment * 13:15 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1295875{{!}}Add SetGlobalPreference maintenance script (T427476)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:14 atsukoito: sudo cumin 'A:lvs-low-traffic-eqiad' 'systemctl restart pybal.service' * 13:14 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1295875{{!}}Add SetGlobalPreference maintenance script (T427476)]] * 13:12 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295536{{!}}swwiki: Enable the Visual Editor on the project namespace (T427117)]] (duration: 10m 06s) * 13:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93438 and previous config saved to /var/cache/conftool/dbconfig/20260601-130949-fceratto.json * 13:08 mszwarc@deploy1003: codenamenoreste, mszwarc: Continuing with deployment * 13:07 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 13:06 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . * 13:05 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 13:04 mszwarc@deploy1003: codenamenoreste, mszwarc: Backport for [[gerrit:1295536{{!}}swwiki: Enable the Visual Editor on the project namespace (T427117)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:04 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 13:04 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 13:03 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 13:02 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1295536{{!}}swwiki: Enable the Visual Editor on the project namespace (T427117)]] * 12:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93437 and previous config saved to /var/cache/conftool/dbconfig/20260601-125941-fceratto.json * 12:56 dpogorzelski@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=inference,name=eqiad * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'readability' for release 'main' . * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'logo-detection' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'edit-check' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 12:55 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . * 12:52 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:50 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:49 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P93436 and previous config saved to /var/cache/conftool/dbconfig/20260601-124934-fceratto.json * 12:48 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:47 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:46 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:44 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:43 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:42 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:41 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93435 and previous config saved to /var/cache/conftool/dbconfig/20260601-123926-fceratto.json * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:29 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:28 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 12:28 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:27 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:27 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2005.codfw.wmnet to plain * 12:26 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2005.codfw.wmnet to plain * 12:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet * 12:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 12:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2005.codfw.wmnet to drbd * 12:20 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 12:17 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 12:15 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in eqiad/ml-serve-eqiad: maintenance * 12:15 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in eqiad/ml-serve-eqiad: maintenance * 12:11 dpogorzelski@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=inference,name=eqiad * 12:07 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2005.codfw.wmnet to drbd * 12:05 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet * 12:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 12:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2027.codfw.wmnet * 12:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet * 11:59 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in eqiad/ml-serve-eqiad: maintenance * 11:59 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in eqiad/ml-serve-eqiad: maintenance * 11:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93434 and previous config saved to /var/cache/conftool/dbconfig/20260601-113911-fceratto.json * 11:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance * 11:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93433 and previous config saved to /var/cache/conftool/dbconfig/20260601-113843-fceratto.json * 11:37 moritzm: installing Exim security updates * 11:36 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:34 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:33 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:32 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:32 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:32 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:28 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93432 and previous config saved to /var/cache/conftool/dbconfig/20260601-112835-fceratto.json * 11:25 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 11:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:23 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:22 moritzm: installing imagemagick security updates * 11:22 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:22 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:22 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 11:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:21 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 11:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93430 and previous config saved to /var/cache/conftool/dbconfig/20260601-111827-fceratto.json * 11:17 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 11:14 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 11:12 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 11:10 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93429 and previous config saved to /var/cache/conftool/dbconfig/20260601-110820-fceratto.json * 11:04 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 11:01 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1055: repool after upgrade * 11:01 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93427 and previous config saved to /var/cache/conftool/dbconfig/20260601-110121-fceratto.json * 11:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance * 10:54 marostegui@dns1004: END - running authdns-update * 10:52 marostegui@dns1004: START - running authdns-update * 10:48 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1050 to es1 eqiad primary [[phab:T427032|T427032]]', diff saved to https://phabricator.wikimedia.org/P93425 and previous config saved to /var/cache/conftool/dbconfig/20260601-104837-marostegui.json * 10:47 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2055 to es1 codfw primary [[phab:T427032|T427032]]', diff saved to https://phabricator.wikimedia.org/P93424 and previous config saved to /var/cache/conftool/dbconfig/20260601-104739-marostegui.json * 10:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1177: Migration of db1177.eqiad.wmnet completed * 10:40 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet * 10:34 kamila@cumin1003: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet * 10:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93421 and previous config saved to /var/cache/conftool/dbconfig/20260601-103316-fceratto.json * 10:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93418 and previous config saved to /var/cache/conftool/dbconfig/20260601-102308-fceratto.json * 10:16 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1055: repool after upgrade * 10:15 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1055.eqiad.wmnet with OS trixie * 10:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P93415 and previous config saved to /var/cache/conftool/dbconfig/20260601-101300-fceratto.json * 10:09 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 10:07 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 10:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93414 and previous config saved to /var/cache/conftool/dbconfig/20260601-100252-fceratto.json * 10:00 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1177: Migration of db1177.eqiad.wmnet completed * 09:58 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1055.eqiad.wmnet with reason: host reimage * 09:56 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 09:54 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 09:53 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1055.eqiad.wmnet with reason: host reimage * 09:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1177.eqiad.wmnet with OS trixie * 09:51 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 09:50 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 09:39 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1055.eqiad.wmnet with OS trixie * 09:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1055: Upgrading es1055.eqiad.wmnet * 09:38 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1055: Upgrading es1055.eqiad.wmnet * 09:37 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1177.eqiad.wmnet with reason: host reimage * 09:31 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1177.eqiad.wmnet with reason: host reimage * 09:17 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1177.eqiad.wmnet with OS trixie * 09:15 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 09:14 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 09:13 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 09:12 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 09:12 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1177: Upgrading db1177.eqiad.wmnet * 09:11 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1177: Upgrading db1177.eqiad.wmnet * 09:11 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93410 and previous config saved to /var/cache/conftool/dbconfig/20260601-090237-fceratto.json * 09:02 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93409 and previous config saved to /var/cache/conftool/dbconfig/20260601-090209-fceratto.json * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P93408 and previous config saved to /var/cache/conftool/dbconfig/20260601-085202-fceratto.json * 08:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P93407 and previous config saved to /var/cache/conftool/dbconfig/20260601-084154-fceratto.json * 08:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93406 and previous config saved to /var/cache/conftool/dbconfig/20260601-083146-fceratto.json * 08:24 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93405 and previous config saved to /var/cache/conftool/dbconfig/20260601-082442-fceratto.json * 08:24 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1168.eqiad.wmnet with reason: Maintenance * 07:58 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295454{{!}}Disable the creation of synthetic main refs in production (T427484)]] (duration: 11m 26s) * 07:56 XioNoX: add no_p2p term to pfw1-codfw BGP_fundraising_export - [[phab:T423384|T423384]] * 07:52 wmde-fisch@deploy1003: lilients, wmde-fisch: Continuing with deployment * 07:51 wmde-fisch@deploy1003: lilients, wmde-fisch: Backport for [[gerrit:1295454{{!}}Disable the creation of synthetic main refs in production (T427484)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:47 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1295454{{!}}Disable the creation of synthetic main refs in production (T427484)]] * 07:45 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294826{{!}}Update VE core submodule to master (9cf5524e7) (T424232)]] (duration: 31m 34s) * 07:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 07:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 07:32 wmde-fisch@deploy1003: wmde-fisch: Continuing with deployment * 07:31 wmde-fisch@deploy1003: wmde-fisch: Backport for [[gerrit:1294826{{!}}Update VE core submodule to master (9cf5524e7) (T424232)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet * 07:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet * 07:13 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1294826{{!}}Update VE core submodule to master (9cf5524e7) (T424232)]] * 06:48 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 06:47 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. == 2026-05-31 == * 02:06 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 30s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-30 == * 16:21 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:21 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:21 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:21 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 06:39 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 06:39 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 06:39 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 06:38 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 27s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-29 == * 23:39 aokoth@cumin1003: END (PASS) - Cookbook sre.vrts.upgrade (exit_code=0) on VRTS host vrts1003.eqiad.wmnet * 23:37 aokoth@cumin1003: START - Cookbook sre.vrts.upgrade on VRTS host vrts1003.eqiad.wmnet * 21:42 catrope@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 21:41 catrope@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 17:40 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295487{{!}}Hide experiment if not active and no assigned group]] (duration: 06m 54s) * 17:35 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 17:34 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1295487{{!}}Hide experiment if not active and no assigned group]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:33 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1295487{{!}}Hide experiment if not active and no assigned group]] * 16:30 jgreen@dns1004: END - running authdns-update * 16:28 jgreen@dns1004: START - running authdns-update * 16:13 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:12 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 15:28 dancy@deploy1003: Installation of scap version "4.267.0" completed for 2 hosts * 15:26 dancy@deploy1003: Installing scap version "4.267.0" for 2 host(s) * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:15 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295466{{!}}GlobalPreferencesHandler: Cast auto-reveal expiry to int (T427625)]] (duration: 07m 58s) * 14:11 kharlan@deploy1003: kharlan: Continuing with deployment * 14:09 kharlan@deploy1003: kharlan: Backport for [[gerrit:1295466{{!}}GlobalPreferencesHandler: Cast auto-reveal expiry to int (T427625)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:07 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1295466{{!}}GlobalPreferencesHandler: Cast auto-reveal expiry to int (T427625)]] * 13:53 moritzm: imported OpenJDK 21 21.0.11+10-1~deb12u1 to component/jdk21 (backport of latest Java 21 security release for Bookworm) * 12:09 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader1006.wikimedia.org * 12:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1006.wikimedia.org with OS trixie * 11:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1006.wikimedia.org with reason: host reimage * 11:47 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1006.wikimedia.org with reason: host reimage * 11:36 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader1006.wikimedia.org with OS trixie * 11:15 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:15 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:13 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader1006.wikimedia.org on all recursors * 11:12 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader1006.wikimedia.org on all recursors * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:06 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1006.wikimedia.org - jmm@cumin2002" * 11:00 jmm@cumin2002: START - Cookbook sre.dns.netbox * 11:00 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader1006.wikimedia.org * 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader1005.wikimedia.org * 10:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1005.wikimedia.org with OS trixie * 10:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1005.wikimedia.org with reason: host reimage * 10:40 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2212: Pooling * 10:37 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1005.wikimedia.org with reason: host reimage * 10:27 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader1005.wikimedia.org with OS trixie * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 10:01 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:59 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:55 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 09:50 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab * 09:49 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:45 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:44 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup2014.codfw.wmnet with OS bookworm * 09:33 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:20 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup2014.codfw.wmnet with reason: host reimage * 09:12 jynus@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on backup2014.codfw.wmnet with reason: host reimage * 09:10 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 09:10 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 09:03 jelto@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM etherpad2002.codfw.wmnet * 08:59 jelto@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM etherpad2002.codfw.wmnet * 08:59 jelto: gnt-instance modify -B memory=4g,vcpus=1 etherpad2002.codfw.wmnet - [[phab:T427588|T427588]] * 08:54 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2014.codfw.wmnet with OS bookworm * 08:51 jelto@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM etherpad1004.eqiad.wmnet * 08:50 atsuko@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams-internal: apply * 08:50 jynus@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host backup2014.codfw.wmnet with OS bookworm * 08:49 atsuko@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams-internal: apply * 08:47 jelto@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM etherpad1004.eqiad.wmnet * 08:46 jelto: gnt-instance modify -B memory=4g,vcpus=1 etherpad1004.eqiad.wmnet - [[phab:T427588|T427588]] * 08:42 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2212: Pooling * 08:42 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 08:39 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2212: Pooling * 08:39 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 08:38 atsuko@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams-internal: apply * 08:37 atsuko@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams-internal: apply * 08:37 atsuko@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams-internal: apply * 08:36 atsuko@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams-internal: apply * 08:33 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2014.codfw.wmnet with OS bookworm * 08:31 jynus@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host backup2014.codfw.wmnet with OS bookworm * 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader1005.wikimedia.org on all recursors * 08:21 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader1005.wikimedia.org on all recursors * 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 08:21 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader1005.wikimedia.org - jmm@cumin2002" * 08:18 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 08:17 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 08:16 jmm@cumin2002: START - Cookbook sre.dns.netbox * 08:16 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader1005.wikimedia.org * 08:05 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2212: Pooling * 07:59 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 07:59 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 07:54 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2212: Pooling * 07:54 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2212.codfw.wmnet * 07:54 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2212.codfw.wmnet * 07:22 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host backup2014.codfw.wmnet with OS bookworm * 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader2006.wikimedia.org * 07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader2006.wikimedia.org with OS trixie * 06:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader2006.wikimedia.org with reason: host reimage * 06:53 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader2006.wikimedia.org with reason: host reimage * 06:34 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader2006.wikimedia.org with OS trixie * 06:32 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:32 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader2006.wikimedia.org on all recursors * 06:31 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader2006.wikimedia.org on all recursors * 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:31 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:31 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2006.wikimedia.org - jmm@cumin2002" * 06:27 jmm@cumin2002: START - Cookbook sre.dns.netbox * 06:27 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader2006.wikimedia.org * 03:01 vriley@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts db1224.eqiad.wmnet * 03:00 vriley@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1224.eqiad.wmnet * 03:00 vriley@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1224.eqiad.wmnet * 02:56 vriley@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1224.eqiad.wmnet * 01:47 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5032.eqsin.wmnet with OS trixie * 01:18 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5032.eqsin.wmnet with reason: host reimage * 01:14 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5032.eqsin.wmnet with reason: host reimage * 00:31 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cp5032.eqsin.wmnet with OS trixie * 00:29 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host cp5032.eqsin.wmnet * 00:23 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 00:22 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply * 00:21 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 00:21 amastilovic@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply == 2026-05-28 == * 23:07 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 23:07 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new ae1.522 interface - pt1979@cumin2002" * 23:07 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new ae1.522 interface - pt1979@cumin2002" * 23:02 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 22:34 andrewbogott: reprepro includedeb trixie-wikimedia /home/andrew/magnum-cluster-api_0.36.6-1~wmf13u2_amd64.deb * 22:31 logmsgbot: dreamyjazz Deployed security patch for [[phab:T426388|T426388]] * 21:33 maryum: Deployed security fix for [[phab:T426867|T426867]] * 21:21 alexsanford: Deployed security fix for [[phab:T426889|T426889]] * 21:07 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host cp5032.eqsin.wmnet * 21:04 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "setup new eqsin vlan - pt1979@cumin2002 - [[phab:T427393|T427393]]" * 21:04 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "setup new eqsin vlan - pt1979@cumin2002 - [[phab:T427393|T427393]]" * 20:48 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1295066{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T420336 T427098 T427354 T427082)]], [[gerrit:1295067{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T427082)]] (duration: 07m 34s) * 20:44 arlolra@deploy1003: arlolra: Continuing with deployment * 20:43 arlolra@deploy1003: arlolra: Backport for [[gerrit:1295066{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T420336 T427098 T427354 T427082)]], [[gerrit:1295067{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T427082)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:41 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1295066{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T420336 T427098 T427354 T427082)]], [[gerrit:1295067{{!}}Bump wikimedia/parsoid to 0.24.0-a6 (T427082)]] * 20:34 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293805{{!}}Deploy PRV to 7 wikis (T427331)]] (duration: 07m 20s) * 20:30 arlolra@deploy1003: arlolra: Continuing with deployment * 20:29 arlolra@deploy1003: arlolra: Backport for [[gerrit:1293805{{!}}Deploy PRV to 7 wikis (T427331)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:27 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1293805{{!}}Deploy PRV to 7 wikis (T427331)]] * 20:22 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291996{{!}}Replace deprecated Hooks::getInstance (T426981)]], [[gerrit:1294393{{!}}Permissions: Create wmf-officeit group on officewiki]], [[gerrit:1294229{{!}}Deploy IRS Direct Reporting feature to enwiki (T427369)]], [[gerrit:1295039{{!}}Add 2FA enforcement demotion config for phase 2 groups (T423119)]] (duration: 09m 07s) * 20:18 stran@deploy1003: alexsanford, stran, catrope, dreamyjazz: Continuing with deployment * 20:14 stran@deploy1003: alexsanford, stran, catrope, dreamyjazz: Backport for [[gerrit:1291996{{!}}Replace deprecated Hooks::getInstance (T426981)]], [[gerrit:1294393{{!}}Permissions: Create wmf-officeit group on officewiki]], [[gerrit:1294229{{!}}Deploy IRS Direct Reporting feature to enwiki (T427369)]], [[gerrit:1295039{{!}}Add 2FA enforcement demotion config for phase 2 groups (T423119)]] synced to the testservers (see https://wikitech. * 20:13 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5032.eqsin.wmnet with OS trixie * 20:13 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1291996{{!}}Replace deprecated Hooks::getInstance (T426981)]], [[gerrit:1294393{{!}}Permissions: Create wmf-officeit group on officewiki]], [[gerrit:1294229{{!}}Deploy IRS Direct Reporting feature to enwiki (T427369)]], [[gerrit:1295039{{!}}Add 2FA enforcement demotion config for phase 2 groups (T423119)]] * 19:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1018.eqiad.wmnet * 19:27 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1018.eqiad.wmnet * 19:09 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1018.eqiad.wmnet with reason: Kernel reboot * 19:09 brett: Stopping pybal/puppet/downtiming lvs1018.eqiad.wmnet for reboot * 19:05 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1019.eqiad.wmnet * 19:05 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1019.eqiad.wmnet * 18:52 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cp5032.eqsin.wmnet with OS trixie * 18:51 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:51 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change cp5032 IP - pt1979@cumin2002" * 18:51 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change cp5032 IP - pt1979@cumin2002" * 18:47 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 18:40 mutante: planet1003/planet2003 - apt-get upgrade - all pending package upgrades * 18:35 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: Kernel reboot * 18:34 brett: Stopping pybal/puppet/downtiming lvs1019.eqiad.wmnet for reboot and BIOS update/memory self-healing - [[phab:T426109|T426109]] * 18:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs2011.codfw.wmnet * 18:25 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs2011.codfw.wmnet * 18:19 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2011.codfw.wmnet with reason: Kernel reboot * 18:19 brett: Stopping pybal/puppet/downtiming lvs2011.codfw.wmnet for reboot * 18:09 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs2013.codfw.wmnet * 18:06 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs2013.codfw.wmnet * 18:00 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2013.codfw.wmnet with reason: Kernel reboot * 17:57 brett: Stopping pybal/puppet/downtiming lvs2013.codfw.wmnet for reboot * 17:19 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 17:18 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 16:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93393 and previous config saved to /var/cache/conftool/dbconfig/20260528-164514-fceratto.json * 16:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P93392 and previous config saved to /var/cache/conftool/dbconfig/20260528-163507-fceratto.json * 16:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P93391 and previous config saved to /var/cache/conftool/dbconfig/20260528-162459-fceratto.json * 16:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db1224.eqiad.wmnet with reason: unreachable [[phab:T427535|T427535]] * 16:17 swfrench-wmf: reprepro include xdebug_3.4.4-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:17 swfrench-wmf: reprepro include wikidiff2_1.14.1-2+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:17 swfrench-wmf: reprepro include php-yaml_2.2.4-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-xhprof_2.3.10-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-wmerrors_2.0.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-uuid_1.3.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:16 swfrench-wmf: reprepro include php-redis_6.2.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 swfrench-wmf: reprepro include php-pcov_1.0.12-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 swfrench-wmf: reprepro include php-memcached_3.3.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 16:15 swfrench-wmf: reprepro include php-luasandbox_4.1.2-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:15 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 16:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93390 and previous config saved to /var/cache/conftool/dbconfig/20260528-161452-fceratto.json * 16:14 swfrench-wmf: reprepro include php-imagick_3.7.0-13+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:14 swfrench-wmf: reprepro include php-excimer_1.2.5-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 16:09 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:09 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:06 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1251 ([[phab:T426633|T426633]])', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20260528-160646-fceratto.json * 16:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1251.eqiad.wmnet with reason: Maintenance * 16:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93388 and previous config saved to /var/cache/conftool/dbconfig/20260528-160613-fceratto.json * 15:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P93387 and previous config saved to /var/cache/conftool/dbconfig/20260528-155605-fceratto.json * 15:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P93386 and previous config saved to /var/cache/conftool/dbconfig/20260528-154557-fceratto.json * 15:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93385 and previous config saved to /var/cache/conftool/dbconfig/20260528-153550-fceratto.json * 15:27 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1235 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93384 and previous config saved to /var/cache/conftool/dbconfig/20260528-152736-fceratto.json * 15:27 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1235.eqiad.wmnet with reason: Maintenance * 15:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93383 and previous config saved to /var/cache/conftool/dbconfig/20260528-152708-fceratto.json * 15:20 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp5032.eqsin.wmnet with reason: Testing reimaging on new subnet * 15:18 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5032.* * 15:17 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P93382 and previous config saved to /var/cache/conftool/dbconfig/20260528-151701-fceratto.json * 15:17 jhathaway: dmarc ingress test on mx-in1001 * 15:14 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:14 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P93381 and previous config saved to /var/cache/conftool/dbconfig/20260528-150653-fceratto.json * 14:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93380 and previous config saved to /var/cache/conftool/dbconfig/20260528-145646-fceratto.json * 14:56 moritzm: installing nginx security updates * 14:49 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1234 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93379 and previous config saved to /var/cache/conftool/dbconfig/20260528-144936-fceratto.json * 14:49 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 14:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1234.eqiad.wmnet with reason: Maintenance * 14:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93378 and previous config saved to /var/cache/conftool/dbconfig/20260528-144909-fceratto.json * 14:48 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host urldownloader2005.wikimedia.org * 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader2005.wikimedia.org with OS trixie * 14:47 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 14:39 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2189.codfw.wmnet * 14:39 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2189.codfw.wmnet * 14:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P93377 and previous config saved to /var/cache/conftool/dbconfig/20260528-143901-fceratto.json * 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader2005.wikimedia.org with reason: host reimage * 14:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P93376 and previous config saved to /var/cache/conftool/dbconfig/20260528-142854-fceratto.json * 14:28 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:28 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader2005.wikimedia.org with reason: host reimage * 14:27 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:19 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294998{{!}}ImageContentLookup: Fix issue created by strict types (T427505)]], [[gerrit:1295001{{!}}Enable hCaptcha for VisualEditor in group 1 (T425940)]] (duration: 11m 29s) * 14:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93375 and previous config saved to /var/cache/conftool/dbconfig/20260528-141846-fceratto.json * 14:15 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1232 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93374 and previous config saved to /var/cache/conftool/dbconfig/20260528-141029-fceratto.json * 14:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1232.eqiad.wmnet with reason: Maintenance * 14:10 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host urldownloader2005.wikimedia.org with OS trixie * 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93373 and previous config saved to /var/cache/conftool/dbconfig/20260528-141001-fceratto.json * 14:09 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1294998{{!}}ImageContentLookup: Fix issue created by strict types (T427505)]], [[gerrit:1295001{{!}}Enable hCaptcha for VisualEditor in group 1 (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:08 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294998{{!}}ImageContentLookup: Fix issue created by strict types (T427505)]], [[gerrit:1295001{{!}}Enable hCaptcha for VisualEditor in group 1 (T425940)]] * 14:00 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down * 13:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P93371 and previous config saved to /var/cache/conftool/dbconfig/20260528-135951-fceratto.json * 13:58 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp6015.drmrs.wmnet,service=(cdn{{!}}ats-be) * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:55 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) urldownloader2005.wikimedia.org on all recursors * 13:55 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache urldownloader2005.wikimedia.org on all recursors * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:55 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM urldownloader2005.wikimedia.org - jmm@cumin2002" * 13:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P93370 and previous config saved to /var/cache/conftool/dbconfig/20260528-134944-fceratto.json * 13:40 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 13:40 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 13:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93369 and previous config saved to /var/cache/conftool/dbconfig/20260528-133936-fceratto.json * 13:39 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 13:38 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 13:36 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294986{{!}}Image Carousel: check candidate pages (T427336)]] (duration: 06m 40s) * 13:34 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 13:33 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 13:32 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93368 and previous config saved to /var/cache/conftool/dbconfig/20260528-133230-fceratto.json * 13:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance * 13:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93367 and previous config saved to /var/cache/conftool/dbconfig/20260528-133202-fceratto.json * 13:31 mlitn@deploy1003: mlitn: Continuing with deployment * 13:31 mlitn@deploy1003: mlitn: Backport for [[gerrit:1294986{{!}}Image Carousel: check candidate pages (T427336)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:29 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1294986{{!}}Image Carousel: check candidate pages (T427336)]] * 13:22 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 13:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P93366 and previous config saved to /var/cache/conftool/dbconfig/20260528-132155-fceratto.json * 13:21 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 13:17 elukey: clean up a lof ot stale Kafka ACLs on Kafka Jumbo - Details in [[phab:T425528|T425528]] * 13:14 jmm@cumin2002: START - Cookbook sre.dns.netbox * 13:14 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host urldownloader2005.wikimedia.org * 13:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P93365 and previous config saved to /var/cache/conftool/dbconfig/20260528-131147-fceratto.json * 13:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93364 and previous config saved to /var/cache/conftool/dbconfig/20260528-130139-fceratto.json * 12:54 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1218 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93363 and previous config saved to /var/cache/conftool/dbconfig/20260528-125439-fceratto.json * 12:54 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1218.eqiad.wmnet with reason: Maintenance * 12:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93362 and previous config saved to /var/cache/conftool/dbconfig/20260528-125412-fceratto.json * 12:48 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 12:48 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 12:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P93361 and previous config saved to /var/cache/conftool/dbconfig/20260528-124404-fceratto.json * 12:44 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 12:43 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 12:39 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 12:38 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 12:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P93360 and previous config saved to /var/cache/conftool/dbconfig/20260528-123357-fceratto.json * 12:25 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest1006.eqiad.wmnet with OS trixie * 12:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93359 and previous config saved to /var/cache/conftool/dbconfig/20260528-122349-fceratto.json * 12:15 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93358 and previous config saved to /var/cache/conftool/dbconfig/20260528-121551-fceratto.json * 12:15 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance * 12:15 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie * 12:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93357 and previous config saved to /var/cache/conftool/dbconfig/20260528-121523-fceratto.json * 12:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P93356 and previous config saved to /var/cache/conftool/dbconfig/20260528-120515-fceratto.json * 12:02 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest1006.eqiad.wmnet with OS trixie * 12:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 12:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 12:01 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 12:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 11:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P93355 and previous config saved to /var/cache/conftool/dbconfig/20260528-115508-fceratto.json * 11:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93354 and previous config saved to /var/cache/conftool/dbconfig/20260528-114500-fceratto.json * 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93353 and previous config saved to /var/cache/conftool/dbconfig/20260528-113635-fceratto.json * 11:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 11:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance * 11:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93352 and previous config saved to /var/cache/conftool/dbconfig/20260528-113559-fceratto.json * 11:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P93351 and previous config saved to /var/cache/conftool/dbconfig/20260528-112551-fceratto.json * 11:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P93350 and previous config saved to /var/cache/conftool/dbconfig/20260528-111543-fceratto.json * 11:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93349 and previous config saved to /var/cache/conftool/dbconfig/20260528-110536-fceratto.json * 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1195 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93348 and previous config saved to /var/cache/conftool/dbconfig/20260528-105820-fceratto.json * 10:58 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie * 10:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1195.eqiad.wmnet with reason: Maintenance * 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93347 and previous config saved to /var/cache/conftool/dbconfig/20260528-105753-fceratto.json * 10:56 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 10:55 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 10:55 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 10:55 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 10:50 moritzm: update trixie netboot image for 13.5 point release [[phab:T427072|T427072]] * 10:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P93346 and previous config saved to /var/cache/conftool/dbconfig/20260528-104745-fceratto.json * 10:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P93345 and previous config saved to /var/cache/conftool/dbconfig/20260528-103738-fceratto.json * 10:29 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P13724 # [[phab:T406971|T406971]] * 10:28 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P14223 # [[phab:T422264|T422264]] * 10:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93344 and previous config saved to /var/cache/conftool/dbconfig/20260528-102730-fceratto.json * 10:26 arthurtaylor@deploy1003: mwscript-k8s job started: extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type external-id --property-id P1748 # [[phab:T422392|T422392]] * 10:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93343 and previous config saved to /var/cache/conftool/dbconfig/20260528-101900-fceratto.json * 10:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance * 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93342 and previous config saved to /var/cache/conftool/dbconfig/20260528-101829-fceratto.json * 10:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P93341 and previous config saved to /var/cache/conftool/dbconfig/20260528-100822-fceratto.json * 09:59 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290687{{!}}stream: webrequest.page_view (T426092 T426091)]] (duration: 06m 41s) * 09:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P93340 and previous config saved to /var/cache/conftool/dbconfig/20260528-095814-fceratto.json * 09:55 javiermonton@deploy1003: javiermonton: Continuing with deployment * 09:54 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1290687{{!}}stream: webrequest.page_view (T426092 T426091)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:52 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1290687{{!}}stream: webrequest.page_view (T426092 T426091)]] * 09:48 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294243{{!}}Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937{{!}}CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] (duration: 07m 37s) * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93339 and previous config saved to /var/cache/conftool/dbconfig/20260528-094807-fceratto.json * 09:44 dreamyjazz@deploy1003: dreamyjazz, stran: Continuing with deployment * 09:44 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:43 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:43 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:43 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:42 dreamyjazz@deploy1003: dreamyjazz, stran: Backport for [[gerrit:1294243{{!}}Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937{{!}}CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294243{{!}}Set minimum edit count for skipcaptcha right to 10 (T426973)]], [[gerrit:1294937{{!}}CheckUserLookupUtils: Fix error introduced by strict types (T427480)]] * 09:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93338 and previous config saved to /var/cache/conftool/dbconfig/20260528-093920-fceratto.json * 09:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: Maintenance * 09:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93337 and previous config saved to /var/cache/conftool/dbconfig/20260528-093849-fceratto.json * 09:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P93336 and previous config saved to /var/cache/conftool/dbconfig/20260528-092842-fceratto.json * 09:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance * 09:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93335 and previous config saved to /var/cache/conftool/dbconfig/20260528-092239-fceratto.json * 09:22 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pki-root1001.eqiad.wmnet * 09:22 elukey@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:22 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki-root1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1003" * 09:22 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki-root1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1003" * 09:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:18 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 09:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P93334 and previous config saved to /var/cache/conftool/dbconfig/20260528-091834-fceratto.json * 09:18 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 09:18 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 09:17 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1165: Reboot completed * 09:17 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 09:17 elukey@cumin1003: START - Cookbook sre.dns.netbox * 09:14 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 09:13 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:13 elukey@cumin1003: START - Cookbook sre.hosts.decommission for hosts pki-root1001.eqiad.wmnet * 09:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P93332 and previous config saved to /var/cache/conftool/dbconfig/20260528-091231-fceratto.json * 09:09 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:09 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93331 and previous config saved to /var/cache/conftool/dbconfig/20260528-090826-fceratto.json * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P93329 and previous config saved to /var/cache/conftool/dbconfig/20260528-090224-fceratto.json * 09:02 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Deploying to prod (duration: 02m 31s) * 09:01 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2216 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93328 and previous config saved to /var/cache/conftool/dbconfig/20260528-090114-fceratto.json * 09:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2216.codfw.wmnet with reason: Maintenance * 09:00 joal@deploy1003: Finished deploy [analytics/refinery@878cb24] (thin): Regular analytics weekly train THIN - 2[analytics/refinery@878cb24a] (duration: 02m 08s) * 08:59 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Deploying to prod * 08:58 joal@deploy1003: Started deploy [analytics/refinery@878cb24] (thin): Regular analytics weekly train THIN - 2[analytics/refinery@878cb24a] * 08:57 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Testing on backup host (duration: 00m 53s) * 08:56 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@6200ab1] (releasing): [[phab:T427406|T427406]] Testing on backup host * 08:56 joal@deploy1003: Finished deploy [analytics/refinery@878cb24]: Regular analytics weekly train - 2 [analytics/refinery@878cb24a] (duration: 06m 54s) * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93327 and previous config saved to /var/cache/conftool/dbconfig/20260528-085216-fceratto.json * 08:50 XioNoX: cr1-codfw# delete protocols bgp group fundraising family inet6 - [[phab:T423384|T423384]] * 08:49 joal@deploy1003: Started deploy [analytics/refinery@878cb24]: Regular analytics weekly train - 2 [analytics/refinery@878cb24a] * 08:49 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294925{{!}}hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] (duration: 09m 20s) * 08:49 joal@deploy1003: Finished deploy [analytics/refinery@878cb24] (hadoop-test): Regular analytics weekly train TEST -2 [analytics/refinery@878cb24a] (duration: 02m 00s) * 08:49 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P93326 and previous config saved to /var/cache/conftool/dbconfig/20260528-084906-fceratto.json * 08:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1209.eqiad.wmnet with reason: Maintenance * 08:48 slyngshede@dns1004: END - running authdns-update * 08:47 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1165: Reboot completed * 08:47 joal@deploy1003: Started deploy [analytics/refinery@878cb24] (hadoop-test): Regular analytics weekly train TEST -2 [analytics/refinery@878cb24a] * 08:47 slyngs: Upgrade IDP to CAS 7.3.7.1 * 08:46 slyngshede@dns1004: START - running authdns-update * 08:45 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 08:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93324 and previous config saved to /var/cache/conftool/dbconfig/20260528-084149-fceratto.json * 08:41 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1294925{{!}}hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1294925{{!}}hCaptcha: Regenerate VisualEditor captcha token per save attempt (T427334)]] * 08:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet * 08:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet * 08:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93323 and previous config saved to /var/cache/conftool/dbconfig/20260528-083504-fceratto.json * 08:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1025].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance * 08:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1165.eqiad.wmnet with reason: Maintenance * 08:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93322 and previous config saved to /var/cache/conftool/dbconfig/20260528-083331-fceratto.json * 08:24 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1209: Test * 08:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P93320 and previous config saved to /var/cache/conftool/dbconfig/20260528-082324-fceratto.json * 08:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2189: repool after crash * 08:17 slyngshede@dns1004: END - running authdns-update * 08:16 slyngshede@dns1004: START - running authdns-update * 08:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P93318 and previous config saved to /var/cache/conftool/dbconfig/20260528-081316-fceratto.json * 08:10 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 08:09 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1209: Test * 08:05 hashar@deploy1003: Finished deploy [integration/docroot@2a51016]: build: update dependencies + eslint fix in comment. f021d3f..2a51016 (duration: 00m 13s) * 08:05 hashar@deploy1003: Started deploy [integration/docroot@2a51016]: build: update dependencies + eslint fix in comment. f021d3f..2a51016 * 08:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93315 and previous config saved to /var/cache/conftool/dbconfig/20260528-080309-fceratto.json * 07:56 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93314 and previous config saved to /var/cache/conftool/dbconfig/20260528-075631-fceratto.json * 07:56 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020,1022-1023].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 07:56 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance * 07:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93313 and previous config saved to /var/cache/conftool/dbconfig/20260528-075521-fceratto.json * 07:47 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab replica * 07:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93311 and previous config saved to /var/cache/conftool/dbconfig/20260528-074513-fceratto.json * 07:37 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2189: repool after crash * 07:36 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab replica * 07:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93309 and previous config saved to /var/cache/conftool/dbconfig/20260528-073506-fceratto.json * 07:34 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica * 07:29 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294808{{!}}Don't run the click intent experiment on mobile (T426743)]] (duration: 06m 29s) * 07:25 wmde-fisch@deploy1003: thiemowmde, wmde-fisch: Continuing with deployment * 07:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93308 and previous config saved to /var/cache/conftool/dbconfig/20260528-072458-fceratto.json * 07:24 wmde-fisch@deploy1003: thiemowmde, wmde-fisch: Backport for [[gerrit:1294808{{!}}Don't run the click intent experiment on mobile (T426743)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:24 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica * 07:23 tgr@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwikisource --logwiki=metawiki Ioed Renamed_user_4232d41570b9e8f46ef150e5e360e446 # [[phab:T427459|T427459]] * 07:22 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1294808{{!}}Don't run the click intent experiment on mobile (T426743)]] * 07:20 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1270986{{!}}Update wikimania wordmark for 2026 (T413331)]] (duration: 06m 54s) * 07:18 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93307 and previous config saved to /var/cache/conftool/dbconfig/20260528-071836-fceratto.json * 07:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1264.eqiad.wmnet with reason: Maintenance * 07:16 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1167: Reboot completed * 07:16 wmde-fisch@deploy1003: wmde-fisch, robertsky: Continuing with deployment * 07:15 wmde-fisch@deploy1003: wmde-fisch, robertsky: Backport for [[gerrit:1270986{{!}}Update wikimania wordmark for 2026 (T413331)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:13 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1270986{{!}}Update wikimania wordmark for 2026 (T413331)]] * 07:11 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1289898{{!}}Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] (duration: 07m 15s) * 07:07 wmde-fisch@deploy1003: wmde-fisch, arthurtaylor: Continuing with deployment * 07:06 wmde-fisch@deploy1003: wmde-fisch, arthurtaylor: Backport for [[gerrit:1289898{{!}}Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:04 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1289898{{!}}Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] * 06:43 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1167: Reboot completed * 06:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93303 and previous config saved to /var/cache/conftool/dbconfig/20260528-064217-fceratto.json * 06:33 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93302 and previous config saved to /var/cache/conftool/dbconfig/20260528-063357-fceratto.json * 06:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance * 06:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance * 06:25 hashar: Restarting CI Jenkins for plugins upgrades * 06:16 fceratto@dns1005: END - running authdns-update * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1209 [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93301 and previous config saved to /var/cache/conftool/dbconfig/20260528-061609-fceratto.json * 06:14 fceratto@dns1005: START - running authdns-update * 06:11 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1193 to s8 primary and set section read-write [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93300 and previous config saved to /var/cache/conftool/dbconfig/20260528-061138-fceratto.json * 06:10 fceratto@cumin1003: dbctl commit (dc=all): 'Set s8 eqiad as read-only for maintenance - [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93299 and previous config saved to /var/cache/conftool/dbconfig/20260528-061048-fceratto.json * 06:10 federico3: Starting s8 eqiad failover from db1209 to db1193 - [[phab:T426095|T426095]] * 06:04 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1193 with weight 0 [[phab:T426095|T426095]]', diff saved to https://phabricator.wikimedia.org/P93298 and previous config saved to /var/cache/conftool/dbconfig/20260528-060412-fceratto.json * 06:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s8 [[phab:T426095|T426095]] * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 41s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:53 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 00:53 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new subnet in eqsin - pt1979@cumin2002" * 00:53 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new subnet in eqsin - pt1979@cumin2002" * 00:49 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 00:25 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294470{{!}}Activate conductwiki (T426984)]] (duration: 07m 12s) * 00:21 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 00:20 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1294470{{!}}Activate conductwiki (T426984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 00:18 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1294470{{!}}Activate conductwiki (T426984)]] * 00:12 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294438{{!}}Init conductwiki (T426984)]] (duration: 07m 25s) * 00:09 swfrench-wmf: reprepro include php-msgpack_3.0.0-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 00:08 swfrench-wmf: reprepro include php-igbinary_3.2.16-4+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 00:08 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 00:06 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1294438{{!}}Init conductwiki (T426984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 00:04 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1294438{{!}}Init conductwiki (T426984)]] * 00:04 swfrench-wmf: reprepro include php-apcu_5.1.24-1+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] == 2026-05-27 == * 23:13 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294432{{!}}Exclude more content from selection (T426308)]], [[gerrit:1285523{{!}}Remove MinervaNightMode config after skin cleanup (T426689)]] (duration: 08m 42s) * 23:09 jdlrobson@deploy1003: jdlrobson, h2o, egardner: Continuing with deployment * 23:06 jdlrobson@deploy1003: jdlrobson, h2o, egardner: Backport for [[gerrit:1294432{{!}}Exclude more content from selection (T426308)]], [[gerrit:1285523{{!}}Remove MinervaNightMode config after skin cleanup (T426689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:04 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294432{{!}}Exclude more content from selection (T426308)]], [[gerrit:1285523{{!}}Remove MinervaNightMode config after skin cleanup (T426689)]] * 22:58 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294435{{!}}passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] (duration: 07m 49s) * 22:55 ladsgroup@cumin1003: END (PASS) - Cookbook sre.mysql.sanitarium_restart (exit_code=0) * 22:54 catrope@deploy1003: catrope: Continuing with deployment * 22:52 catrope@deploy1003: catrope: Backport for [[gerrit:1294435{{!}}passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:50 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1294435{{!}}passwordlessLogin: Limit conditional mediation to the main login form (T427419)]] * 22:46 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294360{{!}}Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322{{!}}Thumbnails are not being optimized in large mode (T427237)]] (duration: 06m 54s) * 22:42 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 22:41 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1294360{{!}}Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322{{!}}Thumbnails are not being optimized in large mode (T427237)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:40 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart * 22:40 ladsgroup@cumin1003: END (FAIL) - Cookbook sre.mysql.sanitarium_restart (exit_code=99) * 22:40 ladsgroup@cumin1003: START - Cookbook sre.mysql.sanitarium_restart * 22:39 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294360{{!}}Thumbnails are not being optimized in large mode (T427237)]], [[gerrit:1294322{{!}}Thumbnails are not being optimized in large mode (T427237)]] * 22:39 ladsgroup@deploy1003: Finished scap sync-world: Add conduct.wikimedia.org ([[phab:T426984|T426984]]) (duration: 07m 16s) * 22:35 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 22:34 ladsgroup@deploy1003: ladsgroup: Add conduct.wikimedia.org ([[phab:T426984|T426984]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:33 ladsgroup@deploy1003: Started scap sync-world: Add conduct.wikimedia.org ([[phab:T426984|T426984]]) * 22:13 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294370{{!}}Carousel only on articles (T427336)]] (duration: 10m 00s) * 22:09 egardner@deploy1003: egardner: Continuing with deployment * 22:05 egardner@deploy1003: egardner: Backport for [[gerrit:1294370{{!}}Carousel only on articles (T427336)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:03 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1294370{{!}}Carousel only on articles (T427336)]] * 21:37 bking@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 15 days, 0:00:00 on relforge[1008-1010].eqiad.wmnet with reason: non-production environment * 21:20 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 21:20 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 21:20 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 21:19 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 21:04 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1288370{{!}}Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819{{!}}Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] (duration: 07m 38s) * 20:59 ebernhardson@deploy1003: matmarex, ebernhardson, pppery: Continuing with deployment * 20:58 ebernhardson@deploy1003: matmarex, ebernhardson, pppery: Backport for [[gerrit:1288370{{!}}Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819{{!}}Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:56 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1288370{{!}}Allow Vector 2022 font size changes in namespace 100 for enwiktionary (T423766)]], [[gerrit:1293819{{!}}Fix case of 'commonsfinder' in $wgUrlProtocols (T426614)]] * 20:51 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294373{{!}}identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374{{!}}Revert^2 "cirrus: AB test query suggester variants" (T407432)]] (duration: 07m 30s) * 20:47 ebernhardson@deploy1003: ebernhardson: Continuing with deployment * 20:46 ebernhardson@deploy1003: ebernhardson: Backport for [[gerrit:1294373{{!}}identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374{{!}}Revert^2 "cirrus: AB test query suggester variants" (T407432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:44 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294373{{!}}identity: Prune private ips from x-forwarded-for (T407432)]], [[gerrit:1294374{{!}}Revert^2 "cirrus: AB test query suggester variants" (T407432)]] * 20:43 swfrench-wmf: reprepro include dh-php_5.5+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 20:39 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts lvs1016.eqiad.wmnet * 20:39 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:39 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brett@cumin2002" * 20:38 swfrench-wmf: reprepro include php-defaults_94+wmf12u1 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 20:37 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brett@cumin2002" * 20:31 brett@cumin2002: START - Cookbook sre.dns.netbox * 20:27 swfrench-wmf: reprepro include php8.3_8.3.31-1+wmf12u2 into component/php83 for bookworm-wikimedia - [[phab:T427312|T427312]] * 20:25 brett@cumin2002: START - Cookbook sre.hosts.decommission for hosts lvs1016.eqiad.wmnet * 20:25 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294342{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344{{!}}frwiki: restrict Article Guidance experiment to junior editors (T426871)]] (duration: 08m 11s) * 20:21 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host lvs1016.eqiad.wmnet with OS bullseye * 20:21 sbisson@deploy1003: sbisson: Continuing with deployment * 20:20 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1020.eqiad.wmnet * 20:19 sbisson@deploy1003: sbisson: Backport for [[gerrit:1294342{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344{{!}}frwiki: restrict Article Guidance experiment to junior editors (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be v * 20:17 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1294342{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294343{{!}}Allow disabling experiment for experienced editors (>=100 edits) (T426871)]], [[gerrit:1294344{{!}}frwiki: restrict Article Guidance experiment to junior editors (T426871)]] * 20:14 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet * 20:05 cmooney@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 12355 * 20:04 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 12355 * 19:51 brett@cumin2002: START - Cookbook sre.hosts.reimage for host lvs1016.eqiad.wmnet with OS bullseye * 19:48 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 19:48 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 19:46 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 19:45 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 19:45 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 19:32 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp6016.drmrs.wmnet,cp[1112,1114].eqiad.wmnet,cp[5024,5031-5032].eqsin.wmnet<nowiki>}</nowiki> and A:cp * 19:32 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5032.eqsin.wmnet * 19:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 19:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 19:01 joal@deploy1003: Finished deploy [analytics/refinery@96cf761] (thin): Regular analytics weekly train THIN [analytics/refinery@96cf761f] (duration: 02m 08s) * 18:59 joal@deploy1003: Started deploy [analytics/refinery@96cf761] (thin): Regular analytics weekly train THIN [analytics/refinery@96cf761f] * 18:58 joal@deploy1003: Finished deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] (duration: 05m 01s) * 18:53 joal@deploy1003: Started deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] * 18:53 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294376{{!}}Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375{{!}}auth: Mark the hidden token field used for reauth as skippable (T427398)]] (duration: 07m 41s) * 18:49 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5031.eqsin.wmnet * 18:49 catrope@deploy1003: catrope: Continuing with deployment * 18:47 catrope@deploy1003: catrope: Backport for [[gerrit:1294376{{!}}Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375{{!}}auth: Mark the hidden token field used for reauth as skippable (T427398)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:45 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1294376{{!}}Fix lastAuthTimestamp hack (T427398)]], [[gerrit:1294375{{!}}auth: Mark the hidden token field used for reauth as skippable (T427398)]] * 18:40 joal@deploy1003: Finished deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] (duration: 01m 05s) * 18:39 joal@deploy1003: Started deploy [analytics/refinery@96cf761]: Regular analytics weekly train [analytics/refinery@96cf761f] * 18:37 joal@deploy1003: Finished deploy [analytics/refinery@96cf761] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96cf761f] (duration: 02m 04s) * 18:35 joal@deploy1003: Started deploy [analytics/refinery@96cf761] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96cf761f] * 18:29 swfrench@deploy1003: Finished scap sync-world: Helmfile-only deployment to clean up unused mesh listeners (duration: 06m 12s) * 18:25 swfrench@deploy1003: swfrench: Continuing with deployment * 18:24 swfrench@deploy1003: swfrench: Helmfile-only deployment to clean up unused mesh listeners synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:23 swfrench@deploy1003: Started scap sync-world: Helmfile-only deployment to clean up unused mesh listeners * 18:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93296 and previous config saved to /var/cache/conftool/dbconfig/20260527-181923-fceratto.json * 18:13 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 18:12 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 18:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 18:11 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 18:11 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 18:10 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 18:10 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 18:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93295 and previous config saved to /var/cache/conftool/dbconfig/20260527-180915-fceratto.json * 18:09 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 18:09 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293776{{!}}ProductionServices: Revert to discovery shellbox listeners]] (duration: 10m 24s) * 18:08 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1017.eqiad.wmnet * 18:08 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs1017.eqiad.wmnet * 18:07 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp5024.eqsin.wmnet * 18:03 swfrench@deploy1003: swfrench: Continuing with deployment * 18:02 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 18:02 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 18:02 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 18:01 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 18:01 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 18:01 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 18:01 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 18:00 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 18:00 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293776{{!}}ProductionServices: Revert to discovery shellbox listeners]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:00 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P93294 and previous config saved to /var/cache/conftool/dbconfig/20260527-175908-fceratto.json * 17:58 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293776{{!}}ProductionServices: Revert to discovery shellbox listeners]] * 17:55 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93293 and previous config saved to /var/cache/conftool/dbconfig/20260527-174900-fceratto.json * 17:43 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293774{{!}}ProductionServices: Temporarily use shellbox in codfw]] (duration: 15m 01s) * 17:38 swfrench@deploy1003: swfrench: Continuing with deployment * 17:31 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293774{{!}}ProductionServices: Temporarily use shellbox in codfw]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:28 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293774{{!}}ProductionServices: Temporarily use shellbox in codfw]] * 17:25 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp1114.eqiad.wmnet * 17:18 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:17 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:16 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:15 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:15 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:14 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:05 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293775{{!}}ProductionServices: Temporarily use shellbox in eqiad]] (duration: 08m 44s) * 17:00 swfrench@deploy1003: swfrench: Continuing with deployment * 16:58 swfrench@deploy1003: swfrench: Backport for [[gerrit:1293775{{!}}ProductionServices: Temporarily use shellbox in eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:56 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1293775{{!}}ProductionServices: Temporarily use shellbox in eqiad]] * 16:53 atsuko@dns1004: END - running authdns-update * 16:51 atsuko@dns1004: START - running authdns-update * 16:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1264 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93292 and previous config saved to /var/cache/conftool/dbconfig/20260527-164846-fceratto.json * 16:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1264.eqiad.wmnet with reason: Maintenance * 16:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93291 and previous config saved to /var/cache/conftool/dbconfig/20260527-164815-fceratto.json * 16:43 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp1112.eqiad.wmnet * 16:41 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1017.eqiad.wmnet with reason: Setting up * 16:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P93290 and previous config saved to /var/cache/conftool/dbconfig/20260527-163808-fceratto.json * 16:37 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2163: Repooling after testing patch * 16:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P93287 and previous config saved to /var/cache/conftool/dbconfig/20260527-162800-fceratto.json * 16:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93285 and previous config saved to /var/cache/conftool/dbconfig/20260527-161753-fceratto.json * 16:14 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply * 16:13 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply * 16:13 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply * 16:12 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply * 16:11 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93284 and previous config saved to /var/cache/conftool/dbconfig/20260527-161101-fceratto.json * 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1224.eqiad.wmnet with reason: Maintenance * 16:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93283 and previous config saved to /var/cache/conftool/dbconfig/20260527-161034-fceratto.json * 16:10 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply * 16:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1178: Recovering from failure in cookbook * 16:10 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply * 16:05 sukhe@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host durum5003.eqsin.wmnet with OS trixie * 16:03 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp6016.drmrs.wmnet * 16:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220', diff saved to https://phabricator.wikimedia.org/P93280 and previous config saved to /var/cache/conftool/dbconfig/20260527-160027-fceratto.json * 15:59 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1017.eqiad.wmnet * 15:53 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2163.codfw.wmnet * 15:53 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2163.codfw.wmnet * 15:52 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host lvs1017.eqiad.wmnet * 15:52 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Repooling after testing patch * 15:52 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp6016.drmrs.wmnet,cp[1112,1114].eqiad.wmnet,cp[5024,5031-5032].eqsin.wmnet<nowiki>}</nowiki> and A:cp * 15:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2163: Testing cookbook * 15:50 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2163: Testing cookbook * 15:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220', diff saved to https://phabricator.wikimedia.org/P93276 and previous config saved to /var/cache/conftool/dbconfig/20260527-155019-fceratto.json * 15:45 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:45 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93274 and previous config saved to /var/cache/conftool/dbconfig/20260527-154011-fceratto.json * 15:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 15:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2163: Migration of db2163.codfw.wmnet completed * 15:32 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Migration of db2163.codfw.wmnet completed * 15:32 cwilliams@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2163: Migration of db2163.codfw.wmnet completed * 15:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1178: Recovering from failure in cookbook * 15:22 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1178.eqiad.wmnet * 15:22 cwilliams@cumin1003: START - Cookbook sre.hosts.remove-downtime for db1178.eqiad.wmnet * 15:19 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 15:19 cdanis: 💙cdanis@cp4047.ulsfo.wmnet ~ 🕦☕ sudo apt install lua5.4-ciderbloom lua5.4-ciderbloom-dbgsym * 15:13 cdanis: 💙cdanis@cp5026.eqsin.wmnet ~ 🕚☕ sudo apt install lua5.4-ciderbloom lua5.4-ciderbloom-dbgsym * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:11 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:11 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:11 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Icinga wait failed during run * 15:10 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:10 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:10 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:09 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:09 cdanis: 💔cdanis@apt1002.wikimedia.org ~ 🕚☕ sudo -i reprepro --component main --restrict cidergrinder update trixie-wikimedia * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:08 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:05 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93268 and previous config saved to /var/cache/conftool/dbconfig/20260527-150508-fceratto.json * 15:05 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1220.eqiad.wmnet with reason: Maintenance * 15:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93267 and previous config saved to /var/cache/conftool/dbconfig/20260527-150438-fceratto.json * 14:59 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2163: Migration of db2163.codfw.wmnet completed * 14:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P93264 and previous config saved to /var/cache/conftool/dbconfig/20260527-145430-fceratto.json * 14:54 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 14:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2163.codfw.wmnet with OS trixie * 14:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply * 14:50 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply * 14:46 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290926{{!}}Re-enable ReadingLists QuickSurvey (T426781)]] (duration: 08m 32s) * 14:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1178.eqiad.wmnet with OS trixie * 14:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P93263 and previous config saved to /var/cache/conftool/dbconfig/20260527-144423-fceratto.json * 14:42 aude@deploy1003: aude: Continuing with deployment * 14:40 aude@deploy1003: aude: Backport for [[gerrit:1290926{{!}}Re-enable ReadingLists QuickSurvey (T426781)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:38 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db2189.codfw.wmnet with reason: crashed [[phab:T427376|T427376]] * 14:38 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1290926{{!}}Re-enable ReadingLists QuickSurvey (T426781)]] * 14:35 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290924{{!}}Make logging of title and page ID optional (T426457)]] (duration: 11m 30s) * 14:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93262 and previous config saved to /var/cache/conftool/dbconfig/20260527-143416-fceratto.json * 14:33 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2163.codfw.wmnet with reason: host reimage * 14:29 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2163.codfw.wmnet with reason: host reimage * 14:29 aude@deploy1003: aude: Continuing with deployment * 14:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1178.eqiad.wmnet with reason: host reimage * 14:27 aude@deploy1003: aude: Backport for [[gerrit:1290924{{!}}Make logging of title and page ID optional (T426457)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:27 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93260 and previous config saved to /var/cache/conftool/dbconfig/20260527-142659-fceratto.json * 14:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1179.eqiad.wmnet with reason: Maintenance * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:23 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1290924{{!}}Make logging of title and page ID optional (T426457)]] * 14:22 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1178.eqiad.wmnet with reason: host reimage * 14:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1033.eqiad.wmnet with reason: Maintenance * 14:18 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294247{{!}}Update Direct Reporting email (T427358)]] (duration: 33m 01s) * 14:10 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2163.codfw.wmnet with OS trixie * 14:09 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1178.eqiad.wmnet with OS trixie * 14:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2163: Upgrading db2163.codfw.wmnet * 14:08 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2163: Upgrading db2163.codfw.wmnet * 14:08 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1178: Upgrading db1178.eqiad.wmnet * 14:07 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1178: Upgrading db1178.eqiad.wmnet * 14:06 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 14:06 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:06 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:06 stran@deploy1003: stran: Continuing with deployment * 14:02 stran@deploy1003: stran: Backport for [[gerrit:1294247{{!}}Update Direct Reporting email (T427358)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:56 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 13:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2164: Migration of db2164.codfw.wmnet completed * 13:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 13:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1192: Migration of db1192.eqiad.wmnet completed * 13:45 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1294247{{!}}Update Direct Reporting email (T427358)]] * 13:40 phuedx@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294217{{!}}ext.wikimediaEvents: Add hoisting error detection test (T427092)]] (duration: 11m 35s) * 13:36 phuedx@deploy1003: phuedx: Continuing with deployment * 13:30 phuedx@deploy1003: phuedx: Backport for [[gerrit:1294217{{!}}ext.wikimediaEvents: Add hoisting error detection test (T427092)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:28 phuedx@deploy1003: Started scap sync-world: Backport for [[gerrit:1294217{{!}}ext.wikimediaEvents: Add hoisting error detection test (T427092)]] * 13:21 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290781{{!}}mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264{{!}}MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] (duration: 13m 23s) * 13:15 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2189: Test * 13:15 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2189: Test * 13:15 mlitn@deploy1003: krinkle, mlitn: Continuing with deployment * 13:13 mlitn@deploy1003: krinkle, mlitn: Backport for [[gerrit:1290781{{!}}mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264{{!}}MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 13:10 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2164: Migration of db2164.codfw.wmnet completed * 13:08 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1290781{{!}}mmv: Fix missing or stale arrow and counter controls (T426960)]], [[gerrit:1294264{{!}}MMV Carousel: Restore click-to-open for carousel thumbnails (T426225)]] * 13:06 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 13:05 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db2212.codfw.wmnet with reason: failed to reboot [[phab:T427388|T427388]] [[phab:T426633|T426633]] * 13:05 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1192: Migration of db1192.eqiad.wmnet completed * 13:01 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2164.codfw.wmnet with OS trixie * 12:57 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1192.eqiad.wmnet with OS trixie * 12:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2164.codfw.wmnet with reason: host reimage * 12:40 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1192.eqiad.wmnet with reason: host reimage * 12:40 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2164.codfw.wmnet with reason: host reimage * 12:35 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1192.eqiad.wmnet with reason: host reimage * 12:28 Amir1: deleting binlogs older than a year * 12:22 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2164.codfw.wmnet with OS trixie * 12:21 cmooney@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 36692 * 12:21 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1192.eqiad.wmnet with OS trixie * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1077 * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1080 * 12:20 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1077 * 12:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2164: Upgrading db2164.codfw.wmnet * 12:20 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 36692 * 12:20 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1080 * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1078 * 12:20 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1079 * 12:20 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2164: Upgrading db2164.codfw.wmnet * 12:19 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:19 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1079 * 12:19 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1078 * 12:19 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:19 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1192: Upgrading db1192.eqiad.wmnet * 12:19 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:18 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1192: Upgrading db1192.eqiad.wmnet * 12:18 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 12:15 jclark@cumin1003: START - Cookbook sre.dns.netbox * 12:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2165: Migration of db2165.codfw.wmnet completed * 12:14 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:14 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:14 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudvirt1078 to eqiad - jclark@cumin1003" * 12:12 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db2189: Test * 12:11 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db2189: Test * 12:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1193: Migration of db1193.eqiad.wmnet completed * 12:09 jclark@cumin1003: START - Cookbook sre.dns.netbox * 12:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2212 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93243 and previous config saved to /var/cache/conftool/dbconfig/20260527-120452-fceratto.json * 12:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: Maintenance * 12:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93242 and previous config saved to /var/cache/conftool/dbconfig/20260527-120205-fceratto.json * 12:01 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 11:58 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 11:58 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "is everything alright? /cc effie - ayounsi@cumin1003" * 11:58 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "is everything alright? /cc effie - ayounsi@cumin1003" * 11:56 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 11:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P93239 and previous config saved to /var/cache/conftool/dbconfig/20260527-115157-fceratto.json * 11:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P93237 and previous config saved to /var/cache/conftool/dbconfig/20260527-114149-fceratto.json * 11:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93235 and previous config saved to /var/cache/conftool/dbconfig/20260527-113142-fceratto.json * 11:29 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2165: Migration of db2165.codfw.wmnet completed * 11:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1193: Migration of db1193.eqiad.wmnet completed * 11:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2188 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93231 and previous config saved to /var/cache/conftool/dbconfig/20260527-112327-fceratto.json * 11:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2188.codfw.wmnet with reason: Maintenance * 11:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93230 and previous config saved to /var/cache/conftool/dbconfig/20260527-112257-fceratto.json * 11:19 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2165.codfw.wmnet with OS trixie * 11:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1193.eqiad.wmnet with OS trixie * 11:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P93229 and previous config saved to /var/cache/conftool/dbconfig/20260527-111250-fceratto.json * 11:10 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:10 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:08 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:08 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:02 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P93227 and previous config saved to /var/cache/conftool/dbconfig/20260527-110242-fceratto.json * 11:02 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 11:02 cmooney@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 11:01 cmooney@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 11:01 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2165.codfw.wmnet with reason: host reimage * 11:00 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2189', diff saved to https://phabricator.wikimedia.org/P93226 and previous config saved to /var/cache/conftool/dbconfig/20260527-110016-marostegui.json * 10:58 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1193.eqiad.wmnet with reason: host reimage * 10:57 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2165.codfw.wmnet with reason: host reimage * 10:56 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 10:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93225 and previous config saved to /var/cache/conftool/dbconfig/20260527-105235-fceratto.json * 10:52 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1193.eqiad.wmnet with reason: host reimage * 10:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1050: repool after maintenance * 10:45 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2176 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93223 and previous config saved to /var/cache/conftool/dbconfig/20260527-104518-fceratto.json * 10:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2176.codfw.wmnet with reason: Maintenance * 10:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93222 and previous config saved to /var/cache/conftool/dbconfig/20260527-104449-fceratto.json * 10:39 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2165.codfw.wmnet with OS trixie * 10:38 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1193.eqiad.wmnet with OS trixie * 10:36 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1193: Upgrading db1193.eqiad.wmnet * 10:35 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1193: Upgrading db1193.eqiad.wmnet * 10:35 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2165: Upgrading db2165.codfw.wmnet * 10:35 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2165: Upgrading db2165.codfw.wmnet * 10:34 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P93218 and previous config saved to /var/cache/conftool/dbconfig/20260527-103441-fceratto.json * 10:29 daniel@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:29 daniel@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P93217 and previous config saved to /var/cache/conftool/dbconfig/20260527-102434-fceratto.json * 10:22 daniel@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:21 daniel@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93215 and previous config saved to /var/cache/conftool/dbconfig/20260527-101426-fceratto.json * 10:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1203: Migration of db1203.eqiad.wmnet completed * 10:10 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2166: Migration of db2166.codfw.wmnet completed * 10:08 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2174 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93212 and previous config saved to /var/cache/conftool/dbconfig/20260527-100701-fceratto.json * 10:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance * 10:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93211 and previous config saved to /var/cache/conftool/dbconfig/20260527-100632-fceratto.json * 10:05 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1050: repool after maintenance * 10:04 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 10:02 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1050.eqiad.wmnet with OS trixie * 09:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P93208 and previous config saved to /var/cache/conftool/dbconfig/20260527-095624-fceratto.json * 09:47 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 09:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P93206 and previous config saved to /var/cache/conftool/dbconfig/20260527-094616-fceratto.json * 09:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1050.eqiad.wmnet with reason: host reimage * 09:43 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 09:41 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1050.eqiad.wmnet with reason: host reimage * 09:38 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 09:38 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 09:37 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 09:37 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 09:36 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 09:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93203 and previous config saved to /var/cache/conftool/dbconfig/20260527-093609-fceratto.json * 09:34 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 09:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2173 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93202 and previous config saved to /var/cache/conftool/dbconfig/20260527-092842-fceratto.json * 09:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance * 09:28 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1203: Migration of db1203.eqiad.wmnet completed * 09:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93200 and previous config saved to /var/cache/conftool/dbconfig/20260527-092814-fceratto.json * 09:27 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1050.eqiad.wmnet with OS trixie * 09:26 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1050: Upgrading es1050.eqiad.wmnet * 09:25 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1050: Upgrading es1050.eqiad.wmnet * 09:25 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1050: repool after maintenance * 09:25 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1050: repool after maintenance * 09:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2166: Migration of db2166.codfw.wmnet completed * 09:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2051: repool after maintenance * 09:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1203.eqiad.wmnet with OS trixie * 09:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P93196 and previous config saved to /var/cache/conftool/dbconfig/20260527-091806-fceratto.json * 09:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2166.codfw.wmnet with OS trixie * 09:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P93194 and previous config saved to /var/cache/conftool/dbconfig/20260527-090759-fceratto.json * 09:03 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp3074.* * 09:03 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp3066.* * 09:03 fabfur: repooling cp3074 and cp3066 ([[phab:T419825|T419825]]) * 09:02 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp6015.drmrs.wmnet * 09:02 slyngshede@cumin1003: START - Cookbook sre.hosts.remove-downtime for cp6015.drmrs.wmnet * 09:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1203.eqiad.wmnet with reason: host reimage * 09:02 slyngshede@cumin1003: conftool action : set/pooled=yes; selector: name=cp6015.* * 08:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2166.codfw.wmnet with reason: host reimage * 08:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93193 and previous config saved to /var/cache/conftool/dbconfig/20260527-085751-fceratto.json * 08:55 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1203.eqiad.wmnet with reason: host reimage * 08:54 Emperor: restart swift on ms-fe2011 [[phab:T360913|T360913]] * 08:54 jayme@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:54 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2166.codfw.wmnet with reason: host reimage * 08:54 jayme@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:53 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 08:52 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 08:51 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 08:51 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 08:51 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp3066.* * 08:51 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp3074.* * 08:51 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 08:50 fabfur: depooling and installing haproxy-awslc on cp3074 and cp3066 ([[phab:T419825|T419825]]) * 08:50 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2170 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93191 and previous config saved to /var/cache/conftool/dbconfig/20260527-085024-fceratto.json * 08:50 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance * 08:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93190 and previous config saved to /var/cache/conftool/dbconfig/20260527-085005-fceratto.json * 08:41 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1203.eqiad.wmnet with OS trixie * 08:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P93189 and previous config saved to /var/cache/conftool/dbconfig/20260527-083957-fceratto.json * 08:38 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2051: repool after maintenance * 08:37 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 08:36 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1203: Upgrading db1203.eqiad.wmnet * 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1004.wikimedia.org * 08:36 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1203: Upgrading db1203.eqiad.wmnet * 08:36 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:35 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2166.codfw.wmnet with OS trixie * 08:35 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2051.codfw.wmnet with OS trixie * 08:34 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2166: Upgrading db2166.codfw.wmnet * 08:33 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2166: Upgrading db2166.codfw.wmnet * 08:33 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader1004.wikimedia.org * 08:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org * 08:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P93185 and previous config saved to /var/cache/conftool/dbconfig/20260527-082950-fceratto.json * 08:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org * 08:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93184 and previous config saved to /var/cache/conftool/dbconfig/20260527-081942-fceratto.json * 08:18 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2051.codfw.wmnet with reason: host reimage * 08:15 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2051.codfw.wmnet with reason: host reimage * 08:11 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 08:11 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2153 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93183 and previous config saved to /var/cache/conftool/dbconfig/20260527-081112-fceratto.json * 08:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance * 08:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93182 and previous config saved to /var/cache/conftool/dbconfig/20260527-081054-fceratto.json * 08:07 jmm@dns1004: END - running authdns-update * 08:05 jmm@dns1004: START - running authdns-update * 08:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248', diff saved to https://phabricator.wikimedia.org/P93181 and previous config saved to /var/cache/conftool/dbconfig/20260527-080046-fceratto.json * 07:59 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2051.codfw.wmnet with OS trixie * 07:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248', diff saved to https://phabricator.wikimedia.org/P93180 and previous config saved to /var/cache/conftool/dbconfig/20260527-075039-fceratto.json * 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1026.eqiad.wmnet * 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:43 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1026.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:43 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1026.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2051: Upgrading es2051.codfw.wmnet * 07:42 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2051: Upgrading es2051.codfw.wmnet * 07:41 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93178 and previous config saved to /var/cache/conftool/dbconfig/20260527-074031-fceratto.json * 07:40 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1294125{{!}}Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126{{!}}Add script to demote ineligible members of restricted global groups (T425395)]] (duration: 06m 42s) * 07:36 mszwarc@deploy1003: mszwarc: Continuing with deployment * 07:35 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1294125{{!}}Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126{{!}}Add script to demote ineligible members of restricted global groups (T425395)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2248 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93177 and previous config saved to /var/cache/conftool/dbconfig/20260527-073504-fceratto.json * 07:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2248.codfw.wmnet with reason: Maintenance * 07:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93176 and previous config saved to /var/cache/conftool/dbconfig/20260527-073434-fceratto.json * 07:33 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1294125{{!}}Add script to demote ineligible members of restricted global groups (T425395)]], [[gerrit:1294126{{!}}Add script to demote ineligible members of restricted global groups (T425395)]] * 07:28 jmm@cumin2002: START - Cookbook sre.dns.netbox * 07:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P93175 and previous config saved to /var/cache/conftool/dbconfig/20260527-072426-fceratto.json * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) * 07:23 marostegui@cumin1003: Removing pc1014 from zarcillo [[phab:T427190|T427190]] * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1014.eqiad.wmnet * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:23 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 07:23 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 07:18 marostegui@cumin1003: START - Cookbook sre.dns.netbox * 07:15 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1026.eqiad.wmnet * 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1025.eqiad.wmnet * 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P93174 and previous config saved to /var/cache/conftool/dbconfig/20260527-071418-fceratto.json * 07:13 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1014.eqiad.wmnet * 07:13 marostegui@cumin1003: START - Cookbook sre.mysql.decommission * 07:13 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org * 07:07 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2055: repool after maintenance * 07:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org * 07:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1003.wikimedia.org * 07:06 jmm@cumin2002: START - Cookbook sre.dns.netbox * 07:06 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1190.eqiad.wmnet with reason: Maintenance on db1190 * 07:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93172 and previous config saved to /var/cache/conftool/dbconfig/20260527-070410-fceratto.json * 07:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host urldownloader1003.wikimedia.org * 06:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2247 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93171 and previous config saved to /var/cache/conftool/dbconfig/20260527-065545-fceratto.json * 06:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2247.codfw.wmnet with reason: Maintenance * 06:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93170 and previous config saved to /var/cache/conftool/dbconfig/20260527-065526-fceratto.json * 06:54 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1025.eqiad.wmnet * 06:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P93168 and previous config saved to /var/cache/conftool/dbconfig/20260527-064519-fceratto.json * 06:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P93166 and previous config saved to /var/cache/conftool/dbconfig/20260527-063511-fceratto.json * 06:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93165 and previous config saved to /var/cache/conftool/dbconfig/20260527-062503-fceratto.json * 06:22 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2055: repool after maintenance * 06:21 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 06:21 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2055.codfw.wmnet with OS trixie * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2246 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93163 and previous config saved to /var/cache/conftool/dbconfig/20260527-061643-fceratto.json * 06:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2246.codfw.wmnet with reason: Maintenance * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93162 and previous config saved to /var/cache/conftool/dbconfig/20260527-061613-fceratto.json * 06:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P93161 and previous config saved to /var/cache/conftool/dbconfig/20260527-060606-fceratto.json * 06:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2055.codfw.wmnet with reason: host reimage * 05:56 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2055.codfw.wmnet with reason: host reimage * 05:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P93160 and previous config saved to /var/cache/conftool/dbconfig/20260527-055558-fceratto.json * 05:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93159 and previous config saved to /var/cache/conftool/dbconfig/20260527-054550-fceratto.json * 05:41 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2055.codfw.wmnet with OS trixie * 05:40 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2055: Upgrading es2055.codfw.wmnet * 05:40 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2055: Upgrading es2055.codfw.wmnet * 05:40 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 05:38 moritzm: remove ganeti1026 from eqiad Ganeti cluster [[phab:T424680|T424680]] * 05:37 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2245 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93157 and previous config saved to /var/cache/conftool/dbconfig/20260527-053727-fceratto.json * 05:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: Maintenance * 05:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93156 and previous config saved to /var/cache/conftool/dbconfig/20260527-053708-fceratto.json * 05:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P93155 and previous config saved to /var/cache/conftool/dbconfig/20260527-052700-fceratto.json * 05:26 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1014 from dbctl [[phab:T427270|T427270]]', diff saved to https://phabricator.wikimedia.org/P93154 and previous config saved to /var/cache/conftool/dbconfig/20260527-052624-marostegui.json * 05:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P93153 and previous config saved to /var/cache/conftool/dbconfig/20260527-051653-fceratto.json * 05:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93152 and previous config saved to /var/cache/conftool/dbconfig/20260527-050645-fceratto.json * 04:58 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2237 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93151 and previous config saved to /var/cache/conftool/dbconfig/20260527-045827-fceratto.json * 04:58 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: Maintenance * 04:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93150 and previous config saved to /var/cache/conftool/dbconfig/20260527-045759-fceratto.json * 04:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P93149 and previous config saved to /var/cache/conftool/dbconfig/20260527-044751-fceratto.json * 04:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P93148 and previous config saved to /var/cache/conftool/dbconfig/20260527-043744-fceratto.json * 04:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93147 and previous config saved to /var/cache/conftool/dbconfig/20260527-042737-fceratto.json * 04:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2236 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93146 and previous config saved to /var/cache/conftool/dbconfig/20260527-041921-fceratto.json * 04:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2236.codfw.wmnet with reason: Maintenance * 04:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93145 and previous config saved to /var/cache/conftool/dbconfig/20260527-041852-fceratto.json * 04:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P93144 and previous config saved to /var/cache/conftool/dbconfig/20260527-040844-fceratto.json * 03:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P93143 and previous config saved to /var/cache/conftool/dbconfig/20260527-035836-fceratto.json * 03:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93142 and previous config saved to /var/cache/conftool/dbconfig/20260527-034828-fceratto.json * 03:40 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2219 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93141 and previous config saved to /var/cache/conftool/dbconfig/20260527-034008-fceratto.json * 03:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2219.codfw.wmnet with reason: Maintenance * 03:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93140 and previous config saved to /var/cache/conftool/dbconfig/20260527-033938-fceratto.json * 03:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P93139 and previous config saved to /var/cache/conftool/dbconfig/20260527-032931-fceratto.json * 03:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P93138 and previous config saved to /var/cache/conftool/dbconfig/20260527-031923-fceratto.json * 03:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93137 and previous config saved to /var/cache/conftool/dbconfig/20260527-030915-fceratto.json * 03:00 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2210 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93136 and previous config saved to /var/cache/conftool/dbconfig/20260527-030045-fceratto.json * 03:00 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2210.codfw.wmnet with reason: Maintenance * 03:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93135 and previous config saved to /var/cache/conftool/dbconfig/20260527-030016-fceratto.json * 02:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P93134 and previous config saved to /var/cache/conftool/dbconfig/20260527-025008-fceratto.json * 02:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P93133 and previous config saved to /var/cache/conftool/dbconfig/20260527-024000-fceratto.json * 02:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93132 and previous config saved to /var/cache/conftool/dbconfig/20260527-022953-fceratto.json * 02:21 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2206 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93131 and previous config saved to /var/cache/conftool/dbconfig/20260527-022133-fceratto.json * 02:21 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2206.codfw.wmnet with reason: Maintenance * 02:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93130 and previous config saved to /var/cache/conftool/dbconfig/20260527-022100-fceratto.json * 02:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P93129 and previous config saved to /var/cache/conftool/dbconfig/20260527-021053-fceratto.json * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 29s) * 02:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P93128 and previous config saved to /var/cache/conftool/dbconfig/20260527-020045-fceratto.json * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93127 and previous config saved to /var/cache/conftool/dbconfig/20260527-015037-fceratto.json * 01:42 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2179 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93126 and previous config saved to /var/cache/conftool/dbconfig/20260527-014204-fceratto.json * 01:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2179.codfw.wmnet with reason: Maintenance * 01:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93125 and previous config saved to /var/cache/conftool/dbconfig/20260527-014134-fceratto.json * 01:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P93124 and previous config saved to /var/cache/conftool/dbconfig/20260527-013126-fceratto.json * 01:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P93123 and previous config saved to /var/cache/conftool/dbconfig/20260527-012119-fceratto.json * 01:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93122 and previous config saved to /var/cache/conftool/dbconfig/20260527-011111-fceratto.json * 01:02 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2172 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93121 and previous config saved to /var/cache/conftool/dbconfig/20260527-010234-fceratto.json * 01:02 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance * 01:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93120 and previous config saved to /var/cache/conftool/dbconfig/20260527-010205-fceratto.json * 00:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P93119 and previous config saved to /var/cache/conftool/dbconfig/20260527-005157-fceratto.json * 00:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P93118 and previous config saved to /var/cache/conftool/dbconfig/20260527-004149-fceratto.json * 00:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93117 and previous config saved to /var/cache/conftool/dbconfig/20260527-003141-fceratto.json * 00:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2155 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93116 and previous config saved to /var/cache/conftool/dbconfig/20260527-002309-fceratto.json * 00:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance * 00:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93115 and previous config saved to /var/cache/conftool/dbconfig/20260527-002228-fceratto.json * 00:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P93114 and previous config saved to /var/cache/conftool/dbconfig/20260527-001220-fceratto.json * 00:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P93113 and previous config saved to /var/cache/conftool/dbconfig/20260527-000209-fceratto.json == 2026-05-26 == * 23:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93112 and previous config saved to /var/cache/conftool/dbconfig/20260526-235201-fceratto.json * 23:44 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93111 and previous config saved to /var/cache/conftool/dbconfig/20260526-234451-fceratto.json * 23:44 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance * 23:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93110 and previous config saved to /var/cache/conftool/dbconfig/20260526-234421-fceratto.json * 23:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P93109 and previous config saved to /var/cache/conftool/dbconfig/20260526-233414-fceratto.json * 23:27 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5026.* * 23:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P93108 and previous config saved to /var/cache/conftool/dbconfig/20260526-232406-fceratto.json * 23:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93107 and previous config saved to /var/cache/conftool/dbconfig/20260526-231358-fceratto.json * 23:07 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5026.* * 23:06 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93106 and previous config saved to /var/cache/conftool/dbconfig/20260526-230650-fceratto.json * 23:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance * 23:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93105 and previous config saved to /var/cache/conftool/dbconfig/20260526-230620-fceratto.json * 22:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P93104 and previous config saved to /var/cache/conftool/dbconfig/20260526-225612-fceratto.json * 22:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P93103 and previous config saved to /var/cache/conftool/dbconfig/20260526-224604-fceratto.json * 22:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93101 and previous config saved to /var/cache/conftool/dbconfig/20260526-223556-fceratto.json * 22:28 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93100 and previous config saved to /var/cache/conftool/dbconfig/20260526-222848-fceratto.json * 22:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance * 22:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93099 and previous config saved to /var/cache/conftool/dbconfig/20260526-222828-fceratto.json * 22:23 robh@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts cp6015.drmrs.wmnet * 22:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P93098 and previous config saved to /var/cache/conftool/dbconfig/20260526-221819-fceratto.json * 22:10 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1009.eqiad.wmnet with OS trixie * 22:08 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1008.eqiad.wmnet with OS trixie * 22:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P93097 and previous config saved to /var/cache/conftool/dbconfig/20260526-220811-fceratto.json * 22:04 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293701{{!}}MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] (duration: 09m 30s) * 22:03 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1009.eqiad.wmnet with reason: host reimage * 22:00 egardner@deploy1003: egardner, mfossati: Continuing with deployment * 21:59 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1008.eqiad.wmnet with reason: host reimage * 21:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93096 and previous config saved to /var/cache/conftool/dbconfig/20260526-215803-fceratto.json * 21:57 egardner@deploy1003: egardner, mfossati: Backport for [[gerrit:1293701{{!}}MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:56 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp6015.drmrs.wmnet * 21:56 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1010.eqiad.wmnet with OS trixie * 21:56 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp6015.drmrs.wmnet * 21:55 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1293701{{!}}MultimediaViewer: enable image carousel as a beta feature on testwiki (T426799)]] * 21:54 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1009.eqiad.wmnet with reason: host reimage * 21:51 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1008.eqiad.wmnet with reason: host reimage * 21:50 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93095 and previous config saved to /var/cache/conftool/dbconfig/20260526-215043-fceratto.json * 21:50 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance * 21:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93094 and previous config saved to /var/cache/conftool/dbconfig/20260526-215011-fceratto.json * 21:49 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1010.eqiad.wmnet with reason: host reimage * 21:47 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp6015.drmrs.wmnet * 21:44 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1009 * 21:44 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host relforge1009 * 21:43 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host relforge1009 * 21:43 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) relforge1009.eqiad.wmnet 120.48.64.10.in-addr.arpa 0.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:43 bking@cumin2002: START - Cookbook sre.dns.wipe-cache relforge1009.eqiad.wmnet 120.48.64.10.in-addr.arpa 0.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:43 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:42 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1009 - bking@cumin2002" * 21:42 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1010.eqiad.wmnet with reason: host reimage * 21:42 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1009 - bking@cumin2002" * 21:41 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1008 * 21:40 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host relforge1008 * 21:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P93093 and previous config saved to /var/cache/conftool/dbconfig/20260526-214003-fceratto.json * 21:36 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host relforge1008 * 21:36 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) relforge1008.eqiad.wmnet 100.32.64.10.in-addr.arpa 0.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:36 bking@cumin2002: START - Cookbook sre.dns.wipe-cache relforge1008.eqiad.wmnet 100.32.64.10.in-addr.arpa 0.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 21:36 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:36 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1008 - bking@cumin2002" * 21:36 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host relforge1008 - bking@cumin2002" * 21:35 bking@cumin2002: START - Cookbook sre.dns.netbox * 21:32 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host relforge1010 * 21:32 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1010 * 21:31 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1010.eqiad.wmnet with OS trixie * 21:31 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1009 * 21:30 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1009.eqiad.wmnet with OS trixie * 21:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P93092 and previous config saved to /var/cache/conftool/dbconfig/20260526-212955-fceratto.json * 21:29 bking@cumin2002: START - Cookbook sre.dns.netbox * 21:29 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host relforge1008 * 21:29 bking@cumin2002: START - Cookbook sre.hosts.reimage for host relforge1008.eqiad.wmnet with OS trixie * 21:27 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist "all.dblist - mediamoderation-continuous-scan.dblist - preinstall.dblist" extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` in tmux session - [[phab:T421688|T421688]] * 21:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93091 and previous config saved to /var/cache/conftool/dbconfig/20260526-211948-fceratto.json * 21:19 jhathaway: dmarc ingress test run mx-in1001 * 21:15 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_codfw and A:cp * 21:15 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2057.codfw.wmnet * 21:14 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_codfw and A:cp * 21:14 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2058.codfw.wmnet * 21:12 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2222 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93090 and previous config saved to /var/cache/conftool/dbconfig/20260526-211238-fceratto.json * 21:12 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance * 21:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93089 and previous config saved to /var/cache/conftool/dbconfig/20260526-211207-fceratto.json * 21:06 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 21:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P93088 and previous config saved to /var/cache/conftool/dbconfig/20260526-210159-fceratto.json * 20:55 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on phab2003.codfw.wmnet with reason: WIP * 20:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P93087 and previous config saved to /var/cache/conftool/dbconfig/20260526-205152-fceratto.json * 20:50 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:50 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 20:50 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 20:45 dzahn@cumin2002: START - Cookbook sre.dns.netbox * 20:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93086 and previous config saved to /var/cache/conftool/dbconfig/20260526-204143-fceratto.json * 20:38 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2055.codfw.wmnet * 20:34 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2221 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93085 and previous config saved to /var/cache/conftool/dbconfig/20260526-203430-fceratto.json * 20:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance * 20:34 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2056.codfw.wmnet * 20:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93084 and previous config saved to /var/cache/conftool/dbconfig/20260526-203357-fceratto.json * 20:32 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:32 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:32 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:31 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P93083 and previous config saved to /var/cache/conftool/dbconfig/20260526-202349-fceratto.json * 20:18 alexsanford@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293161{{!}}Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794{{!}}Re-enable ReadingLists survey on beta cluster (T426781)]] (duration: 09m 14s) * 20:14 alexsanford@deploy1003: alexsanford, aude: Continuing with deployment * 20:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P93082 and previous config saved to /var/cache/conftool/dbconfig/20260526-201341-fceratto.json * 20:11 alexsanford@deploy1003: alexsanford, aude: Backport for [[gerrit:1293161{{!}}Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794{{!}}Re-enable ReadingLists survey on beta cluster (T426781)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:09 alexsanford@deploy1003: Started scap sync-world: Backport for [[gerrit:1293161{{!}}Enforce 2FA requirements for phase 3 groups (T423120)]], [[gerrit:1293794{{!}}Re-enable ReadingLists survey on beta cluster (T426781)]] * 20:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93081 and previous config saved to /var/cache/conftool/dbconfig/20260526-200333-fceratto.json * 19:59 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2053.codfw.wmnet * 19:58 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2029.codfw.wmnet with OS trixie * 19:57 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2028.codfw.wmnet with OS trixie * 19:56 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2208 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93080 and previous config saved to /var/cache/conftool/dbconfig/20260526-195632-fceratto.json * 19:56 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance * 19:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93079 and previous config saved to /var/cache/conftool/dbconfig/20260526-195557-fceratto.json * 19:55 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2054.codfw.wmnet * 19:51 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:51 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:45 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P93078 and previous config saved to /var/cache/conftool/dbconfig/20260526-194549-fceratto.json * 19:45 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 19:44 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 19:43 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2029 * 19:43 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 * 19:43 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 * 19:43 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 19:40 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb2014.codfw.wmnet with OS trixie * 19:40 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:40 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb2013.codfw.wmnet with OS trixie * 19:40 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:39 brett@cumin2002: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 19:38 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host durum5003.eqsin.wmnet with OS trixie * 19:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P93077 and previous config saved to /var/cache/conftool/dbconfig/20260526-193541-fceratto.json * 19:35 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:35 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 19:30 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_IPs - dzahn@cumin2002" * 19:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93076 and previous config saved to /var/cache/conftool/dbconfig/20260526-192533-fceratto.json * 19:24 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:21 dzahn@cumin2002: START - Cookbook sre.dns.netbox * 19:20 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2051.codfw.wmnet * 19:19 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" * 19:19 brett@cumin2002: START - Cookbook sre.hosts.reimage for host durum5003.eqsin.wmnet with OS trixie * 19:18 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2182 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93075 and previous config saved to /var/cache/conftool/dbconfig/20260526-191818-fceratto.json * 19:18 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance * 19:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93074 and previous config saved to /var/cache/conftool/dbconfig/20260526-191748-fceratto.json * 19:16 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2052.codfw.wmnet * 19:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P93073 and previous config saved to /var/cache/conftool/dbconfig/20260526-190740-fceratto.json * 19:07 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb2014.codfw.wmnet with reason: host reimage * 19:03 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb2013.codfw.wmnet with reason: host reimage * 18:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1026.eqiad.wmnet * 18:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P93072 and previous config saved to /var/cache/conftool/dbconfig/20260526-185732-fceratto.json * 18:56 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2014.codfw.wmnet with reason: host reimage * 18:56 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2013.codfw.wmnet with reason: host reimage * 18:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93071 and previous config saved to /var/cache/conftool/dbconfig/20260526-184724-fceratto.json * 18:44 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb2014.codfw.wmnet with OS trixie * 18:43 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host rdb2013.codfw.wmnet with OS trixie * 18:41 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host rdb2014.codfw.wmnet with OS trixie * 18:41 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2049.codfw.wmnet * 18:40 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2168 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93070 and previous config saved to /var/cache/conftool/dbconfig/20260526-184009-fceratto.json * 18:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance * 18:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93069 and previous config saved to /var/cache/conftool/dbconfig/20260526-183939-fceratto.json * 18:37 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2050.codfw.wmnet * 18:30 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 18:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P93068 and previous config saved to /var/cache/conftool/dbconfig/20260526-182931-fceratto.json * 18:29 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:29 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_magru-v4 - dzahn@cumin2002" * 18:29 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: activate_gitlab-lb_magru-v4 - dzahn@cumin2002" * 18:24 dzahn@cumin2002: START - Cookbook sre.dns.netbox * 18:21 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 18:21 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 18:21 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 18:20 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 18:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P93066 and previous config saved to /var/cache/conftool/dbconfig/20260526-181923-fceratto.json * 18:15 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 18:15 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 18:15 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 18:15 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 18:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93065 and previous config saved to /var/cache/conftool/dbconfig/20260526-180915-fceratto.json * 18:02 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2159 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93064 and previous config saved to /var/cache/conftool/dbconfig/20260526-180205-fceratto.json * 18:01 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2159.codfw.wmnet with reason: Maintenance * 18:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93063 and previous config saved to /var/cache/conftool/dbconfig/20260526-180132-fceratto.json * 18:00 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2047.codfw.wmnet * 17:59 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2048.codfw.wmnet * 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:54 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:54 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:54 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P93062 and previous config saved to /var/cache/conftool/dbconfig/20260526-175124-fceratto.json * 17:42 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293779{{!}}Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] (duration: 07m 25s) * 17:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P93060 and previous config saved to /var/cache/conftool/dbconfig/20260526-174117-fceratto.json * 17:39 mvernon@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ms-be2089.codfw.wmnet * 17:37 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 17:37 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:36 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:36 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:36 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1293779{{!}}Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:36 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:34 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1293779{{!}}Enable hCaptcha for VisualEditor and MobileFrontend for group0 (T425940)]] * 17:33 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:33 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:33 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:33 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:32 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93059 and previous config saved to /var/cache/conftool/dbconfig/20260526-173109-fceratto.json * 17:27 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-wdqs1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:26 jclark@cumin1003: START - Cookbook sre.hosts.provision for host dse-k8s-wdqs1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:25 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:25 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:25 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:24 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:24 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1001 to eqiad - jclark@cumin1003" * 17:24 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:24 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1001 to eqiad - jclark@cumin1003" * 17:23 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2227 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93058 and previous config saved to /var/cache/conftool/dbconfig/20260526-172332-fceratto.json * 17:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2227.codfw.wmnet with reason: Maintenance * 17:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93057 and previous config saved to /var/cache/conftool/dbconfig/20260526-172303-fceratto.json * 17:21 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2045.codfw.wmnet * 17:20 jclark@cumin1003: START - Cookbook sre.dns.netbox * 17:20 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2046.codfw.wmnet * 17:18 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:17 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:17 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:17 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:17 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:17 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:17 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:17 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:16 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:16 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:15 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 17:14 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:14 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:14 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:14 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:13 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:13 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:13 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:13 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:13 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:12 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P93056 and previous config saved to /var/cache/conftool/dbconfig/20260526-171255-fceratto.json * 17:11 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:11 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:11 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:11 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:09 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:07 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 17:05 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:05 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:05 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:05 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P93055 and previous config saved to /var/cache/conftool/dbconfig/20260526-170247-fceratto.json * 17:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:00 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:57 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:55 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:52 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93054 and previous config saved to /var/cache/conftool/dbconfig/20260526-165240-fceratto.json * 16:50 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:50 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:50 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:50 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:45 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:45 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 16:45 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:45 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:45 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:44 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:44 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2209 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93053 and previous config saved to /var/cache/conftool/dbconfig/20260526-164421-fceratto.json * 16:44 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:44 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1002 to eqiad - jclark@cumin1003" * 16:44 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2209.codfw.wmnet with reason: Maintenance * 16:44 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1002 to eqiad - jclark@cumin1003" * 16:43 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93052 and previous config saved to /var/cache/conftool/dbconfig/20260526-164352-fceratto.json * 16:42 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2043.codfw.wmnet * 16:41 brett@cumin2002: cookbooks.sre.cdn.roll-reboot finished rebooting cp2044.codfw.wmnet * 16:40 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:40 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:40 jclark@cumin1003: START - Cookbook sre.dns.netbox * 16:40 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:40 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:40 brett: reboot lvs 101[345].eqiad.wmnet * 16:39 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:39 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:39 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:39 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:37 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:37 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. * 16:37 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:37 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:37 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:37 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. * 16:37 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 16:36 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 16:36 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:36 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:36 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:36 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:35 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 16:34 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 16:34 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:34 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:34 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:34 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:33 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_codfw and A:cp * 16:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P93051 and previous config saved to /var/cache/conftool/dbconfig/20260526-163344-fceratto.json * 16:33 brett@cumin2002: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_codfw and A:cp * 16:31 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:31 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:30 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:30 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P93050 and previous config saved to /var/cache/conftool/dbconfig/20260526-162336-fceratto.json * 16:13 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2089.codfw.wmnet * 16:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93049 and previous config saved to /var/cache/conftool/dbconfig/20260526-161328-fceratto.json * 16:11 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:11 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:10 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:10 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:07 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search,name=eqiad * 16:06 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:06 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:06 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:06 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2194 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93047 and previous config saved to /var/cache/conftool/dbconfig/20260526-160450-fceratto.json * 16:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2194.codfw.wmnet with reason: Maintenance * 16:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93046 and previous config saved to /var/cache/conftool/dbconfig/20260526-160420-fceratto.json * 16:03 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:03 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:03 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] (duration: 00m 28s) * 16:02 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] * 16:00 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:00 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:00 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:00 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:57 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:55 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:55 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:55 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:55 aokoth@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] (duration: 00m 22s) * 15:55 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:55 aokoth@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2003 - [[phab:T423727|T423727]] * 15:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P93045 and previous config saved to /var/cache/conftool/dbconfig/20260526-155413-fceratto.json * 15:46 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search,name=eqiad * 15:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P93044 and previous config saved to /var/cache/conftool/dbconfig/20260526-154405-fceratto.json * 15:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93043 and previous config saved to /var/cache/conftool/dbconfig/20260526-153357-fceratto.json * 15:30 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:30 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:30 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:30 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:29 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:28 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2190 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93042 and previous config saved to /var/cache/conftool/dbconfig/20260526-152629-fceratto.json * 15:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2190.codfw.wmnet with reason: Maintenance * 15:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93041 and previous config saved to /var/cache/conftool/dbconfig/20260526-152559-fceratto.json * 15:24 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:24 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:24 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:24 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:23 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:22 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:22 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:22 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P93040 and previous config saved to /var/cache/conftool/dbconfig/20260526-151552-fceratto.json * 15:12 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2196: Rack maintenance completed * 15:10 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2196.codfw.wmnet * 15:10 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2196.codfw.wmnet * 15:07 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw * 15:06 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2222: Rack maintenance completed * 15:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P93037 and previous config saved to /var/cache/conftool/dbconfig/20260526-150546-fceratto.json * 15:04 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2221: Rack maintenance completed * 15:04 brennen@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab1004 for [[phab:T427286|T427286]] (duration: 00m 39s) * 15:03 brennen@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab1004 for [[phab:T427286|T427286]] * 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@939557b]: deploy phab2002 for [[phab:T427286|T427286]] (duration: 00m 45s) * 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@939557b]: deploy phab2002 for [[phab:T427286|T427286]] * 15:02 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2002.codfw.wmnet with reason: Phabricator deploy * 15:01 bjensen: uploading prometheus-memcached-exporter_0.16.0-1_amd64 on apt1002 * 15:01 jelto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator deploy * 15:00 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2223: switch maintenance * 14:56 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2196: Rack maintenance completed * 14:55 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2221.codfw.wmnet * 14:55 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2221.codfw.wmnet * 14:55 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2222.codfw.wmnet * 14:55 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2222.codfw.wmnet * 14:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93033 and previous config saved to /var/cache/conftool/dbconfig/20260526-145538-fceratto.json * 14:55 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet * 14:54 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1026.eqiad.wmnet * 14:52 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet * 14:52 moritzm: remove ganeti1025 from eqiad Ganeti cluster [[phab:T424680|T424680]] * 14:51 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2030.codfw.wmnet to cluster codfw and group A * 14:51 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2222: Rack maintenance completed * 14:49 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:49 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2221: Rack maintenance completed * 14:49 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:49 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2030.codfw.wmnet to cluster codfw and group A * 14:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2029.codfw.wmnet to cluster codfw and group A * 14:47 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2029.codfw.wmnet to cluster codfw and group A * 14:47 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2177 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93030 and previous config saved to /var/cache/conftool/dbconfig/20260526-144718-fceratto.json * 14:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance * 14:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93029 and previous config saved to /var/cache/conftool/dbconfig/20260526-144651-fceratto.json * 14:45 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=wdqs-scholarly,name=codfw * 14:45 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=wdqs-scholarly,name=codfw * 14:43 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw * 14:40 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:40 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2167: Migration of db2167.codfw.wmnet completed * 14:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P93026 and previous config saved to /var/cache/conftool/dbconfig/20260526-143643-fceratto.json * 14:31 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1054.eqiad.wmnet with OS trixie * 14:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P93023 and previous config saved to /var/cache/conftool/dbconfig/20260526-142636-fceratto.json * 14:26 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:25 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:24 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1014: Rack maintenance completed * 14:24 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.parsercache (exit_code=99) * 14:24 fceratto@cumin1003: START - Cookbook sre.mysql.parsercache * 14:24 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool pc1014: Rack maintenance completed * 14:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1025.eqiad.wmnet * 14:19 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for backup2015.codfw.wmnet,db2197.codfw.wmnet * 14:19 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for backup2015.codfw.wmnet,db2197.codfw.wmnet * 14:18 jynus: restarting mediabackups@codfw after maintenance on a codfw backup media storage server [[phab:T426199|T426199]] * 14:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93021 and previous config saved to /var/cache/conftool/dbconfig/20260526-141628-fceratto.json * 14:16 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:14 fabfur: repooled cp2043 ([[phab:T426199|T426199]]) * 14:14 ayounsi@cumin1003: START - Cookbook sre.mysql.pool pool db2223: switch maintenance * 14:14 blake@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1054.eqiad.wmnet with reason: host reimage * 14:14 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2043.* * 14:13 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293710{{!}}Site info should output thumblimits as array (T427066)]] (duration: 06m 40s) * 14:12 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:10 blake@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1054.eqiad.wmnet with reason: host reimage * 14:10 fabfur@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2011.codfw.wmnet * 14:10 fabfur@cumin1003: START - Cookbook sre.hosts.remove-downtime for lvs2011.codfw.wmnet * 14:09 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 14:09 fabfur: restoring lvs2011 as primary ([[phab:T426199|T426199]]) * 14:08 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1293710{{!}}Site info should output thumblimits as array (T427066)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:08 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 14:08 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 14:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2156 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93017 and previous config saved to /var/cache/conftool/dbconfig/20260526-140748-fceratto.json * 14:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance * 14:07 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93016 and previous config saved to /var/cache/conftool/dbconfig/20260526-140718-fceratto.json * 14:07 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1293710{{!}}Site info should output thumblimits as array (T427066)]] * 14:05 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.decommission (exit_code=99) * 14:05 marostegui@cumin1003: Removing pc1013 from zarcillo [[phab:T427190|T427190]] * 14:04 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1013.eqiad.wmnet * 14:04 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:04 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 14:04 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 14:00 marostegui@cumin1003: START - Cookbook sre.dns.netbox * 13:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P93014 and previous config saved to /var/cache/conftool/dbconfig/20260526-135711-fceratto.json * 13:56 blake@cumin1003: START - Cookbook sre.hosts.reimage for host mc1054.eqiad.wmnet with OS trixie * 13:55 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2167: Migration of db2167.codfw.wmnet completed * 13:53 Amir1: drop flaggedrevs tables on cawikinews ([[phab:T423577|T423577]]) * 13:49 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1013.eqiad.wmnet * 13:49 marostegui@cumin1003: START - Cookbook sre.mysql.decommission * 13:48 Lucas_WMDE: UTC afternoon backport+config window done * 13:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P93012 and previous config saved to /var/cache/conftool/dbconfig/20260526-134703-fceratto.json * 13:46 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2167.codfw.wmnet with OS trixie * 13:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93011 and previous config saved to /var/cache/conftool/dbconfig/20260526-133656-fceratto.json * 13:36 XioNoX: reboot lsw1-a2-codfw for software upgrade - [[phab:T426199|T426199]] * 13:36 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2223: switch maintenance * 13:35 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2223: switch maintenance * 13:35 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2222: switch maintenance * 13:35 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2222: switch maintenance * 13:35 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2221: switch maintenance * 13:35 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293662{{!}}Enable IRS Direct Reporting on testwiki (T425025)]] (duration: 09m 28s) * 13:34 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2221: switch maintenance * 13:34 ayounsi@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2196: switch maintenance * 13:34 ayounsi@cumin1003: START - Cookbook sre.mysql.depool depool db2196: switch maintenance * 13:31 ayounsi@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 13:30 stran@deploy1003: stran: Continuing with deployment * 13:29 ayounsi@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2003.codfw.wmnet,wikikube-worker[2248-2250].codfw.wmnet * 13:29 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2238 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93006 and previous config saved to /var/cache/conftool/dbconfig/20260526-132927-fceratto.json * 13:29 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2167.codfw.wmnet with reason: host reimage * 13:29 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance * 13:29 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 34 hosts with reason: Switch maintenance * 13:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93005 and previous config saved to /var/cache/conftool/dbconfig/20260526-132857-fceratto.json * 13:28 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lsw1-a2-codfw,lsw1-a2-codfw IPv6,lsw1-a2-codfw.mgmt with reason: Switch maintenance * 13:27 stran@deploy1003: stran: Backport for [[gerrit:1293662{{!}}Enable IRS Direct Reporting on testwiki (T425025)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:25 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1293662{{!}}Enable IRS Direct Reporting on testwiki (T425025)]] * 13:25 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2167.codfw.wmnet with reason: host reimage * 13:22 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293706{{!}}Disable the `no` language code for translation (T424613)]] (duration: 08m 30s) * 13:22 ladsgroup@dns1004: END - running authdns-update * 13:20 ladsgroup@dns1004: START - running authdns-update * 13:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P93004 and previous config saved to /var/cache/conftool/dbconfig/20260526-131850-fceratto.json * 13:18 lucaswerkmeister-wmde@deploy1003: jhsoby, lucaswerkmeister-wmde: Continuing with deployment * 13:16 lucaswerkmeister-wmde@deploy1003: jhsoby, lucaswerkmeister-wmde: Backport for [[gerrit:1293706{{!}}Disable the `no` language code for translation (T424613)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:14 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1293706{{!}}Disable the `no` language code for translation (T424613)]] * 13:12 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293177{{!}}Instrumentation: log new articles namespace and source (T422146)]] (duration: 07m 09s) * 13:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P93003 and previous config saved to /var/cache/conftool/dbconfig/20260526-130842-fceratto.json * 13:08 sbisson@deploy1003: sbisson: Continuing with deployment * 13:07 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2167.codfw.wmnet with OS trixie * 13:07 sbisson@deploy1003: sbisson: Backport for [[gerrit:1293177{{!}}Instrumentation: log new articles namespace and source (T422146)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:05 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2167: Upgrading db2167.codfw.wmnet * 13:05 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1293177{{!}}Instrumentation: log new articles namespace and source (T422146)]] * 13:04 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2167: Upgrading db2167.codfw.wmnet * 13:04 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:04 kart_: Update Recommendation API to 2026-05-26-074931-production * 13:03 kartik@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 13:00 topranks: deactivate CR BGP to doh2002 to test backup path via doh2001 * 12:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P93000 and previous config saved to /var/cache/conftool/dbconfig/20260526-125834-fceratto.json * 12:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2226 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92999 and previous config saved to /var/cache/conftool/dbconfig/20260526-125135-fceratto.json * 12:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance * 12:51 kartik@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 12:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92998 and previous config saved to /var/cache/conftool/dbconfig/20260526-125105-fceratto.json * 12:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P92997 and previous config saved to /var/cache/conftool/dbconfig/20260526-124059-fceratto.json * 12:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2003.wikimedia.org * 12:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1214: Migration of db1214.eqiad.wmnet completed * 12:33 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc2003.wikimedia.org * 12:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P92995 and previous config saved to /var/cache/conftool/dbconfig/20260526-123052-fceratto.json * 12:26 fabfur: depooled cp204 for network activity ([[phab:T426199|T426199]]) * 12:26 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2043.* * 12:24 ayounsi@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-a1-codfw,ssw1-a1-codfw IPv6,ssw1-a1-codfw.mgmt with reason: Switch maintenance * 12:24 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mirror1001.wikimedia.org * 12:23 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:23 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:22 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92993 and previous config saved to /var/cache/conftool/dbconfig/20260526-122044-fceratto.json * 12:20 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:19 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host mirror1001.wikimedia.org * 12:13 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2225 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92991 and previous config saved to /var/cache/conftool/dbconfig/20260526-121336-fceratto.json * 12:13 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance * 12:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92990 and previous config saved to /var/cache/conftool/dbconfig/20260526-121306-fceratto.json * 12:09 fabfur@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2011.codfw.wmnet with reason: Planned downtime for rack maintenance * 12:08 fabfur: downtime, disable puppet and stop pybal for rack maintenance ([[phab:T426199|T426199]]) * 12:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 12:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2181: Migration of db2181.codfw.wmnet completed * 12:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P92987 and previous config saved to /var/cache/conftool/dbconfig/20260526-120258-fceratto.json * 12:01 XioNoX: start ssw1-a1-codfw network maintenance (no impact expected as the spines are redundant) * 11:59 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293167{{!}}hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055{{!}}hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] (duration: 15m 26s) * 11:56 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on backup2015.codfw.wmnet,db2197.codfw.wmnet with reason: network maintenance * 11:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1005.eqiad.wmnet * 11:55 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment * 11:54 jynus: stopping mediabackups@codfw for maintenance on a codfw backup media storage server [[phab:T426199|T426199]] * 11:54 jmm@dns1004: END - running authdns-update * 11:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P92985 and previous config saved to /var/cache/conftool/dbconfig/20260526-115251-fceratto.json * 11:52 jmm@dns1004: START - running authdns-update * 11:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1005.eqiad.wmnet * 11:49 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1214: Migration of db1214.eqiad.wmnet completed * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1004.eqiad.wmnet * 11:47 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet * 11:46 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1293167{{!}}hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055{{!}}hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:45 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1004.eqiad.wmnet * 11:44 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1293167{{!}}hCaptcha: Complete rollout to all wikis (group2 + cleanup) (T425354)]], [[gerrit:1290055{{!}}hCaptcha: Exempt CommunityRequests pages from edit/create triggers (T426897)]] * 11:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92983 and previous config saved to /var/cache/conftool/dbconfig/20260526-114243-fceratto.json * 11:42 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet * 11:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1214.eqiad.wmnet with OS trixie * 11:35 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293691{{!}}Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] (duration: 06m 46s) * 11:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92981 and previous config saved to /var/cache/conftool/dbconfig/20260526-113542-fceratto.json * 11:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance * 11:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92980 and previous config saved to /var/cache/conftool/dbconfig/20260526-113521-fceratto.json * 11:31 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 11:31 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1293691{{!}}Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1222: Migration of db1222.eqiad.wmnet completed * 11:29 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1293691{{!}}Fix path to wikibase.wikiprojects.tracking.js (T421856 T427252)]] * 11:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P92978 and previous config saved to /var/cache/conftool/dbconfig/20260526-112513-fceratto.json * 11:24 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1214.eqiad.wmnet with reason: host reimage * 11:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc4 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92977 and previous config saved to /var/cache/conftool/dbconfig/20260526-112326-marostegui.json * 11:22 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2181: Migration of db2181.codfw.wmnet completed * 11:22 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1024 to dbctl [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92975 and previous config saved to /var/cache/conftool/dbconfig/20260526-112215-marostegui.json * 11:20 fceratto@cumin1003: dbctl commit (dc=all): 'Switchover es2042 es2041 for [[phab:T426199|T426199]]', diff saved to https://phabricator.wikimedia.org/P92974 and previous config saved to /var/cache/conftool/dbconfig/20260526-112028-fceratto.json * 11:17 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1214.eqiad.wmnet with reason: host reimage * 11:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P92972 and previous config saved to /var/cache/conftool/dbconfig/20260526-111506-fceratto.json * 11:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2181.codfw.wmnet with OS trixie * 11:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92971 and previous config saved to /var/cache/conftool/dbconfig/20260526-110458-fceratto.json * 11:02 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1214.eqiad.wmnet with OS trixie * 11:00 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293095{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] (duration: 15m 50s) * 11:00 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1214: Upgrading db1214.eqiad.wmnet * 10:59 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1214: Upgrading db1214.eqiad.wmnet * 10:59 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2189 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92968 and previous config saved to /var/cache/conftool/dbconfig/20260526-105755-fceratto.json * 10:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance * 10:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92967 and previous config saved to /var/cache/conftool/dbconfig/20260526-105726-fceratto.json * 10:56 jiji@deploy1003: jiji: Continuing with deployment * 10:55 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2181.codfw.wmnet with reason: host reimage * 10:51 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2181.codfw.wmnet with reason: host reimage * 10:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P92966 and previous config saved to /var/cache/conftool/dbconfig/20260526-104718-fceratto.json * 10:46 jiji@deploy1003: jiji: Backport for [[gerrit:1293095{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:44 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1293095{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6382 (T418261 T419976)]] * 10:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P92964 and previous config saved to /var/cache/conftool/dbconfig/20260526-103711-fceratto.json * 10:36 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2181.codfw.wmnet with OS trixie * 10:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply * 10:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply * 10:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92963 and previous config saved to /var/cache/conftool/dbconfig/20260526-102703-fceratto.json * 10:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1226: Migration of db1226.eqiad.wmnet completed * 10:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2181: Upgrading db2181.codfw.wmnet * 10:24 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2181: Upgrading db2181.codfw.wmnet * 10:24 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 10:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2175 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92960 and previous config saved to /var/cache/conftool/dbconfig/20260526-101936-fceratto.json * 10:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance * 10:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92959 and previous config saved to /var/cache/conftool/dbconfig/20260526-101842-fceratto.json * 10:16 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-codfw@codfw * 10:16 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 10:15 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 10:10 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293668{{!}}hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] (duration: 06m 42s) * 10:09 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-codfw@codfw * 10:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92957 and previous config saved to /var/cache/conftool/dbconfig/20260526-100834-fceratto.json * 10:06 kharlan@deploy1003: kharlan: Continuing with deployment * 10:05 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293668{{!}}hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:03 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293668{{!}}hCaptcha: Avoid URL.searchParams in Grade C bundle (T422222)]] * 10:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 10:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2195: Migration of db2195.codfw.wmnet completed * 10:01 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>kubestage200*<nowiki>}</nowiki> and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw) * 10:01 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet * 10:01 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet * 10:00 jmm@cumin2002: END (PASS) - Cookbook sre.netbox.restart-reboot (exit_code=0) rolling reboot on A:netbox * 09:58 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 09:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P92955 and previous config saved to /var/cache/conftool/dbconfig/20260526-095827-fceratto.json * 09:58 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 09:58 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 09:57 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 09:56 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-eqiad@eqiad * 09:56 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:55 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 09:55 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 09:55 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:55 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet * 09:54 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet * 09:54 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet * 09:54 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet * 09:53 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>kubestage100*<nowiki>}</nowiki> and (A:wikikube-staging-master-eqiad or A:wikikube-staging-worker-eqiad) * 09:53 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet * 09:53 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet * 09:52 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-eqiad@eqiad * 09:52 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293665{{!}}hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] (duration: 08m 07s) * 09:51 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2043.* * 09:51 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp2044.* * 09:48 fabfur: repooling cp2043 and cp2044 (haproxy-awslc) ([[phab:T419825|T419825]]) * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92953 and previous config saved to /var/cache/conftool/dbconfig/20260526-094819-fceratto.json * 09:47 kharlan@deploy1003: kharlan: Continuing with deployment * 09:46 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet * 09:45 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293665{{!}}hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:44 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs3009.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:44 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293665{{!}}hCaptcha: Avoid `for (const ... of ...)` in Grade C bundle (T422222)]] * 09:41 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet * 09:41 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet * 09:41 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet * 09:41 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2229 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92951 and previous config saved to /var/cache/conftool/dbconfig/20260526-094115-fceratto.json * 09:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2229.codfw.wmnet with reason: Maintenance * 09:41 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs3009.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92950 and previous config saved to /var/cache/conftool/dbconfig/20260526-094045-fceratto.json * 09:40 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1226: Migration of db1226.eqiad.wmnet completed * 09:39 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: aux-master-codfw@codfw * 09:39 elukey@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:38 elukey@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:34 fabfur: depooling cp2044 to install haproxy-awslc ([[phab:T419825|T419825]]) * 09:34 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet * 09:34 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet * 09:34 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2044.* * 09:33 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet * 09:33 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet * 09:33 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet * 09:33 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp2043.* * 09:32 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293661{{!}}hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] (duration: 06m 52s) * 09:32 fabfur: depooling cp2043 to install haproxy-awslc ([[phab:T419825|T419825]]) * 09:31 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1226.eqiad.wmnet with OS trixie * 09:30 elukey@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: aux-master-codfw@codfw * 09:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P92947 and previous config saved to /var/cache/conftool/dbconfig/20260526-093031-fceratto.json * 09:29 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet * 09:29 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet * 09:29 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet * 09:28 kharlan@deploy1003: kharlan: Continuing with deployment * 09:28 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs3008.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:28 kharlan@deploy1003: kharlan: Backport for [[gerrit:1293661{{!}}hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:27 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet * 09:26 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet * 09:26 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet * 09:26 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet * 09:26 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1293661{{!}}hCaptcha: Ship a self-contained Grade C captcha bundle (T422222)]] * 09:25 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs3008.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:25 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs3010.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet * 09:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet * 09:21 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs3010.esams.wmnet<nowiki>}</nowiki> and A:liberica * 09:20 fabfur: start rebooting esams liberica instances ([[phab:T426563|T426563]]) * 09:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P92946 and previous config saved to /var/cache/conftool/dbconfig/20260526-092024-fceratto.json * 09:20 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet * 09:16 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2195: Migration of db2195.codfw.wmnet completed * 09:15 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet * 09:14 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1226.eqiad.wmnet with reason: host reimage * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>kubestage100*<nowiki>}</nowiki> and (A:wikikube-staging-master-eqiad or A:wikikube-staging-worker-eqiad) * 09:14 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>kubestage200*<nowiki>}</nowiki> and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw) * 09:14 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293658{{!}}Fix TypeError in Mandatory2FAChecker (T427251)]] (duration: 06m 47s) * 09:10 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1226.eqiad.wmnet with reason: host reimage * 09:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92944 and previous config saved to /var/cache/conftool/dbconfig/20260526-091016-fceratto.json * 09:09 mszwarc@deploy1003: mszwarc: Continuing with deployment * 09:09 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1293658{{!}}Fix TypeError in Mandatory2FAChecker (T427251)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:07 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2195.codfw.wmnet with OS trixie * 09:07 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293658{{!}}Fix TypeError in Mandatory2FAChecker (T427251)]] * 09:06 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs4009.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 09:03 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2224 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92943 and previous config saved to /var/cache/conftool/dbconfig/20260526-090315-fceratto.json * 09:03 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2224.codfw.wmnet with reason: Maintenance * 09:03 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs4009.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92942 and previous config saved to /var/cache/conftool/dbconfig/20260526-090256-fceratto.json * 08:57 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs4008.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 08:56 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox.discovery.wmnet. on all recursors * 08:56 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors * 08:55 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1226.eqiad.wmnet with OS trixie * 08:53 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs4008.ulsfo.wmnet<nowiki>}</nowiki> and A:liberica * 08:53 fabfur: start rebooting ulsfo liberica instances ([[phab:T426563|T426563]]) * 08:53 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293594{{!}}Allow to remove passkeys when there's only one standard 2FA method (T426872)]] (duration: 07m 23s) * 08:53 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs5005.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:53 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1226: Upgrading db1226.eqiad.wmnet * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P92941 and previous config saved to /var/cache/conftool/dbconfig/20260526-085248-fceratto.json * 08:51 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox.discovery.wmnet. on all recursors * 08:51 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors * 08:51 jmm@cumin2002: START - Cookbook sre.netbox.restart-reboot rolling reboot on A:netbox * 08:50 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1226: Upgrading db1226.eqiad.wmnet * 08:50 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs5005.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:50 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2195.codfw.wmnet with reason: host reimage * 08:49 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1222: Migration of db1222.eqiad.wmnet completed * 08:48 mszwarc@deploy1003: mszwarc: Continuing with deployment * 08:47 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1293594{{!}}Allow to remove passkeys when there's only one standard 2FA method (T426872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:46 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293594{{!}}Allow to remove passkeys when there's only one standard 2FA method (T426872)]] * 08:43 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs5004.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet * 08:43 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2195.codfw.wmnet with reason: host reimage * 08:43 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1292032{{!}}Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964{{!}}hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] (duration: 09m 56s) * 08:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P92939 and previous config saved to /var/cache/conftool/dbconfig/20260526-084240-fceratto.json * 08:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1222.eqiad.wmnet with OS trixie * 08:40 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs5004.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:40 fabfur: start rebooting eqsin liberica instances ([[phab:T426563|T426563]]) * 08:39 kartik@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 08:39 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet * 08:39 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 08:39 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs5006.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:35 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs5006.eqsin.wmnet<nowiki>}</nowiki> and A:liberica * 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1024.eqiad.wmnet * 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1024.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 08:35 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1292032{{!}}Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964{{!}}hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:33 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs6002.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:33 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1292032{{!}}Grant globalblock-local-status to groups with globalblock-whitelist (T277942)]], [[gerrit:1290964{{!}}hCaptcha CommonSettings.php: Don't define sitekeys as config vars]] * 08:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92938 and previous config saved to /var/cache/conftool/dbconfig/20260526-083233-fceratto.json * 08:30 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs6002.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:25 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2217 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92937 and previous config saved to /var/cache/conftool/dbconfig/20260526-082531-fceratto.json * 08:25 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2217.codfw.wmnet with reason: Maintenance * 08:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92936 and previous config saved to /var/cache/conftool/dbconfig/20260526-082458-fceratto.json * 08:23 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2195.codfw.wmnet with OS trixie * 08:23 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage * 08:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2195: Upgrading db2195.codfw.wmnet * 08:20 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2195: Upgrading db2195.codfw.wmnet * 08:19 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 08:18 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage * 08:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P92934 and previous config saved to /var/cache/conftool/dbconfig/20260526-081451-fceratto.json * 08:13 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs6001.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 08:10 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs6001.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:09 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1024.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 08:04 jmm@cumin2002: START - Cookbook sre.dns.netbox * 08:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P92932 and previous config saved to /var/cache/conftool/dbconfig/20260526-080443-fceratto.json * 08:01 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1222.eqiad.wmnet with OS trixie * 08:00 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs6003.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 08:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet * 07:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet * 07:59 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1024.eqiad.wmnet * 07:59 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1023.eqiad.wmnet * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1023.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:59 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 07:59 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 07:58 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1023.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" * 07:56 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs6003.drmrs.wmnet<nowiki>}</nowiki> and A:liberica * 07:56 fabfur: start rebooting drmrs liberica instances ([[phab:T426563|T426563]]) * 07:56 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs7002.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:54 jmm@cumin2002: START - Cookbook sre.dns.netbox * 07:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92931 and previous config saved to /var/cache/conftool/dbconfig/20260526-075435-fceratto.json * 07:52 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs7002.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:51 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1047.eqiad.wmnet * 07:51 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:51 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1047.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:49 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1023.eqiad.wmnet * 07:47 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2193 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92930 and previous config saved to /var/cache/conftool/dbconfig/20260526-074739-fceratto.json * 07:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2193.codfw.wmnet with reason: Maintenance * 07:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92929 and previous config saved to /var/cache/conftool/dbconfig/20260526-074710-fceratto.json * 07:46 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet * 07:45 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet * 07:45 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:45 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs7001.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:44 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1025.eqiad.wmnet * 07:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:43 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:43 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 07:41 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs7001.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:40 fabfur@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1046.eqiad.wmnet * 07:40 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1046.eqiad.wmnet * 07:38 arthurtaylor@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291951{{!}}Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] (duration: 12m 01s) * 07:38 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1047.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P92928 and previous config saved to /var/cache/conftool/dbconfig/20260526-073702-fceratto.json * 07:37 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Upgrading db1222.eqiad.wmnet * 07:36 fabfur@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Upgrading db1222.eqiad.wmnet * 07:36 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance * 07:35 fabfur: start rebooting magru liberica instances ([[phab:T426563|T426563]]) * 07:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92926 and previous config saved to /var/cache/conftool/dbconfig/20260526-073459-fceratto.json * 07:32 arthurtaylor@deploy1003: arthurtaylor: Continuing with deployment * 07:31 arthurtaylor@deploy1003: arthurtaylor: Backport for [[gerrit:1291951{{!}}Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:30 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1046.eqiad.wmnet * 07:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20260526-072643-fceratto.json * 07:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1046.eqiad.wmnet * 07:26 arthurtaylor@deploy1003: Started scap sync-world: Backport for [[gerrit:1291951{{!}}Enable and configure WikiProjects prototype on Test Wikidata (T424329)]] * 07:25 jiji@cumin1003: START - Cookbook sre.dns.netbox * 07:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P92924 and previous config saved to /var/cache/conftool/dbconfig/20260526-072452-fceratto.json * 07:24 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet * 07:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1047.eqiad.wmnet * 07:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1047.eqiad.wmnet * 07:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92923 and previous config saved to /var/cache/conftool/dbconfig/20260526-071635-fceratto.json * 07:15 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet * 07:15 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1026.eqiad.wmnet * 07:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P92922 and previous config saved to /var/cache/conftool/dbconfig/20260526-071444-fceratto.json * 07:13 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1026.eqiad.wmnet * 07:11 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1025.eqiad.wmnet * 07:10 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1025.eqiad.wmnet * 07:09 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2180 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92921 and previous config saved to /var/cache/conftool/dbconfig/20260526-070946-fceratto.json * 07:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2180.codfw.wmnet with reason: Maintenance * 07:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92920 and previous config saved to /var/cache/conftool/dbconfig/20260526-070916-fceratto.json * 07:09 moritzm: failover Ganeti master in eqiad to ganeti1048 * 07:09 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1047.eqiad.wmnet * 07:07 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1046.eqiad.wmnet * 07:07 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:06 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1046.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org * 07:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92919 and previous config saved to /var/cache/conftool/dbconfig/20260526-070436-fceratto.json * 07:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet * 07:04 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1046.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 07:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet * 07:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org * 06:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P92918 and previous config saved to /var/cache/conftool/dbconfig/20260526-065909-fceratto.json * 06:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org * 06:58 jiji@cumin1003: START - Cookbook sre.dns.netbox * 06:58 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet * 06:55 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet * 06:53 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1046.eqiad.wmnet * 06:53 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1045.eqiad.wmnet * 06:53 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:53 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1045.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 06:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org * 06:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P92917 and previous config saved to /var/cache/conftool/dbconfig/20260526-064901-fceratto.json * 06:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T419635|T419635]])', diff saved to https://phabricator.wikimedia.org/P92916 and previous config saved to /var/cache/conftool/dbconfig/20260526-064833-fceratto.json * 06:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1222.eqiad.wmnet with reason: Maintenance * 06:47 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1222: Switchover * 06:41 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92914 and previous config saved to /var/cache/conftool/dbconfig/20260526-063853-fceratto.json * 06:35 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 06:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2169 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92912 and previous config saved to /var/cache/conftool/dbconfig/20260526-063155-fceratto.json * 06:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance * 06:28 fceratto@cumin1003: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance * 06:23 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1222: Switchover * 06:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db1222 [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92910 and previous config saved to /var/cache/conftool/dbconfig/20260526-061656-fceratto.json * 06:15 fceratto@dns1005: END - running authdns-update * 06:14 fceratto@dns1005: START - running authdns-update * 06:11 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db1162 to s2 primary and set section read-write [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92909 and previous config saved to /var/cache/conftool/dbconfig/20260526-061114-fceratto.json * 06:10 fceratto@cumin1003: dbctl commit (dc=all): 'Set s2 eqiad as read-only for maintenance - [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92908 and previous config saved to /var/cache/conftool/dbconfig/20260526-061021-fceratto.json * 06:10 federico3: Starting s2 eqiad failover from db1222 to db1162 - [[phab:T425622|T425622]] * 06:04 fceratto@cumin1003: dbctl commit (dc=all): 'Set db1162 with weight 0 [[phab:T425622|T425622]]', diff saved to https://phabricator.wikimedia.org/P92907 and previous config saved to /var/cache/conftool/dbconfig/20260526-060443-fceratto.json * 06:04 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s2 [[phab:T425622|T425622]] * 06:02 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:02 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 06:01 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) * 06:00 fceratto@cumin1003: START - Cookbook sre.mysql.global-read-only * 05:15 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1014.eqiad.wmnet: Maintenance on pc4 * 05:15 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 05:15 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 05:15 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1014.eqiad.wmnet: Maintenance on pc4 * 05:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2024.codfw.wmnet,pc[1014,1024].eqiad.wmnet with reason: Maintenance on pc4 * 04:37 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 04:34 pt1979@cumin2002: START - Cookbook sre.dns.netbox * 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.1 (duration: 02m 32s) * 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] (duration: 36m 24s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.4 refs [[phab:T423913|T423913]] * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 20s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-25 == * 21:00 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1045.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:49 jiji@cumin1003: START - Cookbook sre.dns.netbox * 20:38 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1045.eqiad.wmnet * 20:37 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1044.eqiad.wmnet * 20:37 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:37 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1044.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:25 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1044.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 20:15 moritzm: truncate krb5kdc.log1 (which made log rotation fail) * 20:06 jiji@cumin1003: START - Cookbook sre.dns.netbox * 19:57 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1044.eqiad.wmnet * 19:25 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1043.eqiad.wmnet * 19:25 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 19:25 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1043.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 19:22 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1043.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 18:49 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_eqiad * 18:49 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1115.eqiad.wmnet * 18:34 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5023.eqsin.wmnet [reason: manually pooling after reboot as icinga was down] * 18:33 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5030.eqsin.wmnet [reason: manually pooling after reboot as icinga was down] * 18:22 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp5030*<nowiki>}</nowiki> and A:cp * 18:22 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5030.eqsin.wmnet * 18:15 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp5023*<nowiki>}</nowiki> and A:cp * 18:15 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5023.eqsin.wmnet * 18:10 jiji@cumin1003: START - Cookbook sre.dns.netbox * 18:10 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp5030*<nowiki>}</nowiki> and A:cp * 18:09 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp1113*<nowiki>}</nowiki> and A:cp * 18:09 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet * 18:09 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet * 18:03 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp1113*<nowiki>}</nowiki> and A:cp * 18:02 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp5023*<nowiki>}</nowiki> and A:cp * 18:01 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-text_eqiad * 18:01 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-upload_eqsin * 18:01 sukhe: sre.cdn.roll-reboot cookbooks stalled due to icinga reboot * 18:00 sukhe@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-text_eqsin * 17:35 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1043.eqiad.wmnet * 17:31 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1110.eqiad.wmnet [reason: manually pooling after reboot as icinga was down] * 17:30 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1042.eqiad.wmnet * 17:30 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:30 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1042.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:29 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1111.eqiad.wmnet * 17:28 sukhe: sukhe@alert1002:~$ sudo systemctl restart icinga.service * 17:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92903 and previous config saved to /var/cache/conftool/dbconfig/20260525-171310-fceratto.json * 17:11 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1042.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 17:06 jiji@cumin1003: START - Cookbook sre.dns.netbox * 17:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P92902 and previous config saved to /var/cache/conftool/dbconfig/20260525-170302-fceratto.json * 16:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P92901 and previous config saved to /var/cache/conftool/dbconfig/20260525-165255-fceratto.json * 16:51 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1042.eqiad.wmnet * 16:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92900 and previous config saved to /var/cache/conftool/dbconfig/20260525-164247-fceratto.json * 16:42 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1041.eqiad.wmnet * 16:42 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:42 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1041.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:41 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1041.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:40 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5021.eqsin.wmnet * 16:39 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5029.eqsin.wmnet * 16:36 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2158 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92899 and previous config saved to /var/cache/conftool/dbconfig/20260525-163559-fceratto.json * 16:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2158.codfw.wmnet with reason: Maintenance * 16:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92898 and previous config saved to /var/cache/conftool/dbconfig/20260525-163512-fceratto.json * 16:34 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1108.eqiad.wmnet * 16:30 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1109.eqiad.wmnet * 16:26 jiji@cumin1003: START - Cookbook sre.dns.netbox * 16:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249', diff saved to https://phabricator.wikimedia.org/P92897 and previous config saved to /var/cache/conftool/dbconfig/20260525-162505-fceratto.json * 16:20 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1041.eqiad.wmnet * 16:20 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1040.eqiad.wmnet * 16:20 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:20 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1040.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:16 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1040.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 16:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249', diff saved to https://phabricator.wikimedia.org/P92896 and previous config saved to /var/cache/conftool/dbconfig/20260525-161457-fceratto.json * 16:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92895 and previous config saved to /var/cache/conftool/dbconfig/20260525-160450-fceratto.json * 16:02 jiji@cumin1003: START - Cookbook sre.dns.netbox * 15:59 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2249 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92894 and previous config saved to /var/cache/conftool/dbconfig/20260525-155930-fceratto.json * 15:59 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2249.codfw.wmnet with reason: Maintenance * 15:57 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5020.eqsin.wmnet * 15:57 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5028.eqsin.wmnet * 15:52 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1106.eqiad.wmnet * 15:51 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1107.eqiad.wmnet * 15:29 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1040.eqiad.wmnet * 15:29 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1039.eqiad.wmnet * 15:29 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:29 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1039.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:27 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1039.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 15:17 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1013 from dbctl [[phab:T427190|T427190]]', diff saved to https://phabricator.wikimedia.org/P92893 and previous config saved to /var/cache/conftool/dbconfig/20260525-151718-marostegui.json * 15:15 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5019.eqsin.wmnet * 15:15 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5027.eqsin.wmnet * 15:12 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1104.eqiad.wmnet * 15:11 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1105.eqiad.wmnet * 15:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92892 and previous config saved to /var/cache/conftool/dbconfig/20260525-150309-fceratto.json * 14:53 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P92891 and previous config saved to /var/cache/conftool/dbconfig/20260525-145301-fceratto.json * 14:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P92890 and previous config saved to /var/cache/conftool/dbconfig/20260525-144253-fceratto.json * 14:33 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1102.eqiad.wmnet * 14:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92889 and previous config saved to /var/cache/conftool/dbconfig/20260525-143246-fceratto.json * 14:32 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5026.eqsin.wmnet * 14:32 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5018.eqsin.wmnet * 14:31 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1103.eqiad.wmnet * 14:25 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2228 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92888 and previous config saved to /var/cache/conftool/dbconfig/20260525-142551-fceratto.json * 14:25 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance * 14:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92887 and previous config saved to /var/cache/conftool/dbconfig/20260525-142520-fceratto.json * 14:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P92885 and previous config saved to /var/cache/conftool/dbconfig/20260525-141513-fceratto.json * 14:12 jiji@cumin1003: START - Cookbook sre.dns.netbox * 14:06 sukhe: curl localhost:9090/pools/inference-staging-grpc_30051 shows ml-staging200[1-3].codfw.wmnet as enabled and pooled: [[phab:T424049|T424049]] * 14:05 sukhe: sukhe@lvs2013:~$ sudo systemctl restart pybal.service: [[phab:T424049|T424049]] * 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P92884 and previous config saved to /var/cache/conftool/dbconfig/20260525-140505-fceratto.json * 14:03 sukhe: sudo cumin 'A:lvs and A:lvs-low-traffic-codfw' 'run-puppet-agent --enable "adding new ml-serve (grpc) [[phab:T424049|T424049]]"' * 14:02 sukhe: sukhe@lvs2014:~$ sudo systemctl restart pybal.service": [[phab:T424049|T424049]] * 14:02 sukhe: sukhe@lvs2014:~$ sudo systemctl restart pybal.service * 14:00 sukhe: sudo cumin 'A:lvs and A:lvs-secondary-codfw' 'run-puppet-agent --enable "adding new ml-serve (grpc) [[phab:T424049|T424049]]"' * 13:59 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1039.eqiad.wmnet * 13:58 sukhe: sudo cumin 'A:lvs and A:eqiad' 'run-puppet-agent --enable "adding new ml-serve (grpc) [[phab:T424049|T424049]]": NOOP change, since service is codfw only * 13:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92882 and previous config saved to /var/cache/conftool/dbconfig/20260525-135458-fceratto.json * 13:52 Msz2001: Everything deployed, UTC afternoon config+backport window done * 13:52 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293119{{!}}Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] (duration: 09m 43s) * 13:51 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1101.eqiad.wmnet * 13:51 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp1100.eqiad.wmnet * 13:50 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5025.eqsin.wmnet * 13:50 sukhe@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5017.eqsin.wmnet * 13:49 kart_: Updated Recommendation API to 2026-05-21-044522-production * 13:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2223 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92881 and previous config saved to /var/cache/conftool/dbconfig/20260525-134807-fceratto.json * 13:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2223.codfw.wmnet with reason: Maintenance * 13:47 mszwarc@deploy1003: vadymts1, mszwarc: Continuing with deployment * 13:47 kartik@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 13:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92880 and previous config saved to /var/cache/conftool/dbconfig/20260525-134737-fceratto.json * 13:45 mszwarc@deploy1003: vadymts1, mszwarc: Backport for [[gerrit:1293119{{!}}Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:45 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1162: Reboot * 13:43 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293119{{!}}Set $wgAutoconfirmCount to 25 on plwiktionary (T427177)]] * 13:40 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqiad * 13:39 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqiad * 13:38 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290813{{!}}Article Guidance: enable experiment on phase 2 wikis (T426871)]] (duration: 08m 14s) * 13:38 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqsin * 13:38 sukhe@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqsin * 13:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P92878 and previous config saved to /var/cache/conftool/dbconfig/20260525-133729-fceratto.json * 13:34 sbisson@deploy1003: sbisson: Continuing with deployment * 13:33 kartik@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 13:32 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1038.eqiad.wmnet * 13:32 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:32 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1038.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 13:31 sbisson@deploy1003: sbisson: Backport for [[gerrit:1290813{{!}}Article Guidance: enable experiment on phase 2 wikis (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:30 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1290813{{!}}Article Guidance: enable experiment on phase 2 wikis (T426871)]] * 13:27 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1293094{{!}}Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953{{!}}Update logo, wordmark and tagline for zghwiki (T426406)]] (duration: 07m 43s) * 13:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P92876 and previous config saved to /var/cache/conftool/dbconfig/20260525-132722-fceratto.json * 13:23 mszwarc@deploy1003: mszwarc, jhsoby: Continuing with deployment * 13:21 mszwarc@deploy1003: mszwarc, jhsoby: Backport for [[gerrit:1293094{{!}}Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953{{!}}Update logo, wordmark and tagline for zghwiki (T426406)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:20 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1038.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 13:20 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1293094{{!}}Update plwikimedia logo to monochrome, following on-wiki change (T427193)]], [[gerrit:1290953{{!}}Update logo, wordmark and tagline for zghwiki (T426406)]] * 13:19 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1291966{{!}}Modify various configurations for English Wikibooks (T426992)]] (duration: 15m 53s) * 13:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92875 and previous config saved to /var/cache/conftool/dbconfig/20260525-131714-fceratto.json * 13:12 mszwarc@deploy1003: vadymts1, mszwarc: Continuing with deployment * 13:12 jiji@cumin1003: START - Cookbook sre.dns.netbox * 13:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2211 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92873 and previous config saved to /var/cache/conftool/dbconfig/20260525-131023-fceratto.json * 13:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2211.codfw.wmnet with reason: Maintenance * 13:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92872 and previous config saved to /var/cache/conftool/dbconfig/20260525-130950-fceratto.json * 13:07 mszwarc@deploy1003: vadymts1, mszwarc: Backport for [[gerrit:1291966{{!}}Modify various configurations for English Wikibooks (T426992)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:03 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1291966{{!}}Modify various configurations for English Wikibooks (T426992)]] * 12:59 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1162: Reboot * 12:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P92870 and previous config saved to /var/cache/conftool/dbconfig/20260525-125942-fceratto.json * 12:59 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1162: Reboot * 12:59 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1162: Reboot * 12:58 kart_: Updated cxserver to 2026-05-24-103047-production ([[phab:T426808|T426808]], [[phab:T373418|T373418]]) * 12:56 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply * 12:56 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply * 12:54 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db1162: Reboot * 12:54 fceratto@cumin1003: START - Cookbook sre.mysql.depool depool db1162: Reboot * 12:54 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply * 12:53 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply * 12:49 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1162.eqiad.wmnet with reason: Reboot * 12:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P92868 and previous config saved to /var/cache/conftool/dbconfig/20260525-124934-fceratto.json * 12:40 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply * 12:39 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply * 12:39 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1038.eqiad.wmnet * 12:39 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92867 and previous config saved to /var/cache/conftool/dbconfig/20260525-123927-fceratto.json * 12:32 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2192 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92866 and previous config saved to /var/cache/conftool/dbconfig/20260525-123239-fceratto.json * 12:32 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2192.codfw.wmnet with reason: Maintenance * 12:32 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92865 and previous config saved to /var/cache/conftool/dbconfig/20260525-123208-fceratto.json * 12:22 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P92864 and previous config saved to /var/cache/conftool/dbconfig/20260525-122201-fceratto.json * 12:17 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1037.eqiad.wmnet * 12:17 jiji@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:17 jiji@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1037.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 12:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P92863 and previous config saved to /var/cache/conftool/dbconfig/20260525-121153-fceratto.json * 12:10 jiji@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1037.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" * 12:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92862 and previous config saved to /var/cache/conftool/dbconfig/20260525-120145-fceratto.json * 11:58 jiji@cumin1003: START - Cookbook sre.dns.netbox * 11:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2178 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92861 and previous config saved to /var/cache/conftool/dbconfig/20260525-115504-fceratto.json * 11:54 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2178.codfw.wmnet with reason: Maintenance * 11:54 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92860 and previous config saved to /var/cache/conftool/dbconfig/20260525-115434-fceratto.json * 11:44 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P92859 and previous config saved to /var/cache/conftool/dbconfig/20260525-114426-fceratto.json * 11:43 jiji@cumin1003: START - Cookbook sre.hosts.decommission for hosts mc1037.eqiad.wmnet * 11:34 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P92858 and previous config saved to /var/cache/conftool/dbconfig/20260525-113419-fceratto.json * 11:28 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2160.codfw.wmnet with OS trixie * 11:24 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92857 and previous config saved to /var/cache/conftool/dbconfig/20260525-112411-fceratto.json * 11:17 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2171 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92856 and previous config saved to /var/cache/conftool/dbconfig/20260525-111717-fceratto.json * 11:17 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: Maintenance * 11:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92855 and previous config saved to /var/cache/conftool/dbconfig/20260525-111648-fceratto.json * 11:06 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P92854 and previous config saved to /var/cache/conftool/dbconfig/20260525-110640-fceratto.json * 11:05 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2160.codfw.wmnet with reason: host reimage * 11:00 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2160.codfw.wmnet with reason: host reimage * 10:58 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 10:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 10:57 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 10:56 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:56 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P92853 and previous config saved to /var/cache/conftool/dbconfig/20260525-105633-fceratto.json * 10:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92852 and previous config saved to /var/cache/conftool/dbconfig/20260525-104625-fceratto.json * 10:43 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2160.codfw.wmnet with OS trixie * 10:41 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc3 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92851 and previous config saved to /var/cache/conftool/dbconfig/20260525-104141-marostegui.json * 10:40 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1023 to pc3 as master [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92850 and previous config saved to /var/cache/conftool/dbconfig/20260525-104055-marostegui.json * 10:40 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1023 to dbctl', diff saved to https://phabricator.wikimedia.org/P92849 and previous config saved to /var/cache/conftool/dbconfig/20260525-104027-marostegui.json * 10:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2157 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92848 and previous config saved to /var/cache/conftool/dbconfig/20260525-103944-fceratto.json * 10:39 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance * 10:31 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply * 10:30 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply * 10:27 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 10:18 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 10:16 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1011.eqiad.wmnet * 10:08 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1011.eqiad.wmnet * 10:08 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1007.eqiad.wmnet * 09:59 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1007.eqiad.wmnet * 09:59 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1006.eqiad.wmnet * 09:57 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:49 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1006.eqiad.wmnet * 09:48 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:46 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:45 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:40 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:40 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:28 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:17 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:13 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92847 and previous config saved to /var/cache/conftool/dbconfig/20260525-091302-fceratto.json * 09:12 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 09:02 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231', diff saved to https://phabricator.wikimedia.org/P92846 and previous config saved to /var/cache/conftool/dbconfig/20260525-090255-fceratto.json * 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231', diff saved to https://phabricator.wikimedia.org/P92845 and previous config saved to /var/cache/conftool/dbconfig/20260525-085247-fceratto.json * 08:42 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92844 and previous config saved to /var/cache/conftool/dbconfig/20260525-084239-fceratto.json * 08:35 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2231 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92843 and previous config saved to /var/cache/conftool/dbconfig/20260525-083540-fceratto.json * 08:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2231.codfw.wmnet with reason: Maintenance * 08:35 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92842 and previous config saved to /var/cache/conftool/dbconfig/20260525-083511-fceratto.json * 08:25 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215', diff saved to https://phabricator.wikimedia.org/P92841 and previous config saved to /var/cache/conftool/dbconfig/20260525-082504-fceratto.json * 08:14 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215', diff saved to https://phabricator.wikimedia.org/P92840 and previous config saved to /var/cache/conftool/dbconfig/20260525-081456-fceratto.json * 08:04 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92839 and previous config saved to /var/cache/conftool/dbconfig/20260525-080448-fceratto.json * 07:57 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2215 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92838 and previous config saved to /var/cache/conftool/dbconfig/20260525-075739-fceratto.json * 07:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2215.codfw.wmnet with reason: Maintenance * 07:57 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92837 and previous config saved to /var/cache/conftool/dbconfig/20260525-075708-fceratto.json * 07:47 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196', diff saved to https://phabricator.wikimedia.org/P92836 and previous config saved to /var/cache/conftool/dbconfig/20260525-074700-fceratto.json * 07:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196', diff saved to https://phabricator.wikimedia.org/P92835 and previous config saved to /var/cache/conftool/dbconfig/20260525-073653-fceratto.json * 07:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92834 and previous config saved to /var/cache/conftool/dbconfig/20260525-072645-fceratto.json * 07:19 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2196 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92833 and previous config saved to /var/cache/conftool/dbconfig/20260525-071953-fceratto.json * 07:19 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2196.codfw.wmnet with reason: Maintenance * 07:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92832 and previous config saved to /var/cache/conftool/dbconfig/20260525-071924-fceratto.json * 07:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186', diff saved to https://phabricator.wikimedia.org/P92831 and previous config saved to /var/cache/conftool/dbconfig/20260525-070917-fceratto.json * 07:03 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2233.codfw.wmnet with OS trixie * 06:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186', diff saved to https://phabricator.wikimedia.org/P92830 and previous config saved to /var/cache/conftool/dbconfig/20260525-065909-fceratto.json * 06:49 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92829 and previous config saved to /var/cache/conftool/dbconfig/20260525-064902-fceratto.json * 06:43 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2186 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92828 and previous config saved to /var/cache/conftool/dbconfig/20260525-064305-fceratto.json * 06:42 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance * 06:40 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2233.codfw.wmnet with reason: host reimage * 06:35 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2233.codfw.wmnet with reason: host reimage * 06:19 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2233.codfw.wmnet with OS trixie * 06:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2233.codfw.wmnet with reason: Reimage to Trixie * 06:17 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) * 06:17 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade * 06:15 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2160.codfw.wmnet with reason: Reboot upgrade m2 * 06:15 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2233.codfw.wmnet with reason: Reboot upgrade m2 * 06:08 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy1027.eqiad.wmnet with reason: Reboot * 05:18 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2023.codfw.wmnet,pc[1013,1023].eqiad.wmnet with reason: Maintenance on pc3 * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1013.eqiad.wmnet: Maintenance on pc3 * 05:17 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 05:17 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache * 05:17 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1013.eqiad.wmnet: Maintenance on pc3 * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 43s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-24 == * 19:08 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down * 02:06 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 23s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-23 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 35s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-05-22 == * 23:39 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 23:39 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 23:39 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 23:39 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 23:38 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 23:37 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 23:37 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 23:37 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 22:20 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 22:12 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 22:11 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 20:29 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T426585|T426585]] - bking@cumin2002 * 20:28 inflatador: bking@deploy1003 set eqiad prod cirrus `node_concurrent_recoveries` up to 7 from 4 [[phab:T426585|T426585]] * 20:27 inflatador: bking@deploy1003 set codfw prod cirrus `node_concurrent_recoveries` back down to 4 from 7 [[phab:T426585|T426585]] * 18:39 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 17:34 topranks: enable ttl protection on esams CRs IBGP session * 17:28 topranks: enable ttl protection on ulsfo CRs IBGP session * 16:50 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 16:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 16:16 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 16:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 16:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 15:58 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 15:14 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 15:02 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 15:02 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 14:34 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudnet2008-dev.codfw.wmnet * 14:34 andrew@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:34 andrew@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2008-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 14:33 andrew@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2008-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 14:33 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb[1020,1022-1025].eqiad.wmnet * 14:29 andrew@cumin2002: START - Cookbook sre.dns.netbox * 14:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply * 14:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply * 14:23 andrew@cumin2002: START - Cookbook sre.hosts.decommission for hosts cloudnet2008-dev.codfw.wmnet * 14:23 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudnet2007-dev.codfw.wmnet * 14:23 andrew@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:23 andrew@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2007-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 14:03 andrew@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet2007-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin2002" * 13:59 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb[1020,1022-1025].eqiad.wmnet * 13:58 andrew@cumin2002: START - Cookbook sre.dns.netbox * 13:53 andrew@cumin2002: START - Cookbook sre.hosts.decommission for hosts cloudnet2007-dev.codfw.wmnet * 13:52 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1018.eqiad.wmnet * 13:50 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 13:50 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 13:46 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1018.eqiad.wmnet * 13:25 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for clouddb1018.eqiad.wmnet * 13:25 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1018.eqiad.wmnet * 13:25 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for 6 hosts * 13:16 inflatador: bking@deploy1002 set search_codfw cluster recovery settings from 4 to 7 [[phab:T426560|T426560]] * 13:15 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for 6 hosts * 13:15 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 13:11 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp5017.eqsin.wmnet<nowiki>}</nowiki> and A:cp * 13:11 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp5017.eqsin.wmnet * 13:10 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1017.eqiad.wmnet * 13:09 elukey: uploaded spicerack_12.6.0 to apt.wikimedia.org bookworm-wikimedia * 13:08 fnegri@cumin1003: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for clouddb1017.eqiad.wmnet * 12:59 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp5017.eqsin.wmnet<nowiki>}</nowiki> and A:cp * 12:57 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp308[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 12:57 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3081.esams.wmnet * 12:54 isaranto@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 12:41 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 12:15 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3080.esams.wmnet * 12:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 12:03 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp308[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[2-3].esams.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3073.esams.wmnet * 11:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:28 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2154: Migration of db2154.codfw.wmnet completed * 11:19 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3072.esams.wmnet * 11:15 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 11:11 fnegri@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb1017.eqiad.wmnet with reason: Rebooting clouddb1017 * 11:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 11:09 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1172: Migration of db1172.eqiad.wmnet completed * 11:07 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[2-3].esams.wmnet<nowiki>}</nowiki> and A:cp * 11:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1058.eqiad.wmnet * 11:01 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 11:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3079.esams.wmnet * 10:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1058.eqiad.wmnet * 10:55 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 10:55 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1058.eqiad.wmnet to cluster eqiad and group C * 10:48 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 10:47 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 10:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1024.eqiad.wmnet * 10:43 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 10:43 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 10:43 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 10:42 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 10:42 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 10:42 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2154: Migration of db2154.codfw.wmnet completed * 10:42 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 10:41 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1024.eqiad.wmnet * 10:37 moritzm: remove ganeti1024 foom eqiad Ganeti cluster [[phab:T424680|T424680]] * 10:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2154.codfw.wmnet with OS trixie * 10:31 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2010.codfw.wmnet with OS trixie * 10:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1024.eqiad.wmnet * 10:24 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1172: Migration of db1172.eqiad.wmnet completed * 10:19 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3078.esams.wmnet * 10:18 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2154.codfw.wmnet with reason: host reimage * 10:16 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1172.eqiad.wmnet with OS trixie * 10:15 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1017.eqiad.wmnet * 10:13 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2154.codfw.wmnet with reason: host reimage * 10:07 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 10:06 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 10:06 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3071.esams.wmnet * 09:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1172.eqiad.wmnet with reason: host reimage * 09:56 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2154.codfw.wmnet with OS trixie * 09:55 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage * 09:53 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1172.eqiad.wmnet with reason: host reimage * 09:51 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage * 09:39 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2154: Upgrading db2154.codfw.wmnet * 09:39 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2154: Upgrading db2154.codfw.wmnet * 09:38 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:38 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1172.eqiad.wmnet with OS trixie * 09:35 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1172: Upgrading db1172.eqiad.wmnet * 09:34 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1172: Upgrading db1172.eqiad.wmnet * 09:34 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 09:34 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2009.codfw.wmnet with OS trixie * 09:33 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS trixie * 09:26 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 09:26 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 09:26 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3070.esams.wmnet * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 09:16 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie * 09:14 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[0-1].esams.wmnet<nowiki>}</nowiki> and A:cp * 09:11 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp307[6-7].esams.wmnet<nowiki>}</nowiki> and A:cp * 09:11 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3077.esams.wmnet * 09:04 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 09:03 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie * 08:47 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 08:46 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie * 08:40 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie * 08:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 08:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 08:30 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3076.esams.wmnet * 08:18 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp307[6-7].esams.wmnet<nowiki>}</nowiki> and A:cp * 08:15 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti1058.eqiad.wmnet on all recursors * 08:15 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:15 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change records for ganeti1058 - cmooney@cumin1003" * 08:15 cmooney@cumin1003: START - Cookbook sre.dns.wipe-cache ganeti1058.eqiad.wmnet on all recursors * 08:15 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change records for ganeti1058 - cmooney@cumin1003" * 08:09 cmooney@cumin1003: START - Cookbook sre.dns.netbox * 08:07 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp306[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 08:07 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3069.esams.wmnet * 08:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 08:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 07:31 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1024.eqiad.wmnet * 07:26 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3068.esams.wmnet * 07:14 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp306[8-9].esams.wmnet<nowiki>}</nowiki> and A:cp * 07:11 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1057.eqiad.wmnet to cluster eqiad and group A * 07:10 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3075.esams.wmnet<nowiki>}</nowiki> and A:cp * 07:10 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3075.esams.wmnet * 07:06 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1057.eqiad.wmnet to cluster eqiad and group A * 07:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1057.eqiad.wmnet * 07:02 jmm@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1057 * 07:01 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1057 * 06:58 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3075.esams.wmnet<nowiki>}</nowiki> and A:cp * 06:58 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3067.esams.wmnet<nowiki>}</nowiki> and A:cp * 06:58 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3067.esams.wmnet * 06:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1057.eqiad.wmnet * 06:46 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3067.esams.wmnet<nowiki>}</nowiki> and A:cp * 06:13 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1024.eqiad.wmnet * 06:08 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1024.eqiad.wmnet * 06:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org * 06:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org * 05:25 marostegui@dns1004: END - running authdns-update * 05:24 marostegui@dns1004: START - running authdns-update * 05:23 marostegui: Failover m5-master [[phab:T426633|T426633]] * 05:19 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy1028.eqiad.wmnet with reason: Reboot * 05:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy2005.codfw.wmnet with reason: Reboot * 05:11 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1012.eqiad.wmnet * 05:11 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 05:11 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1012.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 05:06 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc1012.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" * 05:03 marostegui@cumin1003: START - Cookbook sre.dns.netbox * 04:56 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc1012.eqiad.wmnet == 2026-05-21 == * 23:43 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290954{{!}}Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957{{!}}Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958{{!}}Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960{{!}}Drop unused wpReportIncidentLocalLinks]] (duration: 06m 42s) * 23:38 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 23:38 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1290954{{!}}Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957{{!}}Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958{{!}}Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960{{!}}Drop unused wpReportIncidentLocalLinks]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified * 23:36 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1290954{{!}}Drop not defined config $wgAllowRawHtmlCopyrightMessages]], [[gerrit:1290957{{!}}Drop $wgGraphShowInToolbar definition as unused]], [[gerrit:1290958{{!}}Drop wgMFSearchGenerator definition as unused]], [[gerrit:1290960{{!}}Drop unused wpReportIncidentLocalLinks]] * 22:26 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul2002.codfw.wmnet with OS trixie * 22:08 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul2002.codfw.wmnet with reason: host reimage * 22:03 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul2002.codfw.wmnet with reason: host reimage * 22:02 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 21:49 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 21:49 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 21:44 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host zuul2002.codfw.wmnet with OS trixie * 21:25 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:25 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:20 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:19 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 20:26 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 20:16 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 19:22 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-reboot (exit_code=0) rolling reboot on A:restbase * 19:10 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:59 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:53 papaul: rebooting msw1-codfw * 18:50 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:39 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 17:54 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:53 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:52 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:52 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 17:52 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:51 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:50 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:49 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:49 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:48 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:46 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 17:46 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 17:43 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:43 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:43 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:42 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:42 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:41 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:41 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:41 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:41 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:41 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:41 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:41 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:40 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:40 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:40 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:39 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 * 17:39 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:38 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cp6015.drmrs.wmnet with reason: hardware down * 17:37 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 17:36 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:36 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:30 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:25 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:25 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:24 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:23 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:22 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1016.eqiad.wmnet * 17:22 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:14 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2031.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:14 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs2030.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:13 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1016.eqiad.wmnet * 17:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:09 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:08 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repool pc2 ([[phab:T421705|T421705]])', diff saved to https://phabricator.wikimedia.org/P92810 and previous config saved to /var/cache/conftool/dbconfig/20260521-170823-ladsgroup.json * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:07 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2031.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:07 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2030.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:07 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2029.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:06 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host wdqs2028.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 17:03 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 17:03 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 17:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 17:03 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 17:00 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2029 * 16:58 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2031 * 16:58 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:58 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 * 16:57 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2028 * 16:55 papaul: rebooting msw-d3-codfw * 16:55 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 16:52 papaul: rebooting msw-c7-codfw * 16:51 papaul: rebooting msw-c6-codfw * 16:48 papaul: rebooting msw-b7-codfw * 16:48 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1014.eqiad.wmnet * 16:45 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for clouddb1014.eqiad.wmnet * 16:43 papaul: rebooting msw-b6-codfw * 16:40 papaul: rebooting msw-a1-codfw * 16:37 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 16:37 fnegri@cumin1003: START - Cookbook sre.mysql.upgrade for clouddb1014.eqiad.wmnet * 16:37 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:36 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:35 jhancock@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wdqs2031 * 16:35 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2030 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2031 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2030 * 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2029 * 16:34 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wdqs2028 * 16:34 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:33 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2028 to codfw - jhancock@cumin2002" * 16:33 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wdqs2028 to codfw - jhancock@cumin2002" * 16:26 jhancock@cumin2002: START - Cookbook sre.dns.netbox * 16:24 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on pc1022.eqiad.wmnet with reason: Move to nftables * 16:24 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on pc2022.codfw.wmnet with reason: Move to nftables * 16:18 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2048: Repooling * 16:18 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depool pc2 ([[phab:T421705|T421705]])', diff saved to https://phabricator.wikimedia.org/P92807 and previous config saved to /var/cache/conftool/dbconfig/20260521-161808-ladsgroup.json * 16:15 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:15 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:15 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:15 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:05 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:58 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:57 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:52 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 15:42 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es2048: Repooling * 15:41 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92804 and previous config saved to /var/cache/conftool/dbconfig/20260521-154108-fceratto.json * 15:39 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:38 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:34 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:34 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:34 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:34 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:34 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92803 and previous config saved to /var/cache/conftool/dbconfig/20260521-153400-fceratto.json * 15:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2048.codfw.wmnet with reason: Maintenance * 15:33 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92802 and previous config saved to /var/cache/conftool/dbconfig/20260521-153331-fceratto.json * 15:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:25 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 15:24 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 15:24 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 15:24 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 15:24 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 15:23 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040', diff saved to https://phabricator.wikimedia.org/P92801 and previous config saved to /var/cache/conftool/dbconfig/20260521-152323-fceratto.json * 15:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1045.eqiad.wmnet * 15:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1045.eqiad.wmnet * 15:19 claime: Enabling puppet on A:cp-text - [[phab:T426323|T426323]] * 15:15 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1045.eqiad.wmnet * 15:13 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040', diff saved to https://phabricator.wikimedia.org/P92800 and previous config saved to /var/cache/conftool/dbconfig/20260521-151316-fceratto.json * 15:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1014.eqiad.wmnet * 15:11 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1045.eqiad.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2034.codfw.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2034.codfw.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1037.eqiad.wmnet * 15:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1037.eqiad.wmnet * 15:07 elukey@cumin1003: END (PASS) - Cookbook sre.misc-clusters.restart-reboot-config-master (exit_code=0) rolling reboot on A:config-master * 15:06 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1014.eqiad.wmnet * 15:05 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors * 15:05 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors * 15:04 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290805{{!}}hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] (duration: 10m 11s) * 15:03 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92799 and previous config saved to /var/cache/conftool/dbconfig/20260521-150308-fceratto.json * 15:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1037.eqiad.wmnet * 15:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2034.codfw.wmnet * 15:00 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors * 15:00 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors * 15:00 elukey@cumin1003: START - Cookbook sre.misc-clusters.restart-reboot-config-master rolling reboot on A:config-master * 15:00 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 15:00 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-lab1002.eqiad.wmnet * 14:59 elukey@cumin1003: END (PASS) - Cookbook sre.pki.restart-reboot (exit_code=0) rolling reboot on A:pki * 14:57 claime: Disabling puppet on A:cp-text - [[phab:T426323|T426323]] * 14:56 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1290805{{!}}hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:55 klausman@cumin1003: START - Cookbook sre.hosts.reboot-single for host ml-lab1002.eqiad.wmnet * 14:54 klausman@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-build1001.eqiad.wmnet * 14:54 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1290805{{!}}hCaptcha: Enable for DiscussionTools on Group 0 wikis (T426039)]] * 14:54 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet * 14:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1013.eqiad.wmnet * 14:53 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1037.eqiad.wmnet * 14:53 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1028.eqiad.wmnet * 14:53 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>ml-serve1001.eqiad.wmnet<nowiki>}</nowiki> and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) * 14:53 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1001.eqiad.wmnet * 14:53 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1001.eqiad.wmnet * 14:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1028.eqiad.wmnet * 14:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92798 and previous config saved to /var/cache/conftool/dbconfig/20260521-145132-fceratto.json * 14:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2040.codfw.wmnet with reason: Maintenance * 14:51 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92797 and previous config saved to /var/cache/conftool/dbconfig/20260521-145103-fceratto.json * 14:50 klausman@cumin1003: START - Cookbook sre.hosts.reboot-single for host ml-build1001.eqiad.wmnet * 14:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 14:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: Migration of db2241.codfw.wmnet completed * 14:48 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1001.eqiad.wmnet * 14:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1013.eqiad.wmnet * 14:46 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1028.eqiad.wmnet * 14:45 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:44 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:42 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1001.eqiad.wmnet * 14:42 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>ml-serve1001.eqiad.wmnet<nowiki>}</nowiki> and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) * 14:42 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1028.eqiad.wmnet * 14:42 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-worker-eqiad * 14:42 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1011.eqiad.wmnet * 14:42 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1011.eqiad.wmnet * 14:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039', diff saved to https://phabricator.wikimedia.org/P92795 and previous config saved to /var/cache/conftool/dbconfig/20260521-144055-fceratto.json * 14:39 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1012.eqiad.wmnet * 14:38 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet. on all recursors * 14:37 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet. on all recursors * 14:37 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1011.eqiad.wmnet * 14:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1027.eqiad.wmnet * 14:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1027.eqiad.wmnet * 14:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1011.eqiad.wmnet * 14:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1012.eqiad.wmnet * 14:32 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1010.eqiad.wmnet * 14:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1010.eqiad.wmnet * 14:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039', diff saved to https://phabricator.wikimedia.org/P92793 and previous config saved to /var/cache/conftool/dbconfig/20260521-143045-fceratto.json * 14:30 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet. on all recursors * 14:30 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet. on all recursors * 14:29 elukey@cumin1003: START - Cookbook sre.pki.restart-reboot rolling reboot on A:pki * 14:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1027.eqiad.wmnet * 14:27 slyngshede@cumin1003: END (FAIL) - Cookbook sre.cdn.roll-reboot (exit_code=1) rolling reboot on P<nowiki>{</nowiki>cp601[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 14:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1027.eqiad.wmnet * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:25 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1054.eqiad.wmnet * 14:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1054.eqiad.wmnet * 14:24 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1010.eqiad.wmnet * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:21 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1011.eqiad.wmnet * 14:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92792 and previous config saved to /var/cache/conftool/dbconfig/20260521-142037-fceratto.json * 14:19 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1054.eqiad.wmnet * 14:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:19 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 14:17 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1054.eqiad.wmnet * 14:17 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1053.eqiad.wmnet * 14:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1053.eqiad.wmnet * 14:14 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1010.eqiad.wmnet * 14:14 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1009.eqiad.wmnet * 14:14 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1009.eqiad.wmnet * 14:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:13 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps1011.eqiad.wmnet * 14:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:12 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2218: repool after maintenance * 14:11 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1053.eqiad.wmnet * 14:09 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92789 and previous config saved to /var/cache/conftool/dbconfig/20260521-140906-fceratto.json * 14:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2039.codfw.wmnet with reason: Maintenance * 14:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92788 and previous config saved to /var/cache/conftool/dbconfig/20260521-140837-fceratto.json * 14:08 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1009.eqiad.wmnet * 14:08 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:07 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1053.eqiad.wmnet * 14:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1035.eqiad.wmnet * 14:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1035.eqiad.wmnet * 14:04 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2241: Migration of db2241.codfw.wmnet completed * 14:03 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1009.eqiad.wmnet * 14:03 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1008.eqiad.wmnet * 14:03 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1008.eqiad.wmnet * 14:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2241.codfw.wmnet with OS trixie * 13:59 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 13:59 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1035.eqiad.wmnet * 13:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92786 and previous config saved to /var/cache/conftool/dbconfig/20260521-135830-fceratto.json * 13:58 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1008.eqiad.wmnet * 13:53 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1008.eqiad.wmnet * 13:53 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1007.eqiad.wmnet * 13:53 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1007.eqiad.wmnet * 13:51 Lucas_WMDE: UTC afternoon backport+config window done * 13:51 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290743{{!}}composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347{{!}}Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342{{!}}fix: simplify to show only one icon type for password reveal (T419413)]] (duration: 07m 20s) * 13:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048', diff saved to https://phabricator.wikimedia.org/P92784 and previous config saved to /var/cache/conftool/dbconfig/20260521-134822-fceratto.json * 13:48 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1007.eqiad.wmnet * 13:47 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 13:46 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Continuing with deployment * 13:45 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 13:45 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, migr: Backport for [[gerrit:1290743{{!}}composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347{{!}}Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342{{!}}fix: simplify to show only one icon type for password reveal (T419413)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes * 13:44 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2241.codfw.wmnet with reason: host reimage * 13:44 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 13:43 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1290743{{!}}composer.json: Updated symfony/yaml from 7.4.6 to 7.4.12 (T426861)]], [[gerrit:1289347{{!}}Skip init.test.js test if VisualEditor not installed (T426740)]], [[gerrit:1289342{{!}}fix: simplify to show only one icon type for password reveal (T419413)]] * 13:43 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 13:43 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1007.eqiad.wmnet * 13:42 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1006.eqiad.wmnet * 13:42 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1006.eqiad.wmnet * 13:41 dbrant@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290035{{!}}docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] (duration: 06m 52s) * 13:41 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 13:40 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2241.codfw.wmnet with reason: host reimage * 13:39 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1035.eqiad.wmnet * 13:38 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in codfw/ml-serve-codfw: maintenance * 13:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92782 and previous config saved to /var/cache/conftool/dbconfig/20260521-133815-fceratto.json * 13:37 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1006.eqiad.wmnet * 13:37 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in codfw/ml-serve-codfw: maintenance * 13:37 dbrant@deploy1003: dbrant: Continuing with deployment * 13:36 dbrant@deploy1003: dbrant: Backport for [[gerrit:1290035{{!}}docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1032.eqiad.wmnet * 13:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1032.eqiad.wmnet * 13:35 dbrant@deploy1003: Started scap sync-world: Backport for [[gerrit:1290035{{!}}docroot: Remove non-wikipedias from digital asset links. (T426010 T385520)]] * 13:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1006.eqiad.wmnet * 13:32 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1005.eqiad.wmnet * 13:32 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1005.eqiad.wmnet * 13:31 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290014{{!}}Enable AG on phase 2 wikis (T426871)]] (duration: 09m 11s) * 13:31 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1048 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92781 and previous config saved to /var/cache/conftool/dbconfig/20260521-133116-fceratto.json * 13:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1048.eqiad.wmnet with reason: Maintenance * 13:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92780 and previous config saved to /var/cache/conftool/dbconfig/20260521-133048-fceratto.json * 13:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1032.eqiad.wmnet * 13:28 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1032.eqiad.wmnet * 13:27 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1005.eqiad.wmnet * 13:27 sbisson@deploy1003: sbisson: Continuing with deployment * 13:27 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2218: repool after maintenance * 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1031.eqiad.wmnet * 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1031.eqiad.wmnet * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:25 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2241.codfw.wmnet with OS trixie * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 13:24 sbisson@deploy1003: sbisson: Backport for [[gerrit:1290014{{!}}Enable AG on phase 2 wikis (T426871)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:23 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: Upgrading db2241.codfw.wmnet * 13:23 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2241: Upgrading db2241.codfw.wmnet * 13:23 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 13:22 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1290014{{!}}Enable AG on phase 2 wikis (T426871)]] * 13:22 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1005.eqiad.wmnet * 13:22 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1004.eqiad.wmnet * 13:22 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1004.eqiad.wmnet * 13:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040', diff saved to https://phabricator.wikimedia.org/P92778 and previous config saved to /var/cache/conftool/dbconfig/20260521-132041-fceratto.json * 13:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1031.eqiad.wmnet * 13:20 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290088{{!}}Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032{{!}}Enable 'flood' user group at en.wikiversity (T426882)]] (duration: 11m 55s) * 13:18 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet * 13:17 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1018.eqiad.wmnet with OS trixie * 13:16 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1031.eqiad.wmnet * 13:16 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1039: Repooling * 13:16 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1030.eqiad.wmnet * 13:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1030.eqiad.wmnet * 13:15 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Continuing with deployment * 13:15 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1004.eqiad.wmnet * 13:14 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet * 13:11 eevans@cumin1003: START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:restbase * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . * 13:10 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1004.eqiad.wmnet * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . * 13:10 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040', diff saved to https://phabricator.wikimedia.org/P92776 and previous config saved to /var/cache/conftool/dbconfig/20260521-131033-fceratto.json * 13:10 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1003.eqiad.wmnet * 13:10 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1003.eqiad.wmnet * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 13:10 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2241 [[phab:T426936|T426936]]', diff saved to https://phabricator.wikimedia.org/P92775 and previous config saved to /var/cache/conftool/dbconfig/20260521-131025-cwilliams.json * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'readability' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'logo-detection' for release 'main' . * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . * 13:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1030.eqiad.wmnet * 13:10 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . * 13:10 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, neriah: Backport for [[gerrit:1290088{{!}}Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032{{!}}Enable 'flood' user group at en.wikiversity (T426882)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'llm' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 13:09 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . * 13:08 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1290088{{!}}Disable wgUseFilePatrol in ukwiki (T426905)]], [[gerrit:1290032{{!}}Enable 'flood' user group at en.wikiversity (T426882)]] * 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet * 13:06 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp601[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 13:06 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3074.esams.wmnet<nowiki>}</nowiki> and A:cp * 13:06 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3074.esams.wmnet * 13:06 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2162 to x3 primary [[phab:T426936|T426936]]', diff saved to https://phabricator.wikimedia.org/P92774 and previous config saved to /var/cache/conftool/dbconfig/20260521-130609-cwilliams.json * 13:04 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 13:04 cezmunsta: Starting x3 codfw failover from db2241 to db2162 - [[phab:T426936|T426936]] * 13:04 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1003.eqiad.wmnet * 13:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1030.eqiad.wmnet * 13:03 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet * 13:00 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 13:00 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92772 and previous config saved to /var/cache/conftool/dbconfig/20260521-130018-fceratto.json * 12:59 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1003.eqiad.wmnet * 12:59 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1018.eqiad.wmnet with reason: host reimage * 12:59 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1002.eqiad.wmnet * 12:59 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1002.eqiad.wmnet * 12:58 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:57 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:56 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2162 with weight 0 [[phab:T426936|T426936]]', diff saved to https://phabricator.wikimedia.org/P92771 and previous config saved to /var/cache/conftool/dbconfig/20260521-125645-cwilliams.json * 12:56 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Primary switchover x3 [[phab:T426936|T426936]] * 12:56 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:55 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1029.eqiad.wmnet * 12:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1029.eqiad.wmnet * 12:54 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3074.esams.wmnet<nowiki>}</nowiki> and A:cp * 12:54 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1002.eqiad.wmnet * 12:54 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp600[7-8].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 12:54 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6008.drmrs.wmnet * 12:53 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:52 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1018.eqiad.wmnet with reason: host reimage * 12:51 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:49 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1002.eqiad.wmnet * 12:49 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-worker-eqiad * 12:48 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1029.eqiad.wmnet * 12:48 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp3066.esams.wmnet<nowiki>}</nowiki> and A:cp * 12:48 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp3066.esams.wmnet * 12:47 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:47 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1040 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92770 and previous config saved to /var/cache/conftool/dbconfig/20260521-124707-fceratto.json * 12:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1040.eqiad.wmnet with reason: Maintenance * 12:46 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es1039: Repooling * 12:46 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:45 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1029.eqiad.wmnet * 12:45 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:44 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:43 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:43 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290727{{!}}hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] (duration: 07m 54s) * 12:42 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92768 and previous config saved to /var/cache/conftool/dbconfig/20260521-124014-fceratto.json * 12:39 kharlan@deploy1003: kharlan: Continuing with deployment * 12:38 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1052.eqiad.wmnet * 12:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1052.eqiad.wmnet * 12:37 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1018.eqiad.wmnet with OS trixie * 12:37 kharlan@deploy1003: kharlan: Backport for [[gerrit:1290727{{!}}hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:36 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:36 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp3066.esams.wmnet<nowiki>}</nowiki> and A:cp * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:35 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1290727{{!}}hCaptcha: Finish group1 account creation rollout + itwiki/hewiki for mobile apps (T426045 T425354)]] * 12:35 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:34 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1017.eqiad.wmnet with OS trixie * 12:34 kart_: Updated cxserver to 2026-05-20-034002-production ([[phab:T388690|T388690]], [[phab:T404295|T404295]], [[phab:T391703|T391703]], [[phab:T426605|T426605]]) * 12:34 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet * 12:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1052.eqiad.wmnet * 12:30 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply * 12:30 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply * 12:30 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet * 12:29 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 12:29 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1039 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92767 and previous config saved to /var/cache/conftool/dbconfig/20260521-122905-fceratto.json * 12:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1039.eqiad.wmnet with reason: Maintenance * 12:28 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92766 and previous config saved to /var/cache/conftool/dbconfig/20260521-122839-fceratto.json * 12:27 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply * 12:27 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply * 12:26 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 12:23 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-staging-worker * 12:23 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2003.codfw.wmnet * 12:23 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2003.codfw.wmnet * 12:22 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1052.eqiad.wmnet * 12:21 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply * 12:21 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply * 12:21 moritzm: installing nginx security updates * 12:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1051.eqiad.wmnet * 12:20 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in codfw/ml-serve-codfw: maintenance * 12:19 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1017.eqiad.wmnet with reason: host reimage * 12:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1051.eqiad.wmnet * 12:19 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in codfw/ml-serve-codfw: maintenance * 12:19 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in codfw/ml-staging-codfw: maintenance * 12:19 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in codfw/ml-staging-codfw: maintenance * 12:19 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in codfw/ml-staging-codfw: maintenance * 12:18 dpogorzelski@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster depool all services in codfw/ml-staging-codfw: maintenance * 12:18 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047', diff saved to https://phabricator.wikimedia.org/P92765 and previous config saved to /var/cache/conftool/dbconfig/20260521-121832-fceratto.json * 12:17 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2003.codfw.wmnet * 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet * 12:15 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1017.eqiad.wmnet with reason: host reimage * 12:14 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1051.eqiad.wmnet * 12:13 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6007.drmrs.wmnet * 12:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet * 12:10 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1051.eqiad.wmnet * 12:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047', diff saved to https://phabricator.wikimedia.org/P92764 and previous config saved to /var/cache/conftool/dbconfig/20260521-120824-fceratto.json * 12:07 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2003.codfw.wmnet * 12:07 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2002.codfw.wmnet * 12:07 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2002.codfw.wmnet * 12:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1050.eqiad.wmnet * 12:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1050.eqiad.wmnet * 12:02 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp600[7-8].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp601[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 12:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6014.drmrs.wmnet * 12:00 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1017.eqiad.wmnet with OS trixie * 12:00 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2002.codfw.wmnet * 11:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1002.wikimedia.org * 11:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92763 and previous config saved to /var/cache/conftool/dbconfig/20260521-115817-fceratto.json * 11:57 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1050.eqiad.wmnet * 11:53 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt1002.wikimedia.org * 11:51 taavi: disabling puppet on C:bird to roll out {{Gerrit|1289919}} * 11:51 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92762 and previous config saved to /var/cache/conftool/dbconfig/20260521-115112-fceratto.json * 11:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2047.codfw.wmnet with reason: Maintenance * 11:50 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1050.eqiad.wmnet * 11:50 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2002.codfw.wmnet * 11:50 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92761 and previous config saved to /var/cache/conftool/dbconfig/20260521-115043-fceratto.json * 11:50 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2001.codfw.wmnet * 11:50 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2001.codfw.wmnet * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1049.eqiad.wmnet * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt2002.wikimedia.org * 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1049.eqiad.wmnet * 11:45 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2001.codfw.wmnet * 11:45 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker-exp1001.eqiad.wmnet * 11:44 kartik@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . * 11:44 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1049.eqiad.wmnet * 11:43 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt2002.wikimedia.org * 11:42 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet * 11:40 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet * 11:40 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037', diff saved to https://phabricator.wikimedia.org/P92760 and previous config saved to /var/cache/conftool/dbconfig/20260521-114036-fceratto.json * 11:39 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker-exp1001.eqiad.wmnet * 11:39 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker-exp2001.codfw.wmnet * 11:38 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host testreduce1002.eqiad.wmnet * 11:37 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1049.eqiad.wmnet * 11:36 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet * 11:36 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet * 11:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1038.eqiad.wmnet * 11:35 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2001.codfw.wmnet * 11:35 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-staging-worker * 11:35 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet * 11:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1038.eqiad.wmnet * 11:34 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host testreduce1002.eqiad.wmnet * 11:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker-exp2001.codfw.wmnet * 11:32 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet * 11:31 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet * 11:30 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt-staging2001.codfw.wmnet * 11:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037', diff saved to https://phabricator.wikimedia.org/P92759 and previous config saved to /var/cache/conftool/dbconfig/20260521-113028-fceratto.json * 11:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2014.codfw.wmnet * 11:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1038.eqiad.wmnet * 11:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt-staging2001.codfw.wmnet * 11:26 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet * 11:24 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1038.eqiad.wmnet * 11:24 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1034.eqiad.wmnet * 11:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1034.eqiad.wmnet * 11:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2014.codfw.wmnet * 11:20 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6013.drmrs.wmnet * 11:20 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92758 and previous config saved to /var/cache/conftool/dbconfig/20260521-112021-fceratto.json * 11:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1034.eqiad.wmnet * 11:14 jmm@cumin2002: END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling reboot on A:ldap-replicas-eqiad * 11:13 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2013.codfw.wmnet * 11:11 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1034.eqiad.wmnet * 11:09 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp601[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92757 and previous config saved to /var/cache/conftool/dbconfig/20260521-110851-fceratto.json * 11:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2037.codfw.wmnet with reason: Maintenance * 11:08 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92756 and previous config saved to /var/cache/conftool/dbconfig/20260521-110822-fceratto.json * 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1033.eqiad.wmnet * 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1033.eqiad.wmnet * 11:05 jmm@cumin2002: START - Cookbook sre.ldap.roll-restart-reboot-replica rolling reboot on A:ldap-replicas-eqiad * 11:05 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2013.codfw.wmnet * 11:04 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp600[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 11:04 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6006.drmrs.wmnet * 11:02 jmm@cumin2002: END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling reboot on A:ldap-replicas-codfw * 11:00 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1033.eqiad.wmnet * 10:59 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1016.eqiad.wmnet with reason: host reimage * 10:58 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036', diff saved to https://phabricator.wikimedia.org/P92753 and previous config saved to /var/cache/conftool/dbconfig/20260521-105815-fceratto.json * 10:57 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1033.eqiad.wmnet * 10:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1044.eqiad.wmnet * 10:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1044.eqiad.wmnet * 10:55 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1016.eqiad.wmnet with reason: host reimage * 10:54 jmm@cumin2002: START - Cookbook sre.ldap.roll-restart-reboot-replica rolling reboot on A:ldap-replicas-codfw * 10:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2012.codfw.wmnet * 10:51 dpogorzelski@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:51 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:51 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1044.eqiad.wmnet * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:50 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036', diff saved to https://phabricator.wikimedia.org/P92752 and previous config saved to /var/cache/conftool/dbconfig/20260521-104807-fceratto.json * 10:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2012.codfw.wmnet * 10:46 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1044.eqiad.wmnet * 10:44 jiji@deploy1003: Finished scap sync-world: Backport for [[gerrit:1290709{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] (duration: 08m 02s) * 10:43 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:41 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:40 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2005.codfw.wmnet * 10:40 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:39 jiji@deploy1003: jiji: Continuing with deployment * 10:38 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es2036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92751 and previous config saved to /var/cache/conftool/dbconfig/20260521-103759-fceratto.json * 10:37 jiji@deploy1003: jiji: Backport for [[gerrit:1290709{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:36 jiji@deploy1003: Started scap sync-world: Backport for [[gerrit:1290709{{!}}ProductionServices.php: switch filebackend.php to rdb2011:6381 (T418261 T419976)]] * 10:35 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2005.codfw.wmnet * 10:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1043.eqiad.wmnet * 10:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1043.eqiad.wmnet * 10:34 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:29 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 10:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1043.eqiad.wmnet * 10:27 dcausse: [[phab:T423993|T423993]]: reindexing all archive indices * 10:27 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . * 10:26 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es2036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92749 and previous config saved to /var/cache/conftool/dbconfig/20260521-102630-fceratto.json * 10:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance * 10:26 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1043.eqiad.wmnet * 10:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92748 and previous config saved to /var/cache/conftool/dbconfig/20260521-102601-fceratto.json * 10:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2011.codfw.wmnet * 10:24 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6005.drmrs.wmnet * 10:22 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1042.eqiad.wmnet * 10:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1042.eqiad.wmnet * 10:17 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2011.codfw.wmnet * 10:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1042.eqiad.wmnet * 10:15 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047', diff saved to https://phabricator.wikimedia.org/P92747 and previous config saved to /var/cache/conftool/dbconfig/20260521-101552-fceratto.json * 10:15 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 10:14 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . * 10:13 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1042.eqiad.wmnet * 10:13 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1041.eqiad.wmnet * 10:12 moritzm: installing postgresql security updates * 10:12 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp600[5-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 10:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1041.eqiad.wmnet * 10:10 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry2004.codfw.wmnet * 10:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org * 10:09 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . * 10:08 fnegri@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb1013.eqiad.wmnet * 10:08 fnegri@cumin1003: START - Cookbook sre.hosts.remove-downtime for clouddb1013.eqiad.wmnet * 10:07 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1013.eqiad.wmnet * 10:07 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1041.eqiad.wmnet * 10:05 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047', diff saved to https://phabricator.wikimedia.org/P92746 and previous config saved to /var/cache/conftool/dbconfig/20260521-100545-fceratto.json * 10:05 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry2004.codfw.wmnet * 10:04 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1041.eqiad.wmnet * 10:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1040.eqiad.wmnet * 10:04 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1005.eqiad.wmnet * 10:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1040.eqiad.wmnet * 10:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org * 10:01 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1013.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 10:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1040.eqiad.wmnet * 10:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:00 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 10:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org * 09:59 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1005.eqiad.wmnet * 09:58 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-codfw * 09:58 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2005.codfw.wmnet * 09:58 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2005.codfw.wmnet * 09:56 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1040.eqiad.wmnet * 09:56 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1039.eqiad.wmnet * 09:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1039.eqiad.wmnet * 09:56 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 09:56 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:55 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ml-serve1013.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 09:55 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92745 and previous config saved to /var/cache/conftool/dbconfig/20260521-095536-fceratto.json * 09:54 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1384.eqiad.wmnet * 09:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org * 09:54 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:54 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:52 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply * 09:52 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2005.codfw.wmnet * 09:52 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2005.codfw.wmnet * 09:52 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply * 09:52 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2004.codfw.wmnet * 09:52 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2004.codfw.wmnet * 09:51 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply * 09:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1039.eqiad.wmnet * 09:49 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1384.eqiad.wmnet * 09:49 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 09:49 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1383.eqiad.wmnet * 09:48 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1039.eqiad.wmnet * 09:48 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1036.eqiad.wmnet * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1047 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92744 and previous config saved to /var/cache/conftool/dbconfig/20260521-094829-fceratto.json * 09:48 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1036.eqiad.wmnet * 09:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1047.eqiad.wmnet with reason: Maintenance * 09:48 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92743 and previous config saved to /var/cache/conftool/dbconfig/20260521-094801-fceratto.json * 09:47 fnegri@cumin1003: conftool action : set/pooled=no; selector: name=clouddb1013.eqiad.wmnet * 09:47 fnegri@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1013.eqiad.wmnet with reason: Rebooting clouddb1013 [[phab:T426563|T426563]] * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2004.codfw.wmnet * 09:45 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2004.codfw.wmnet * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet * 09:45 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad * 09:45 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet * 09:45 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet * 09:44 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1383.eqiad.wmnet * 09:44 elukey@cumin1003: START - Cookbook sre.hosts.provision for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 09:44 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1382.eqiad.wmnet * 09:42 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2002.codfw.wmnet * 09:40 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1036.eqiad.wmnet * 09:39 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host registry1004.eqiad.wmnet * 09:38 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1382.eqiad.wmnet * 09:38 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1381.eqiad.wmnet * 09:38 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1036.eqiad.wmnet * 09:38 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2003.codfw.wmnet * 09:38 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2003.codfw.wmnet * 09:38 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2002.codfw.wmnet * 09:38 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2002.codfw.wmnet * 09:37 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037', diff saved to https://phabricator.wikimedia.org/P92742 and previous config saved to /var/cache/conftool/dbconfig/20260521-093754-fceratto.json * 09:37 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 09:37 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet * 09:37 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet * 09:37 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet * 09:37 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet * 09:36 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host build2002.codfw.wmnet * 09:36 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 09:35 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp601[1-2].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 09:35 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6012.drmrs.wmnet * 09:34 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host registry1004.eqiad.wmnet * 09:33 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet * 09:33 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1381.eqiad.wmnet * 09:33 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1380.eqiad.wmnet * 09:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1023.eqiad.wmnet * 09:31 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet * 09:31 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2002.codfw.wmnet * 09:31 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2002.codfw.wmnet * 09:31 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2001.codfw.wmnet * 09:31 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2001.codfw.wmnet * 09:30 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet * 09:30 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet * 09:30 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet * 09:30 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet * 09:29 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet * 09:29 jayme@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad * 09:29 jayme@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw * 09:29 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet * 09:28 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet * 09:27 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037', diff saved to https://phabricator.wikimedia.org/P92741 and previous config saved to /var/cache/conftool/dbconfig/20260521-092746-fceratto.json * 09:27 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1380.eqiad.wmnet * 09:27 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1379.eqiad.wmnet * 09:27 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet * 09:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1023.eqiad.wmnet * 09:25 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet * 09:24 jayme@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw * 09:23 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1056.eqiad.wmnet to cluster eqiad and group A * 09:23 jayme@cumin1003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet * 09:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet * 09:22 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad * 09:22 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1379.eqiad.wmnet * 09:22 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1378.eqiad.wmnet * 09:21 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2001.codfw.wmnet * 09:21 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2001.codfw.wmnet * 09:21 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-codfw * 09:21 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1056.eqiad.wmnet to cluster eqiad and group A * 09:20 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie * 09:18 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-jumbo1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:18 moritzm: remove ganeti1023 foom eqiad Ganeti cluster [[phab:T424680|T424680]] * 09:17 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92740 and previous config saved to /var/cache/conftool/dbconfig/20260521-091738-fceratto.json * 09:16 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1378.eqiad.wmnet * 09:16 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1377.eqiad.wmnet * 09:12 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1377.eqiad.wmnet * 09:12 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1376.eqiad.wmnet * 09:07 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1036: Repooling * 09:07 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1376.eqiad.wmnet * 09:07 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker1375.eqiad.wmnet * 09:06 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1037 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92738 and previous config saved to /var/cache/conftool/dbconfig/20260521-090609-fceratto.json * 09:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1037.eqiad.wmnet with reason: Maintenance * 09:02 jiji@cumin1003: START - Cookbook sre.hosts.reboot-single for host wikikube-worker1375.eqiad.wmnet * 09:01 btullis@cumin1003: START - Cookbook sre.hosts.provision for host kafka-jumbo1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:55 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6011.drmrs.wmnet * 08:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1023.eqiad.wmnet * 08:47 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) * 08:47 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1256: Migration of db1256.eqiad.wmnet completed * 08:44 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp601[1-2].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 08:42 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp600[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 08:42 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6004.drmrs.wmnet * 08:37 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es1036: Repooling * 08:29 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance es1036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92733 and previous config saved to /var/cache/conftool/dbconfig/20260521-082951-fceratto.json * 08:29 hashar@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.3 refs [[phab:T423912|T423912]] * 08:16 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling es1036 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P92731 and previous config saved to /var/cache/conftool/dbconfig/20260521-081642-fceratto.json * 08:16 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1036.eqiad.wmnet with reason: Maintenance * 08:02 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1256: Migration of db1256.eqiad.wmnet completed * 08:01 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6003.drmrs.wmnet * 08:00 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1256.eqiad.wmnet with OS trixie * 07:52 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp600[3-4].drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:51 marostegui@dns1004: END - running authdns-update * 07:50 marostegui@dns1004: START - running authdns-update * 07:48 marostegui: Failover m3-master [[phab:T426633|T426633]] * 07:47 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1023.eqiad.wmnet * 07:46 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp6010.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:46 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6010.drmrs.wmnet * 07:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster1005.eqiad.wmnet to plain * 07:44 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster1005.eqiad.wmnet to plain * 07:43 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1256.eqiad.wmnet with reason: host reimage * 07:42 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster1005.eqiad.wmnet to drbd * 07:38 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1256.eqiad.wmnet with reason: host reimage * 07:35 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp6010.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:35 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P<nowiki>{</nowiki>cp6002.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:35 slyngshede@cumin1003: cookbooks.sre.cdn.roll-reboot finished rebooting cp6002.drmrs.wmnet * 07:27 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster1005.eqiad.wmnet to drbd * 07:24 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P<nowiki>{</nowiki>cp6002.drmrs.wmnet<nowiki>}</nowiki> and A:cp * 07:24 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1256.eqiad.wmnet with OS trixie * 07:22 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1256: Upgrading db1256.eqiad.wmnet * 07:21 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1256: Upgrading db1256.eqiad.wmnet * 07:21 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade * 07:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to plain * 07:18 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to plain * 07:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbproxy1025.eqiad.wmnet with reason: Rebooting * 07:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to drbd * 06:54 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to drbd * 06:53 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to plain * 06:52 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to plain * 06:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to drbd * 06:42 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists1004.wikimedia.org * 06:40 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org * 06:39 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1003.eqiad.wmnet * 06:34 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org * 06:34 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host lists1004.wikimedia.org * 06:33 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host vrts1003.eqiad.wmnet * 06:24 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd1003.eqiad.wmnet to drbd * 06:23 arnaudb@cumin1003: END (FAIL) - Cookbook sre.gerrit.reboot-gerrit (exit_code=99) Rebooting Gerrit on gerrit2003 * 06:22 arnaudb@cumin1003: START - Cookbook sre.gerrit.reboot-gerrit Rebooting Gerrit on gerrit2003 * 06:15 marostegui@dns1004: END - running authdns-update * 06:14 marostegui: Failover m2-master [[phab:T426633|T426633]] * 06:13 marostegui@dns1004: START - running authdns-update * 05:39 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc1012 from dbctl [[phab:T426930|T426930]]', diff saved to https://phabricator.wikimedia.org/P92728 and previous config saved to /var/cache/conftool/dbconfig/20260521-053858-marostegui.json * 05:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc2 [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92727 and previous config saved to /var/cache/conftool/dbconfig/20260521-053000-marostegui.json * 05:29 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc1022 to pc2 master [[phab:T418973|T418973]]', diff saved to https://phabricator.wikimedia.org/P92726 and previous config saved to /var/cache/conftool/dbconfig/20260521-052905-marostegui.json * 05:21 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc1012.eqiad.wmnet with reason: Cloning * 02:41 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on planet1003.eqiad.wmnet with reason: debug wip * 02:11 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 29s) * 02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:29 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1027.eqiad.wmnet * 01:22 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs1027.eqiad.wmnet * 00:55 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: [[phab:T426560|T426560]] - bking@cumin2002 == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> em8bbadk9ehi0u9evi6p6sra69izkuo 498 443628 2426623 2425670 2026-06-13T13:01:03Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27467463956 (https://github.com/cluebotng/component-configs/commits/3dc535380a54d2290621b9d585a5018fdc4669a2) 2426623 wikitext text/x-wiki === 2026-06-13 === * 13:01 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27467463956 (https://github.com/cluebotng/component-configs/commits/3dc535380a54d2290621b9d585a5018fdc4669a2) === 2026-06-10 === * 15:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125849 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b) * 13:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27279260980 (https://github.com/cluebotng/component-configs/commits/83e16c74dca286ed8f7104d49a271dee18c41854) * 12:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277788838 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884) * 12:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276699371 (https://github.com/cluebotng/component-configs/commits/8be9293c4b541d74d39482efd21163eb36cda6bd) * 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276510139 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1) * 12:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275411819 (https://github.com/cluebotng/component-configs/commits/869dfb8d1487914e36184a9d1c5aae1e26dbba01) * 12:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275397723 (https://github.com/cluebotng/component-configs/commits/33d203fb0e6b88ac6dc34e82ee630f7d4e6fdb56) * 12:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27275339407 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11) === 2026-06-09 === * 17:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27225356702 (https://github.com/cluebotng/component-configs/commits/9534cf81437fb2c268eb00e4145978dddbf6322e) === 2026-06-08 === * 12:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27138156804 (https://github.com/cluebotng/component-configs/commits/4677023bc60821948b76a89b2968d9fa3db267d4) === 2026-06-05 === * 14:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27021601458 (https://github.com/cluebotng/component-configs/commits/d4efd5a504c17f41f2d280dabcb635f9c4f07000) === 2026-06-04 === * 22:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26982622098 (https://github.com/cluebotng/component-configs/commits/eb9cf1341e6e78387424cb9070a3aec87971e54d) * 22:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26982622098 (https://github.com/cluebotng/component-configs/commits/eb9cf1341e6e78387424cb9070a3aec87971e54d) === 2026-06-01 === * 15:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26762978145 (https://github.com/cluebotng/component-configs/commits/9a088c9b8375555c696948825fff7700458b4254) * 13:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26757972034 (https://github.com/cluebotng/component-configs/commits/4790ebea51ebfbd67e51894987e6273e5940cbf1) === 2026-05-31 === * 17:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864800 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3) * 17:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719862186 (https://github.com/cluebotng/component-configs/commits/8e921d9dd24ae32755f893363b9dfa897cf71c25) * 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719590163 (https://github.com/cluebotng/component-configs/commits/9fc63fa520f0c2c3790d3d3236682dbb83382a9f) * 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719575482 (https://github.com/cluebotng/component-configs/commits/9bb586c7c04b2a5848b2ebc287497a81506f2d1d) === 2026-05-29 === * 00:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26609639432 (https://github.com/cluebotng/component-configs/commits/8c2fccaaae357774084389157d9a305e72eccb20) === 2026-05-28 === * 18:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26592792702 (https://github.com/cluebotng/component-configs/commits/a7971b7e286e177862e5318c40b0d4d868efc7c8) === 2026-05-21 === * 20:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26251627543 (https://github.com/cluebotng/component-configs/commits/96f9184e66a6e4b35a49f02940a213125945b056) === 2026-05-19 === * 00:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26068076424 (https://github.com/cluebotng/component-configs/commits/f7db7f6fff0d4d6dd451b5f92e75ba755a74129c) === 2026-05-17 === * 08:55 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25986406214 (https://github.com/cluebotng/component-configs/commits/be3bb145d2803394cd0b7dbd8ae1775ac9b7cd09) === 2026-05-14 === * 18:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25878752565 (https://github.com/cluebotng/component-configs/commits/21e928fa1870ddaf5fae15afc6f92aa3cb3fb970) === 2026-05-13 === * 02:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773616046 (https://github.com/cluebotng/component-configs/commits/0fd601991775a24b437113d09438e74b996c991b) === 2026-05-12 === * 10:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25727821548 (https://github.com/cluebotng/component-configs/commits/91aefb7d53013ad152bb721f71980dd26170f297) * 09:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842111 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c) * 09:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724562306 (https://github.com/cluebotng/component-configs/commits/bd0e188642746ab949ec3762676ac730afff1c17) * 08:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723480578 (https://github.com/cluebotng/component-configs/commits/25c0a1035daa67c2225c0f7f7a414ff5cfb6ed2a) * 08:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723280492 (https://github.com/cluebotng/component-configs/commits/51d7c1919958a7672895885cbb3a1061934d2788) === 2026-05-07 === * 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25505647057 (https://github.com/cluebotng/component-configs/commits/874d7f6f407fc9a3995f52f40312cd7d3a712176) === 2026-05-06 === * 18:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25453754913 (https://github.com/cluebotng/component-configs/commits/92f164d1ab158aea1f76cd0a787f33ffe4017e85) === 2026-05-04 === * 07:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25252392662 (https://github.com/cluebotng/component-configs/commits/7352cd4f730ca9f5c276772f0b338230989feef4) === 2026-05-02 === * 12:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25252392662 (https://github.com/cluebotng/component-configs/commits/7352cd4f730ca9f5c276772f0b338230989feef4) === 2026-04-24 === * 22:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914278570 (https://github.com/cluebotng/component-configs/commits/23a4b53f3d291b0c750d44a2c0a661333307786d) * 00:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24865518502 (https://github.com/cluebotng/component-configs/commits/6e953f35fdca38226cde9ea7280f948f34242881) === 2026-04-23 === * 23:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24863962598 (https://github.com/cluebotng/component-configs/commits/07b9ad8616853af2ed49f96de6e55c14e3faabe0) * 09:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24827343249 (https://github.com/cluebotng/component-configs/commits/218a7fa56222cf3e98b642eebbf6b1b2b273a92d) * 09:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24826859245 (https://github.com/cluebotng/component-configs/commits/22ea2eb955d25f4a15e6b234e72a24bc01127a79) * 09:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24826656498 (https://github.com/cluebotng/component-configs/commits/90aa39b3e81f19f42d08d5ec6131ba09c68bd786) === 2026-04-17 === * 00:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24540678933 (https://github.com/cluebotng/component-configs/commits/26849735bbefbe218cbe0ce41db5a35941798c7b) === 2026-04-14 === * 21:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24422836823 (https://github.com/cluebotng/component-configs/commits/10f4f0f81e169fac55d056176a273966c8160078) === 2026-04-10 === * 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442412 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f) * 15:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24249897963 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3) * 14:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247614873 (https://github.com/cluebotng/component-configs/commits/30bda68a3ea7a1674d174e43cc8651d301c7485c) * 14:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247570676 (https://github.com/cluebotng/component-configs/commits/93ced49392782bf65e34d13f10cbeaafa760f115) === 2026-04-09 === * 18:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093216 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253) === 2026-03-27 === * 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656823 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4) * 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659535101 (https://github.com/cluebotng/component-configs/commits/b825610c7f23870a8561b00b5f8546b107643015) === 2026-03-25 === * 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23535729395 (https://github.com/cluebotng/component-configs/commits/c1468d960041cd66ab50902f344fec1ac65ddcad) === 2026-03-21 === * 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383557597 (https://github.com/cluebotng/component-configs/commits/0b67216b47074dd5d1d279dde0aa9144b243cf01) * 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640340 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa) * 16:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383557597 (https://github.com/cluebotng/component-configs/commits/0b67216b47074dd5d1d279dde0aa9144b243cf01) * 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383551615 (https://github.com/cluebotng/component-configs/commits/ffff74b90a37a0c6bdd565128d3c11ae195e0763) * 14:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23381633193 (https://github.com/cluebotng/component-configs/commits/ac40c461942f4541b640a32ff0141268418abc12) === 2026-03-19 === * 20:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23315406679 (https://github.com/cluebotng/component-configs/commits/fd07020c08545c83ab35667616a26081966648df) === 2026-02-15 === * 10:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22034380291 (https://github.com/cluebotng/component-configs/commits/842b50dc5d3160000352a25c5fdf09ea88ebf3eb) === 2025-11-11 === * 15:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642865 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c) * 15:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19270642865 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c) * 15:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270301650 (https://github.com/cluebotng/component-configs/commits/f28dcaec8c5882b4a1b7d861fe7f5e400312a5b4) * 13:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266856634 (https://github.com/cluebotng/component-configs/commits/b0e9170597a778654185be762c580e2a6e19492f) === 2025-11-06 === * 01:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19121350626 (https://github.com/cluebotng/component-configs/commits/80f69f0ab7b09c2e3e5e208d847a954cb1975bc6) === 2025-11-05 === * 19:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19113995211 (https://github.com/cluebotng/component-configs/commits/586f2c46dcbbb09a9f7926e991bc5fbe45f4a1e9) * 18:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19112506240 (https://github.com/cluebotng/component-configs/commits/3f51ec3aa53d1378883a9dc973716e57c283d26c) * 16:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19108852168 (https://github.com/cluebotng/component-configs/commits/24f3dc9fe5e2211d861c754a4b9342a6127f4a4a) * 12:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19102217917 (https://github.com/cluebotng/component-configs/commits/24f3dc9fe5e2211d861c754a4b9342a6127f4a4a) === 2025-10-29 === * 15:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18912872623 (https://github.com/cluebotng/component-configs/commits/3281794d8d1d2e17d9e9859c6f6f7ae3c5216eda) === 2025-10-23 === * 12:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18748267293 (https://github.com/cluebotng/component-configs/commits/bc8f1b883d0d53edf08bea5e5319ee7ee0b4fb82) === 2025-10-08 === * 15:47 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.4.1 * 11:23 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.4.0 === 2025-09-29 === * 16:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101448 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5) * 09:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18092350259 (https://github.com/cluebotng/component-configs/commits/283965c9240c0c5a72e0ea1203439583935295cb) === 2025-09-27 === * 13:07 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.3.0 === 2025-09-26 === * 18:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18046649036 (https://github.com/cluebotng/component-configs/commits/07b907ff75f0289f350549bae5e75bf4e91c91ca) * 12:52 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.4 * 12:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18038096319 (https://github.com/cluebotng/component-configs/commits/e10e601b4fb06b3fd97856ef86a30e5391fb4f17) * 12:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18037926507 (https://github.com/cluebotng/component-configs/commits/4950150f14c22c0a7d3df1739fa5537aeba4157d) * 12:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18037156332 (https://github.com/cluebotng/component-configs/commits/a51fe109bfad3e2df5aa8e89b837a951bf8ad2cf) === 2025-09-25 === * 21:20 wmbot~damian@tools-bastion-15: bot deployed @ v1.2.2 * 21:20 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.2 * 19:17 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.1 * 19:16 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.0 * 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18017003184 (https://github.com/cluebotng/component-configs/commits/e9fc8d46ac1a0ff0ac6203458fa171c6430492ce) * 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18016894528 (https://github.com/cluebotng/component-configs/commits/cfc9adc9516df0f11c8b6d1df68232d0a46cb4eb) * 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998407 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c) * 17:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015392766 (https://github.com/cluebotng/component-configs/commits/61a7ceac210077c3d81bc064c37f8d8668cc2cfb) * 17:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015392766 (https://github.com/cluebotng/component-configs/commits/61a7ceac210077c3d81bc064c37f8d8668cc2cfb) * 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801881 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c) * 16:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221965 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d) * 16:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782359 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e) * 16:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677281 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a) * 16:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531091 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f) === 2025-09-22 === * 19:08 wmbot~damian-scripts@tools-bastion-15: report deployed @ refs/tags/1.2.3 * 19:04 wmbot~damian-scripts@tools-bastion-15: report deployed @ refs/tags/1.2.2 * 19:02 wmbot~damian-scripts@tools-bastion-15: report deployed @ refs/tags/1.2.1 === 2025-08-29 === * 14:30 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.2.0 * 14:30 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main * 00:10 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.1.1 === 2025-08-15 === * 21:14 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.1.0 * 20:58 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/1.0.34 * 12:58 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.33 * 00:23 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.32 * 00:08 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.31 === 2025-08-14 === * 16:13 wmbot~damian@tools-bastion-13: irc-relay deployed @ v1.1.12 * 12:47 wmbot~damian@tools-bastion-13: core deployed @ v0.0.2 * 12:43 wmbot~damian@tools-bastion-13: core deployed @ v0.0.2 === 2025-08-11 === * 13:06 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.1.3 * 12:37 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.30 === 2025-08-10 === * 17:42 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.1.2 * 17:37 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.29 * 16:35 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.1.1 === 2025-08-08 === * 15:37 wmbot~damian@tools-bastion-12: Updated ci-execute-fabric to use dedicated unix account, dropped key from human account * 15:36 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.28 * 15:33 wmbot~damian@tools-bastion-13: report deployed @ refs/tags/v1.0.28 === 2025-08-07 === * 15:24 wmbot~damian@tools-bastion-13: report deployed @ v1.0.27 === 2024-03-21 === * 10:13 dcaro: fixed .lighttpd.conf file to add port and remove socket === 2021-07-23 === * 16:40 majavah: stop cbng_relay grid job, still having issues with irc connection - [[phab:T274871|T274871]] === 2020-01-13 === * 20:53 wm-bot: <root> Restarted webservice to fix broken registration with the front proxy ([[phab:T242538|T242538]]) === 2019-03-07 === * 05:06 bd808: Killed cbng_bot job stuck in deletion state with 4000+ zombie child processes ([[phab:T217817|T217817]]) <noinclude>[[Category:SAL]]</noinclude> i6unucvxyynghvqwauf4z6g0yvk4omd 498 443946 2426626 2426468 2026-06-13T16:13:03Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 5fde4e8ec2 (use authenticated session for Wikifunctions calls: T349966, T423542) 2426626 wikitext text/x-wiki === 2026-06-13 === * 16:13 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|5fde4e8ec2}} (use authenticated session for Wikifunctions calls: [[phab:T349966|T349966]], [[phab:T423542|T423542]]) === 2026-06-11 === * 19:59 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|3358ae7e8c}} (l10n updates: ar, fi, frp, it, ko, nl, sk, zh-hans; using the previously deployed {{GENDER:}} support in duplicates-instructions) === 2026-06-06 === * 19:36 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|22a11f62c4}} (add {{GENDER:}} support to duplicates-instructions message) === 2026-06-04 === * 19:56 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|3c09a6963f}} (l10n updates: nl) === 2026-06-03 === * 17:56 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fd3d67655e}} (bump mwoauth2) === 2026-05-29 === * 17:23 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|11f47f0f24}} (install mwoauth2 as package) === 2026-05-27 === * 21:34 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|994cfa51b3}} (make mwoauth2 strictly typed) * 18:34 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|73fcbaa464}} (extract mwoauth2 module) === 2026-05-25 === * 13:51 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|c163567ccd}} (l10n updates: nb) === 2026-05-21 === * 21:45 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|97d0866c92}} (treat outdated [OAuth 1] access tokens more robustly) * 19:18 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|f0d7740038}} (l10n updates: ko, ta, vi) === 2026-05-20 === * 18:55 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|dfc101f3e0}} (Python 3.14, aka 𝜋thon) * 18:50 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|5b92917005}} (upgrade dependencies) * 18:46 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|a42e65b596}} (configure Gunicorn --forwarded-allow-ips) === 2026-05-19 === * 17:59 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|43b18b8299}} (prevent OAuthLib InsecureTransportError more strongly) * 17:52 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|80583945a8}} (migrate to OAuth 2) === 2026-04-25 === * 23:34 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|4f201ca05a}} (update gunicorn config) * 19:30 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|ab0ee1c0ce}} (Russian imperfective verbs) === 2026-04-13 === * 17:46 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fe623b17c0}} (l10n updates: hi, ms-arab) === 2026-04-06 === * 12:36 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|68d0059881}} (l10n updates: ms-arab) === 2026-03-26 === * 17:21 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|a2dde3c334}} (l10n updates: pa) === 2026-03-23 === * 19:02 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|03da8bff9b}} (l10n updates: kea) === 2026-03-19 === * 13:17 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|cdc85da9aa}} (l10n updates: kea) === 2026-03-16 === * 13:26 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|83b82c6ea4}} (l10n updates: ary, ga) === 2026-03-14 === * 14:55 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|2e36547a31}} (Moroccan Arabic templates – part of Wikimedia Hackathon Northwestern Europe 2026 \o/) === 2026-03-12 === * 16:31 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|0282ad0864}} (l10n updates: kea) === 2026-03-09 === * 13:22 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|f0983e0fcf}} (l10n updates: kea) === 2026-02-10 === * 21:15 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|ca57a0da6a}} (noop – update a test) === 2026-02-09 === * 18:40 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|49f7ed4319}} (l10n updates: el) === 2026-02-02 === * 13:44 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|74bb77b1b4}} (l10n updates: el, pl) === 2026-01-22 === * 12:56 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|4bdcac2b61}} (l10n updates: pa) === 2026-01-18 === * 17:45 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|3ba43046ee}} (avoid changing lemma if not necessary) === 2026-01-05 === * 12:55 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fa52e74def}} (l10n updates: id) === 2026-01-03 === * 17:03 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|2825501536}} (l10n updates: ca, it, pl, sv) === 2025-12-22 === * 19:08 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|40105a8e8b}} (l10n updates: it, vi) === 2025-12-11 === * 21:15 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|5c7dd452be}} (l10n updates: mk) === 2025-12-01 === * 13:01 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|54c6749c45}} (l10n updates: fi) === 2025-11-19 === * 18:55 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|7f658a6675}} (update Bootstrap) === 2025-11-18 === * 20:04 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|c74254856c}} (fix skiplink visibility) === 2025-11-17 === * 13:06 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|34ee7bf7ac}} (l10n updates: cy) === 2025-11-13 === * 12:46 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|808bd196dc}} (l10n updates: anp) === 2025-11-10 === * 19:04 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|730ae77335}} (drop typing_extensions) * 18:58 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|36b8ab588c}} (upgrade dependencies) * 18:47 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|422cb1f05c}} (l10n updates: frp, ro) === 2025-10-27 === * 14:15 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|40579b07e7}} (l10n updates: el, tg) === 2025-10-20 === * 12:30 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|9d7035ced8}} (l10n updates: frp) === 2025-10-13 === * 18:10 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|ffc43e9936}} (l10n updates: lb) === 2025-09-29 === * 18:23 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|f458d2938d}} (l10n updates: ko-kp) === 2025-09-11 === * 18:34 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|afc726918e}} (l10n updates: rki) === 2025-09-04 === * 15:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c35d575859}} (l10n updates: nb) === 2025-09-01 === * 18:40 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1e9e985c5d}} (l10n updates: vi) === 2025-08-25 === * 17:12 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|457dd44066}} (l10n updates: aig, pt) === 2025-08-24 === * 22:40 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3669a5db51}} (upgrade dependencies, including PyMySQL 1.1.2 with Python 3.13 compatibility) === 2025-08-21 === * 19:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3776ee4000}} (l10n updates: yue-hant) === 2025-08-18 === * 19:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|9b5fc1cef3}} (Portuguese Wikifunctions) * 19:07 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2886086be9}} (add missing wikifunctions_intro to german-noun-masculine) * 17:57 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|243f584c59}} (l10n updates: ar, tg, yue-hant) === 2025-08-14 === * 16:25 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d4ef0aa38c}} (l10n updates: yue-hant) === 2025-08-07 === * 19:50 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ebeea040cb}} (l10n updates: yue-hant) === 2025-07-31 === * 17:24 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6b1e8c15ae}} (l10n updates: pt, pt-br, sl) === 2025-07-24 === * 19:15 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ece1469a65}} (l10n updates: pt) === 2025-07-17 === * 19:29 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|672ec5bee8}} (l10n updates: yue-hant, zh-hant) === 2025-07-13 === * 15:50 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3c977ccc7b}} (specify .python-version) * 14:46 lucaswerkmeister: disregard the previous message, wrong tool 🤦 * 14:46 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["CONSUMER_KEY"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_KEY === 2025-07-12 === * 21:01 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|336dd318ca}} (upgrade to Python 3.13) * 20:56 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|dd03e95876}} (update documentation; no-op deployment, just to test the new buildservice procedure and put it in a single command) * 20:50 wmbot~lucaswerkmeister@tools-bastion-13: cp www-unused-tool-now-runs-on-buildservice/python/src/service.template . * 20:49 wmbot~lucaswerkmeister@tools-bastion-13: mv www www-unused-tool-now-runs-on-buildservice * 20:47 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e6d259028e}} (successful migration to buildservice) * 18:09 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|0bf37ac8d8}} (tried but failed to migrate to build service [OSError: No username set in the environment], will try again later, for now running in python3.11 again) * 17:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f8033bda4a}} (read config from envvars) * 17:41 lucaswerkmeister: commented out config.yaml, should use envvars instead * 17:41 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["SECRET_KEY"]))' {{!}} toolforge envvars create TOOL_SECRET_KEY * 17:40 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["CONSUMER_SECRET"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_SECRET * 17:40 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["CONSUMER_KEY"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_KEY * 17:16 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|15261afefb}} (change config keys to uppercase to work around [[phab:T374780|T374780]]) === 2025-07-10 === * 19:59 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8c32f8b90c}} (l10n updates: hu) === 2025-07-07 === * 06:33 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|cc28ff0494}} (l10n updates: et, it, nn, pt-br, ru) === 2025-06-16 === * 17:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c7b450ab94}} (update code for newer mwapi version) === 2025-06-11 === * 12:23 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|cae8c3c341}} (upgrade dependencies, including toolforge 6.1.0; use toolforge.load_private_yaml() from [[phab:T333728|T333728]]) === 2025-05-31 === * 13:53 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|706110e863}} (l10n updates: da, lb) === 2025-05-13 === * 16:58 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8a583bf6ff}} (l10n updates: tg) === 2025-05-06 === * 17:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|7349295f62}} (l10n updates: el) * 17:24 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e969f66351}} (update absolute_construction item ID) === 2025-04-22 === * 23:10 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d9516b6b1c}} (Quechua verb Wikifunctions) === 2025-04-21 === * 18:20 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c6d552c9c3}} (upgrade dependencies, including toolforge-i18n 0.1.2) === 2025-04-19 === * 10:56 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5425b40c0f}} (l10n updates: es) === 2025-04-14 === * 19:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ae10863f8f}} (l10n updates: af) === 2025-04-07 === * 18:01 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e106b7b684}} (Quechua verbs + l10n updates: es, pa, qu, zh-hant) === 2025-04-04 === * 19:48 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a377a0be8c}} (remove unneeded CSS) === 2025-03-29 === * 21:16 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|98e408e5a6}} (Russian perfective verbs) === 2025-03-15 === * 11:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ab6621b22d}} (l10n updates: ar) === 2025-03-11 === * 20:35 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d6def84813}} (l10n updates: lb) === 2025-02-21 === * 20:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|81611bc5dc}} (l10n updates: pa, tr) === 2025-02-04 === * 21:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2ccb28ad17}} (l10n updates: lb) === 2025-01-24 === * 10:21 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|223cafa209}} (l10n updates: ms) === 2025-01-09 === * 21:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|cebad0e4dd}} (l10n updates: ia, pa) === 2025-01-06 === * 20:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e7e3f2a500}} (l10n updates: cs, he) === 2024-12-21 === * 22:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|eb9d0ae3c2}} (l10n updates: lb, pa; also upgrade dependencies, including Flask 3.1.0 and Jinja2 3.1.5) === 2024-12-12 === * 22:13 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5ffdfb2c55}} (l10n updates: he, nl) === 2024-11-18 === * 19:47 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3933dbfa7f}} (l10n updates: af, ar, de, fr, gl, he, krc, mk, pa, sk, zh-hans); manually restored sh-latn ([[phab:T379188|T379188]]) === 2024-11-04 === * 17:32 wmbot~lucaswerkmeister@tools-bastion-13: webservice stop; webservice start # [[phab:T378976|T378976]] === 2024-11-02 === * 16:54 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a6768b885c}} (add setting for using Wikifunctions) * 14:01 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4fdd9491ee}} (improve Wikifunctions UI) * 09:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f28c3414ad}} (upgrade dependencies, including Werkzeug 3.1.0); also upgraded pip from 24.2 to 24.3.1 === 2024-10-25 === * 19:30 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8cdbda6ce3}} (upgrade dependencies, including Werkzeug 3.0.6) === 2024-10-13 === * 11:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|bfcaca2fa3}} (upgrade dependencies, including MarkupSafe 3.0) === 2024-10-03 === * 16:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e6377a9095}} (upgrade dependencies, including toolforge_i18n 0.1.1 and Werkzeug 3.0.4) * 13:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a81b469204}} (l10n updates: ms-arab) === 2024-09-26 === * 15:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|96f45731db}} (l10n updates: ar, ms-arab) === 2024-09-11 === * 21:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|38b3b281ed}} (fix two ZIDs for Breton templates) === 2024-09-01 === * 14:32 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|53a1efcc14}} (l10n updates: cy, uk) === 2024-08-18 === * 12:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|309b33b80b}} (l10n updates: pl, tg) * 12:02 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6deace1e36}} (Italian masculine+feminine nouns, dependency upgrades) === 2024-08-12 === * 18:19 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|246f9d26da}} (l10n updates: tg) === 2024-08-05 === * 13:34 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e3448958a0}} (upgrade toolforge_i18n to 0.0.7) === 2024-07-31 === * 19:15 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4775170045}} (upgrade toolforge_i18n to 0.0.6; also upgrade pip to 24.2) === 2024-07-26 === * 21:11 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|bb61fc3c89}} (l10n updates: vi [no actual translation changes, one addition to the authors, presumably their edit got reverted]) === 2024-07-22 === * 18:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|13c4824e3a}} (change Babel code of kaa from kk to uz) === 2024-07-21 === * 18:12 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e856c9b2d2}} (upgrade toolforge_i18n to 0.0.5; also upgrade pip to 24.1.2) === 2024-07-08 === * 18:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d6fa2d82b8}} (l10n updates: ja) === 2024-07-07 === * 18:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f3b3981ec9}} (upgrade toolforge_i18n to 0.0.2; also upgrade pip from 24.0 to 24.1.1) === 2024-07-05 === * 12:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1013a7234d}} (l10n updates: ar, de, uk) === 2024-06-18 === * 19:05 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8530f5f235}} (l10n updates: eo, fa, kaa, lb) === 2024-06-15 === * 13:58 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|9cb9b3dfde}} (install toolforge_i18n from PyPI) === 2024-06-07 === * 09:06 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|253d1b0f45}} (l10n updates: pa) === 2024-05-26 === * 13:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|48a5585566}} (support opting out of Wikifunctions mode) === 2024-05-20 === * 13:34 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4d952df88b}} (l10n updates: ms) === 2024-05-13 === * 18:19 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1c3d80a5e6}} (l10n updates: eu, zh-hans) === 2024-05-11 === * 12:50 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|bfccf1614c}} (more Hebrew verb templates) === 2024-05-09 === * 15:40 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5b88dd1ce1}} (improve toolforge_i18n and upgrade dependencies for newer Babel and Werkzeug) === 2024-05-06 === * 17:04 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|c5618f5968}} (set bot flag in bulk mode) * 15:43 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|8fa2740a72}} (README update, pulled without webservice restart) === 2024-05-05 === * 11:47 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|400cc9cb84}} (update Hebrew pa'al verbs) * 11:03 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|19c8210d68}} (Hebrew pa'al verbs) === 2024-05-04 === * 12:17 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|deb5b1c44e}} (extract toolforge_i18n library: [[phab:T363626|T363626]]) === 2024-05-03 === * 17:08 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|89c98da81f}} (upgrade dependencies for Python 3.12 compat; also upgraded pip<nowiki>{</nowiki>,-tools<nowiki>}</nowiki> and wheel while I’m at it) === 2024-04-22 === * 20:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1be060cd5c}} (l10n updates: ja) === 2024-04-18 === * 19:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f1a2cd1995}} (use public WikiLambda API) === 2024-04-17 === * 19:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e5d2281cea}} (l10n updates: krc) * 18:13 wmbot~lucaswerkmeister@tools-bastion-13: pulled {{Gerrit|fa6c094165}} (templates CC BY-SA 3.0 → 4.0; no webservice restart needed) === 2024-04-08 === * 17:58 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|559eb5bc47}} (make session permanent after login) === 2024-04-06 === * 13:35 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|1569542ce6}} (l10n updates: el, fa, zh-hant) === 2024-03-24 === * 12:21 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|b630198d56}} (l10n updates: fi, ms-arab) === 2024-03-15 === * 19:41 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|272a303c09}} (Danish adverbs) * 16:33 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|8f4985e682}} (improve tests; should have no production impact but I pulled+restarted anyway ^^) === 2024-03-10 === * 18:40 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|c62a9c1927}} (Maltese templates, including support for non-first forms to be the lemma: Maltese nouns have the third person singular as the lemma) * 12:42 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|bf88439696}} (l10n updates: fi, ko) === 2024-03-04 === * 18:12 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|e7a659802c}} (l10n updates: ar, io, lb) === 2024-03-03 === * 00:26 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|4106259494}} (l10n updates: ht, hu) === 2024-02-28 === * 18:50 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|3030faaa3c}} (health-check-path, [[phab:T341919|T341919]]) === 2024-02-23 === * 20:21 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|968078dbcd}} (l10n updates: hu, lt) [relog from 19:35 UTC, stashbot had problems] === 2024-02-17 === * 10:51 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|f88f2445fc}} (Esperanto adjective+verb Wikifunctions) === 2024-02-13 === * 18:51 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|85b6ec6534}} (l10n updates: ja, kaa) === 2024-02-07 === * 17:59 wmbot~lucaswerkmeister@tools-sgebastion-10: started webservice again (and patched the startup probe into it); took a while to come up but now it seems to be working * 17:49 wmbot~lucaswerkmeister@tools-sgebastion-10: stopped webservice, restart wasn’t working so let’s try harder * 17:45 wmbot~lucaswerkmeister@tools-sgebastion-10: restarted webservice, log was full of various errors === 2024-02-06 === * 20:39 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|344fd43224}} (update Breton noun Wikifunctions) === 2024-01-31 === * 19:13 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|604b43e316}} (l10n updates: it) === 2024-01-26 === * 19:03 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|249d9da0b7}} (l10n updates: id, kaa, ru, th) * 00:22 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|886d99636e}} (more Esperanto noun Wikifunctions) === 2024-01-22 === * 18:34 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|0b062cafa9}} (Norwegian language name templates) === 2024-01-13 === * 15:51 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|d24dc99256}} (l10n updates: ar) === 2024-01-07 === * 13:08 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a97ab796ea}} (wikifunctions: first form from lemma, if missing) === 2024-01-06 === * 16:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ea6b02ac57}} (Wikifunctions returning lists, Z11991→Z12689) === 2024-01-04 === * 12:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|82f5578b9a}} (l10n updates: ca, de, pl) === 2023-12-30 === * 15:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5baa3871d0}} (l10n updates: lb, zh-hans) === 2023-12-28 === * 10:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|45b698823a}} (update Italian adjectives) * 10:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4e68d80748}} (i18n updates: uk) === 2023-12-17 === * 18:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7611d4e980}} (l10n updates: ia, krc, sv) === 2023-12-11 === * 18:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|424615e192}} (l10n updates: de, krc, lb, nl, pnb) === 2023-12-09 === * 16:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fdc7c853c4}} (update Breton noun Wikifunctions) === 2023-12-05 === * 19:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|95ee032c68}} (l10n updates: ca, hno, io, it, pnb, sl, tr; i18n test improvements and fixes) === 2023-12-01 === * 19:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ba19a1cd5f}} (l10n updates: ja, sk, zh-hans) * 19:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7acef657d0}} (update Croation noun Wikifunctions) === 2023-11-29 === * 17:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|54a614fd41}} (fix some spacing) === 2023-11-25 === * 12:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|171fc2ea54}} (l10n updates: br) === 2023-11-19 === * 16:36 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0416376e58}} (German masculine noun Wikifunctions) * 15:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|11e7d12745}} (one more set of German neuter noun Wikifunctions) * 13:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|442f510a5b}} (German neuter noun Wikifunctions) === 2023-11-18 === * 17:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8c123e032e}} (l10n updates: br, he, ko) === 2023-11-12 === * 17:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cc2cf0ceaf}} (l10n updates: bn, fa, fr, gl, it, lb, mk, nb, vi, zh-hans, zh-hant; yue removed, existing settings are automatically replaced with zh-hant) === 2023-11-04 === * 18:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|203bc87b5b}} (more German feminine noun Wikifunctions – m/n will follow later) * 12:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bfa1ad40e0}} (first German Wikifunctions: feminine noun -(e)n plural) * 10:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|365c7e2814}} (cache Wikifunctions results) === 2023-11-01 === * 19:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|240a228f49}} (tests for Wikifunctions, pulled without webservice restart) * 18:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|92a91137e6}} (Wikifunctions for Breton nouns) === 2023-10-30 === * 19:10 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bea713bc0c}} (l10n updates: br) * 00:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f33e56597c}} (update French Wikifunctions button label) === 2023-10-29 === * 17:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cca1b1af23}} (Wikifunctions support in edit mode) * 16:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b5af35ab2b}} (fix Croatian feminine noun instrumental plural) * 16:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|052ba84de7}} (fix crash for users without Wikifunctions account) * 15:49 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5c3dc0dd6d}} (experimental Wikifunctions for Esperanto nouns, nominative plural only) * 14:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0ab3c10890}} (fix Wikifunctions buttons lang= and dir=) * 14:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5657d03fbb}} (experimental Wikifunctions for French nouns) * 14:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a64b857485}} (experimental Wikifunctions for Croatian nouns) * 14:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|40b0df49ee}} (experimental Wikifunctions support – happy birthday Wikidata 🎉) === 2023-10-28 === * 22:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c1f7a335e8}} (fix input patterns) === 2023-10-25 === * 17:59 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cdb1d34e11}} (Werkzeug 3.0.1) === 2023-10-20 === * 17:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|df7cf04757}} (i18n updates: io, ms-arab) === 2023-10-10 === * 19:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ad16425ee2}} (l10n updates: nl, uk, zh-hans) === 2023-10-06 === * 17:36 wm-bot: <lucaswerkmeister> deployed {{Gerrit|72e12c5a2c}} (l10n updates: zh-hans) + remove hardcoded support for Karai-karai now that MediaWiki has it === 2023-10-01 === * 17:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|216afb45fa}} (update dependencies, Flask+Werkzeug 3) === 2023-09-24 === * 13:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e5ae3295bb}} (Babel language code of Aragonese, to silence log warnings) * 13:29 wm-bot: <lucaswerkmeister> deployed {{Gerrit|45aa8fe43b}} (Danish proper nouns) === 2023-09-22 === * 16:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|72c20b3b3e}} (l10n updates: cs, kai [new, with temporary hacks], tr, zh ⇒ zh-hans) === 2023-09-04 === * 16:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|85d978855f}} (Italian adverbs) === 2023-08-28 === * 18:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|48e3991eb6}} (fix typo in armenian-noun-singulare-tantum) === 2023-08-27 === * 14:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ea49f8c2c7}} (update dependencies) * 13:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|05522cee84}} (update Italian) === 2023-08-24 === * 17:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c19c9624ba}} (l10n updates: ca, fa, io) === 2023-08-12 === * 11:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e0cf031e70}} (l10n updates: it) === 2023-08-08 === * 18:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|56acd0944a}} (l10n updates: tr) === 2023-07-27 === * 12:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5d374c3787}} (l10n updates: ban, de, gl) === 2023-07-19 === * 12:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4fa53fae89}} (l10n updates: pt-br) === 2023-07-18 === * 08:03 wm-bot: <lucaswerkmeister> deployed {{Gerrit|474e48d752}} (update Breton grammatical feature) === 2023-07-15 === * 12:03 wm-bot: <lucaswerkmeister> pip-sync (i.e., actually install dependencies in the new venv, which I completely forgot to do earlier) * 11:31 wm-bot: <lucaswerkmeister> kubectl patch deployment lexeme-forms --patch-file patch-add-startup-probe.yml * 11:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|02f72f81a2}} (Python 3.11) === 2023-07-13 === * 13:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d7fea069ba}} (l10n updates: pl) === 2023-07-10 === * 17:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|42679bb5dc}} (l10n updates: yue) === 2023-07-09 === * 14:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|78711ad373}} (l10n updates: ms) === 2023-07-02 === * 13:00 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4e2653cf19}} (revert recent punjabi-noun-masculine-guru change) * 12:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c9e84dfb8d}} (add separators to Dutch nouns) === 2023-06-30 === * 18:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1ed453b5d5}} (l10n updates: sh → sh-latn, tt → tt-cyrl, [[phab:T336606|T336606]]) === 2023-06-27 === * 20:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fe5983c571}} (l10n updates: ba) === 2023-06-25 === * 14:00 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3ad131b7bf}} (Aragonese common nouns) === 2023-06-24 === * 09:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|213bfabfb4}} (underline links on hover again) === 2023-06-22 === * 20:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|63c042d9b3}} (l10n updates: it) === 2023-06-20 === * 18:27 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7081d2769e}} (support language fallback and ?uselang) * 17:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3e76345eb5}} (l10n updates: ba, id, nb, xmf) === 2023-06-18 === * 11:36 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bebc116e22}} (Bootstrap 5) * 11:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d53e455ef7}} (update Malayalam nouns and add adjective template) === 2023-06-16 === * 17:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|248590aeb0}} (l10n updates: ba, id, pl) === 2023-06-13 === * 17:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e9112d022e}} (l10n updates: es) === 2023-06-11 === * 11:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fb8c4a30ff}} (update punjabi-noun-masculine-guru) === 2023-06-09 === * 16:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e059c8bbd6}} (l10n updates: fi); also, last time I forgot to git rebase, so this actually includes {{Gerrit|2035050d28}} (l10n updates: sv) as well === 2023-06-07 === * 07:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2035050d28}} (l10n updates: sv) === 2023-06-04 === * 22:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|08962e4902}} (update past transgressive item ID after merge; only affects czech-verb-perfective) === 2023-05-31 === * 20:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1ec8c72304}} (Russian adjectivse: remove compound lexical categories) === 2023-05-29 === * 15:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a5e90a0e02}} (update dependencies) === 2023-05-27 === * 19:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|07deb7a083}} (Punjabi additive double causative verbs) * 17:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|889b4ce276}} (Punjabi additive causative verbs) * 15:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|467d5b9f34}} (Punjabi transitive verbs) * 15:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6c76e2d3b5}} (fix two Punjabi placeholders) === 2023-05-25 === * 20:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a50e668166}} (l10n updates: ca, es, fa, fi, ru, tr, ur) === 2023-05-19 === * 17:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b59c2f0aad}} (l10n updates: es, hi, zh-hant) * 16:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b80c8ff9db}} (fix “logged in” indicator in several languages) === 2023-05-18 === * 08:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7f76b0f203}} (l10n updates: br, de, fr, he, hi, hno, ia, mk, pa, pnb, ru, sa, sl, ur) === 2023-05-13 === * 17:29 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7d3ab49b06}} (l10n updates: ar, bn, de, eo, fa, fi, fr, he, hy, ia, it, ja, ko, mk, ms, nb, pnb, ru, skr-arab, sl, zh-hant) * 12:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|dfcf34ed51}} (make “logged in as” translatable) * 11:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|65cb94f3c7}} (punjabi-verb-basic-intransitive templates) === 2023-05-12 === * 20:41 wm-bot: <lucaswerkmeister> deployed {{Gerrit|15b7403971}} (fix stray character) === 2023-05-08 === * 21:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|db3dd67b8a}} (make more translations available and tweak Babel language codes) * 20:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a7bf757be9}} (fix message keys broken by previous deployment) * 20:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5f01c59794}} (refactor message keys from _ to -, should make no difference) * 19:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b51f930220}} (user interface language setting) === 2023-05-05 === * 12:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|72a006c6ea}} (l10n updates: mrh, ta) === 2023-05-02 === * 00:08 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5f83647d21}} (test-only change, pulled without webservice restart) === 2023-05-01 === * 23:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|88da33ddc5}} (GitHub actions only change, pulled without webservice restart) * 17:41 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1380884cce}} (upgrade dependescies, GHSA-m2qf-hxjv-5gpq) * 15:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|75230357a4}} (l10n updates: lt) * 15:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1554678038}} (improve matching.py for upcoming templates, should make no difference at the moment) * 14:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|692e255a50}} (refactor matching.py, should make no difference) === 2023-04-30 === * 15:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|db66a9373c}} (refactor statement groups; should make no difference) === 2023-04-25 === * 21:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9059e45cda}} (update dependencies, Werkzeug 2.3.0 / Flask 2.3.1) * 18:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c6dc908e1e}} (refactoring for somevalue support, should make no difference yet) === 2023-04-24 === * 19:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d5b5c8994f}} (preparation & refactoring, no visible changes) === 2023-04-23 === * 18:41 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0f96d60736}} (Punjabi adverbs) * 18:00 wm-bot: <lucaswerkmeister> deployed {{Gerrit|75af96b851}} (Punjabi adjectives) * 15:27 wm-bot: <lucaswerkmeister> deployed {{Gerrit|934f5cffdb}} (Yoruba adjectives) === 2023-04-22 === * 16:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a074fd9c64}} (trim spaces) * 15:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fdb0552957}} (remove spaces) * 15:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b6a1268b21}} (Punjabi nouns) === 2023-04-15 === * 15:41 wm-bot: <lucaswerkmeister> deployed {{Gerrit|604df5c72e}} (two more variables) * 15:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1b999f4661}} (use variables for entity IDs; should make no difference at runtime) * 14:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|24fb20fd19}} (sort sets for JSON output) === 2023-04-12 === * 20:27 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5b07592a7e}} (two style improvements) === 2023-04-10 === * 17:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|282a7b6b18}} (l10n updates: anp; currently skipped because unsupported by Babel) === 2023-04-08 === * 11:41 wm-bot: <lucaswerkmeister> deployed {{Gerrit|994cbd48b0}} (fix typo in a Hindustani template) === 2023-04-01 === * 18:41 wm-bot: <lucaswerkmeister> deployed {{Gerrit|08ac04d468}} (fix Hindko template order) === 2023-03-22 === * 20:54 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b40cefa378}} (change Hindko templates to hno) === 2023-03-19 === * 21:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cf1e031a43}} (l10n updates: fi, tt) === 2023-03-13 === * 21:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d7ba3ddc23}} (l10n updates: hi, pa, tt, ur) === 2023-03-08 === * 22:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8da3525baf}} (fix lowercase item ID in portuguese-noun-biform) * 22:08 wm-bot: <lucaswerkmeister> deployed {{Gerrit|de17c6bdf6}} (fix hindustani-verb-additive-causative-double-ur label) * 22:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a99078e1c5}} (hindustani-verb-additive-causative-double templates) === 2023-03-06 === * 21:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8828e3269e}} (l10n updates: tt) * 21:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2cbf107d6e}} (hindustani-verb-additive-causative templates) * 20:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0f7a634e72}} (fix Hindustani verb placeholders) === 2023-03-05 === * 21:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c325634bc3}} (hindustani-verb-additive-transitive templates) * 19:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|85cbe15d08}} (hindustani-verb-basic-transitive templates) * 13:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|00f87cf139}} (hindustani-verb-basic-intransitive templates) === 2023-03-03 === * 20:59 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c310fd9d88}} (update Hindustani labels, and l10n update: tt) === 2023-02-27 === * 19:54 wm-bot: <lucaswerkmeister> deployed {{Gerrit|50aa1e2dc5}} (l10n updates: hi, hno, pa, pnb, ur) === 2023-02-26 === * 21:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1d6c0caecd}} (Hindustani non-verb templates – verbs still TBD, need more time) * 15:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2feff85812}} (use hno translations) === 2023-02-22 === * 20:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9e667986b4}} (l10n updates: hi, ur) === 2023-02-14 === * 19:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9debac9385}} (update dependencies, especially Werkzeug 2.2.3 with two security fixes; venv rebuilt from scratch to avoid NFS issues) * 19:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bfd63ebac1}} (l10n updates: hno); also, turns out I didn’t git rebase in the last deployment, so this *actually* deploys the Danish nouns update and pl l10n update === 2023-02-09 === * 20:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2912ebfa68}} (update Danish nouns, and l10n updates: pl) === 2023-01-31 === * 19:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bfaf13f447}} (update github actions; pulled without webservice restart) * 19:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f9bf85df5f}} (l10n updates: cy) === 2023-01-29 === * 12:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3ca9650fe1}} (Danish adjectives) === 2023-01-09 === * 19:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4857d874ce}} (l10n updates: pa) === 2023-01-03 === * 15:29 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5c27eaec33}} (l10n updates: pl) === 2022-12-30 === * 12:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|95b9026d22}} (l10n updates: pa, zh) === 2022-12-28 === * 15:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3c47032838}} (fix bulk result display when given lexeme ID) === 2022-12-26 === * 11:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b51ddc8c08}} (update Armenian noun templates) * 11:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bdaa43aef3}} (preserve target_hash in more places) === 2022-12-16 === * 21:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4802902384}} (l10n updates: yue) === 2022-12-08 === * 19:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3f6b15c1f0}} (l10n updates: fa, gl, pl, sl) === 2022-12-06 === * 13:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|97001e468b}} (fix missing statements) * 13:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|45a026916c}} (fix Hindko feminine noun template) === 2022-12-05 === * 21:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4d781fb933}} (Hindko noun templates) * 20:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2cb7ac792f}} (l10n updates: pnb) === 2022-12-04 === * 17:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|82a2272a2f}} (three new Norwegian Nynorsk noun templates) === 2022-11-29 === * 21:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b0ebae4629}} (l10n updates: el) === 2022-11-27 === * 19:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bb7cf271ae}} (l10n updates: fa) === 2022-11-19 === * 15:03 wm-bot: <lucaswerkmeister> deployed {{Gerrit|10af55574b}} (more Bokmål and Nynorsk templates) === 2022-11-15 === * 20:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5897fd06ee}} (Danish nouns fix) * 20:23 wm-bot: <lucaswerkmeister> ionice -c3 zstd --rm uwsgi.log.1668543276 # 8.85%, {{Gerrit|520591680}} => {{Gerrit|46091850}} bytes) * 20:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2b53b1199c}} (rotate uwsgi.log after 100 MiB) * 19:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0429d7d80b}} (update Danish nouns+verbs) === 2022-11-10 === * 13:41 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5160edb9ca}} (l10n updates: pnb) * 13:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|127e065522}} (NFC-normalize lemma for search) === 2022-11-07 === * 21:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0c7095c96d}} (Polish adjectives, positive only) * 20:54 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c653fb07e2}} (l10n updates: es, hy, pnb) === 2022-11-05 === * 14:02 wm-bot: <lucaswerkmeister> git gc (.git 19M → 1.1M) * 13:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8feb3f86d4}} (extra GitHub actions job, pulled without webservice restart) * 12:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d38d5ba55c}} (uninstall dev dependencies in production; reduces venv size from ca. 142 MB to ca. 75 MB, or about by half) * 12:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b7f4d4ba31}} (added test; pulled without webservice restart) * 11:03 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ccecd3bb87}} (l10n updates: krc, zh) === 2022-10-27 === * 12:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|03b6dd3b71}} (l10n updates: pnb) === 2022-10-26 === * 20:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2feba604c7}} (update dependencies, use PEP 655 NotRequired) * 19:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|55f9b203e5}} (l10n updates: sl) === 2022-10-23 === * 16:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3844a7df05}} (French verbs) === 2022-10-17 === * 19:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b098904d43}} (l10n updates: ja, pnb) === 2022-10-14 === * 18:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a829f83124}} (l10n updates: ca, hi, sh, sl) === 2022-10-05 === * 20:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|953b553968}} (translate Hebrew adjective template label) * 18:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|93ebb772c5}} (more Spanish templates) === 2022-10-01 === * 19:00 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b770688eb1}} (Hebrew adjectives) * 18:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8137259ca6}} (Flask 2.2) === 2022-09-23 === * 19:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c66922341d}} (l10n updates: ar, ku, sl) === 2022-09-18 === * 16:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8d996c1fa4}} (l10n updates: ar) === 2022-09-10 === * 18:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|609066f02b}} (README fix, pulled without webservice restart) * 16:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|52570991cd}} (diffusion → gitlab) === 2022-08-29 === * 20:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fa8f5d87a4}} (l10n updates) === 2022-08-25 === * 14:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|019b4ecc79}} (optimize messages with unused GENDER magic word) * 14:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|dd6cb7f08b}} (l10n updates) === 2022-08-03 === * 19:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a11b6a55f6}} (l10n updates) === 2022-07-21 === * 23:03 wm-bot: <lucaswerkmeister> deployed {{Gerrit|38141487d1}} (l10n updates) === 2022-07-17 === * 17:59 wm-bot: <lucaswerkmeister> deployed {{Gerrit|238f943e8a}} (add more typing; hopefully no functional changes) === 2022-07-13 === * 20:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d5cb20368d}} (l10n updates) === 2022-07-02 === * 19:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d3e2185bbc}} (l10n updates) === 2022-06-29 === * 19:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6ac757a997}} (Igbo verbs + pronouns) === 2022-06-16 === * 21:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|466976ba49}} (l10n updates) === 2022-06-14 === * 22:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b0143851e0}} (l10n updates) === 2022-05-26 === * 20:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|24d9b273c5}} (l10n updates) === 2022-05-17 === * 19:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8cdef0cf20}} (l10n updates) === 2022-05-03 === * 20:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d8429a8740}} (l10n updates) === 2022-04-29 === * 19:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fd45333563}} (l10n updates, extra unit test) === 2022-04-28 === * 23:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|860abb205b}} (Bokmål passive verbs) === 2022-04-27 === * 20:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c92b363387}} (Mandarin templates) === 2022-04-25 === * 19:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7b5d0d7298}} (l10n updates) === 2022-04-22 === * 11:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d769b4ed8b}} (l10n updates) === 2022-04-20 === * 19:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|89a5273967}} (l10n updates) === 2022-04-15 === * 18:16 wm-bot: <lucaswerkmeister> pulled {{Gerrit|24d5774c5f}} (test-only change, so no restart) * 18:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|54a5376631}} (update German verbs) * 16:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9a2cefe8e6}} (updated Portuguese templates) === 2022-04-04 === * 19:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|197baf2940}} (l10n updates) === 2022-03-30 === * 18:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c6001bf897}} (l10n updates; use pip-tools, includes some package updates such as Flask 2.0.2→2.1.0; clean up service.template) === 2022-03-19 === * 12:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f573b558d4}} (l10n updates) === 2022-03-11 === * 00:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d7787d7536}} (l10n updates) === 2022-03-05 === * 18:08 wm-bot: <lucaswerkmeister> deployed {{Gerrit|72f2adc394}} (l10n updates) === 2022-02-28 === * 12:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|04ba7580ab}} (l10n updates) === 2022-02-25 === * 00:49 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1506d1a9e9}} (l10n updates) === 2022-02-22 === * 00:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1fc2f98450}} (l10n updates) === 2022-02-15 === * 13:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|56e69bad1a}} (l10n updates) === 2022-02-11 === * 23:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b4624e0bbc}} (l10n updates) === 2022-02-07 === * 13:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b3c5446831}} (l10n updates) === 2022-01-30 === * 12:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c1d6a79ed2}} (update Odia nongendered adjectives) === 2022-01-22 === * 17:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b1cc42ef84}} (Odia nouns) * 16:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b62723fb6f}} (update Odia adverbs) === 2022-01-16 === * 19:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|504c5481e9}} (update Spanish verbs) * 18:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|68234bd17d}} (Odia adjectives and adverbs) === 2022-01-10 === * 18:59 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d1da801731}} (l10n updates) === 2022-01-06 === * 18:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|57dc392b8f}} (l10n updates) === 2022-01-03 === * 18:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|aacaae3cd6}} (revert update of indefinite item ID after merge, I flipped the items) * 15:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2eb6822ed2}} (l10n updates) * 15:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7312514fc8}} (update indefinite item ID after merge) === 2022-01-01 === * 23:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d6110ed631}} (l10n updates) === 2021-12-17 === * 21:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|20c4392de6}} (l10n updates) === 2021-12-02 === * 23:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2a2cb9b211}} (l10n updates) === 2021-11-25 === * 21:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|baef3a16f6}} (l10n updates) === 2021-11-18 === * 13:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e001c252c5}} (l10n updates, including initial Yoruba translations) === 2021-11-14 === * 14:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c113d4dd77}} (Yoruba nouns) === 2021-11-08 === * 22:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|85719cf3ae}} (update Portuguese idioms) * 22:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e58c43ab3e}} (Portuguese idioms quickfix) === 2021-11-07 === * 19:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|91216ed64b}} (Portuguese idioms) === 2021-11-06 === * 12:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7ef5eb34a3}} (fix Manbhumi bulk mode link) === 2021-11-04 === * 12:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d649d7a24a}} (l10n updates) === 2021-10-25 === * 19:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0f5b5de66a}} (bump startupProbe failureThreshold 3→10) * 19:34 wm-bot: <lucaswerkmeister> deployment was successful after all 🤷 * 19:31 wm-bot: <lucaswerkmeister> belay that, the new pod hasn’t actually started properly. investigating * 19:29 wm-bot: <lucaswerkmeister> deployed {{Gerrit|754342b9a3}} (language name for bn-x-Q6747180) === 2021-10-18 === * 12:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|eae6c8d594}} (l10n updates) === 2021-10-16 === * 14:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1903c3d0eb}} (don’t show duplicate warning errors) * 12:09 wm-bot: <lucaswerkmeister> pulled {{Gerrit|8700382f98}} (rename confusingly named deplyoment patch file) without webservice restart * 12:04 wm-bot: <lucaswerkmeister> (correction on that last message, it’s a startup probe now, not a readiness probe) * 12:03 wm-bot: <lucaswerkmeister> patched readiness probe into deployment again * 12:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|19fb8c90ee}} (findDuplicates fix) with full stop/start to pick up label changes === 2021-10-13 === * 23:31 wm-bot: <lucaswerkmeister> fully restarted webservice (stop/start) to avoid label issues * 17:49 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e5c87ff53c}} (remove type ignore comments) and updated dependencies, including Flask 2.0.2 === 2021-10-11 === * 12:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fb32d04132}} (l10n updates) === 2021-10-10 === * 11:20 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bf2834c472}} (improve error handling) === 2021-10-04 === * 19:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1697521bf5}} (l10n updates) === 2021-09-25 === * 14:45 wm-bot: <lucaswerkmeister> removed old venv-3.7 * 13:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6f9e530018}} (mobile-friendly navbar) * 13:41 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ea93caf2ee}} (l10n updates) === 2021-09-19 === * 13:49 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3c1b6e0810}} (readinessProbe → startupProbe to avoid bloating access log); deployed by adding readinessProbe: null to the patch file and patching the deployment with that === 2021-09-14 === * 20:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c36ae4154a}} (l10n updates) * 19:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|902156ddb8}} (Croatian item ID fix) === 2021-09-12 === * 21:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4da7f64c4b}} (updates without downtime) * 20:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f21554ab71}} (refactoring, noop) * 15:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a4b05045d6}} (Croatian nouns) === 2021-09-08 === * 20:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2aa32a0f7f}} (l10n updates) === 2021-09-03 === * 15:54 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3698f0b79c}} (add passive forms to Norwegian Bokmal verbs) * 15:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8051248b60}} (l10n updates) === 2021-08-30 === * 18:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|dfc0838301}} (l10n updates) === 2021-08-25 === * 20:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|237a5414d5}} (l10n updates) === 2021-08-19 === * 20:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bcc4c3aa63}} (l10n updates) === 2021-08-17 === * 21:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0ca42b7cdb}} (more types) * 18:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2382c30c01}} (initial mypy setup) * 17:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c66572938e}} (python3.9) === 2021-08-16 === * 12:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|92e5e0d70c}} (l10n updates) === 2021-08-14 === * 12:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7a1980f4e2}} (l10n updates) === 2021-08-11 === * 19:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|37acc67c90}} (l10n updates) === 2021-08-02 === * 19:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|de5ab0e740}} (l10n updates) === 2021-07-19 === * 18:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0c9f1015c0}} (work around Firefox bug) === 2021-07-18 === * 18:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fa64f7e021}} (refuse to load non-user-readable config file, guard against recurrence of [[phab:T286414|T286414]]) * 13:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|61b1d0fd93}} (Igbo adjectives and fix nouns) === 2021-07-17 === * 11:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0d1f3d924e}} (load config file differently) === 2021-07-16 === * 19:23 wm-bot: <lucaswerkmeister> deployed {{Gerrit|37766a8002}} (l10n updates) === 2021-07-11 === * 20:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5dbc39eb5e}} (l10n update) * 17:03 wm-bot: <lucaswerkmeister> restarted webservice to pick up 1.3 version of OAuth consumer ([[phab:T286414|T286414]]) * 13:36 wm-bot: <lucaswerkmeister> chmod go-rwx www/python/src/config.yaml # [[phab:T286414|T286414]] === 2021-07-01 === * 23:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ac8779515d}} (l10n updates) * 23:37 wm-bot: <lucaswerkmeister> unlink ~/services.template # new version of webservice doesn’t like the symlink :( === 2021-06-28 === * 17:54 wm-bot: <lucaswerkmeister> deployed {{Gerrit|64c5584c9d}} (remove workaround for [[phab:T241422|T241422]]) * 17:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5565da07e5}} (l10n updates, especially Igbo translations) === 2021-06-22 === * 19:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c88b1962fa}} (Igbo nouns) === 2021-06-21 === * 20:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|19098277f4}} (l10n updates) === 2021-06-20 === * 12:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|afc6f6f242}} (update German verbs) === 2021-06-19 === * 19:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c5b12d5dc1}} (Malayalam proper nouns) * 19:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|05cd31e9bd}} (update Malayalam noun) === 2021-06-15 === * 20:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0b6fed0054}} (even more optional grammatical features) * 19:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d8eadd1cae}} (more optional grammatical features) * 18:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|61a5e0fc18}} (optional grammatical features) === 2021-06-14 === * 23:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|626b73a005}} (l10n updates) * 23:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|70efbdc1a7}} (update volitive item ID) === 2021-06-10 === * 20:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1f94df1209}} (l10n updates) === 2021-06-07 === * 21:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|547231388b}} (add create link for duplicates in bulk mode) * 20:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|daf88503e0}} (l10n updates) === 2021-06-06 === * 14:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2040a7497e}} (target_hash URL parameter) === 2021-06-05 === * 20:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fcf67b1016}} (improve title) === 2021-06-04 === * 23:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|16c0cd2606}} (improve batch mode results page) === 2021-05-31 === * 20:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|43a29c4369}} (replace deprecated function) * 20:00 wm-bot: <lucaswerkmeister> pip upgrade (Flask 2.0.1 and other updates) * 19:59 wm-bot: <lucaswerkmeister> briefly stopping tool to upgrade venv * 18:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|148dafa60b}} (l10n updates) === 2021-05-30 === * 14:59 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3c047f6aca}} (l10n updates) === 2021-05-24 === * 18:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6ffd1a2c1b}} (update Esperanto verb) * 16:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7d43094e56}} (l10n updates) * 11:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e0099e68d5}} (Swedish adjective) === 2021-05-22 === * 09:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|31e85bafcf}} (l10n updates) * 09:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|44812d4446}} (add Portuguese modal adverb) === 2021-05-15 === * 14:01 wm-bot: <lucaswerkmeister> tool should be back up (uwsgi.log went from 181M to 77M after moving pre-2021 data to separate files) * 13:56 wm-bot: <lucaswerkmeister> briefly stopping tool (few minutes) to cycle the uwsgi.log === 2021-05-13 === * 23:03 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3e2ceb0513}} (l10n updates) * 14:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|67e7cf3dfb}} (rename Swedish adjective template) * 13:49 wm-bot: <lucaswerkmeister> deployed {{Gerrit|95f40ac9d5}} (Norwegian Bokmål masculine/neuter nouns) === 2021-05-10 === * 16:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|248527544d}} (l10n updates) === 2021-05-09 === * 13:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5951b46450}} (fix lang= and dir= on index) === 2021-05-03 === * 19:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b159dd1060}} (l10n updates) === 2021-05-02 === * 11:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4c9a5f0ebf}} (duplicate check JS fixes) === 2021-05-01 === * 14:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|61744950f0}} (l10n updates) === 2021-04-26 === * 19:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|abf6719d31}} (Python 3.7 fix) * 19:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d15d0c5f2d}} (rename Dutch templates) * 18:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|868ee95cf2}} (l10n updates) === 2021-04-22 === * 19:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8ab4ceb62a}} (l10n updates) === 2021-04-19 === * 20:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2f8f589a62}} (Swedish proper nouns) * 20:23 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4effbc2a36}} (l10n updates) === 2021-04-17 === * 10:20 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1d10ab467e}} (fix bulk mode) === 2021-04-15 === * 19:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|051e3789a2}} (l10n updates) === 2021-04-14 === * 20:59 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b17ed175fe}} (move login hint up) * 20:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0006696173}} (remove automatic login redirect) * 12:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|30c561955f}} (login link in navbar) === 2021-04-12 === * 18:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e4682a00bd}} (Breton noun fixes) * 18:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a3a81d0c4b}} (l10n updates) === 2021-04-09 === * 18:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|18bb25abd0}} (l10n updates) === 2021-04-05 === * 13:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f5439f66a2}} (l10n updates) === 2021-04-04 === * 13:49 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9507991400}} (Malayalam verb fix) === 2021-04-03 === * 19:08 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3e2bc5b577}} (language code refactorings; should not result in any observable changes) * 18:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8416f8d861}} (more Breton nouns + adverbs) * 16:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|21201880f5}} (MarkupSafe-aware formatters; should not result in any observable changes) * 15:08 wm-bot: <lucaswerkmeister> deployed {{Gerrit|615bba5934}} (better bulk mode errors) === 2021-04-02 === * 19:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|be73b49e29}} (better language code handling) === 2021-04-01 === * 18:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f2b128273d}} (l10n updates) === 2021-03-30 === * 21:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7ff57d504e}} (l10n updates) === 2021-03-28 === * 19:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|43d0c29996}} (update Portuguese nouns) * 14:16 wm-bot: <lucaswerkmeister> <em>actually</em> deployed {{Gerrit|2ece3adc91}} (this time I did the <code>git rebase</code> but forgot the <code>webservice restart</code>, how’s that for a change) * 13:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2ece3adc91}} (Portuguese updates) === 2021-03-27 === * 14:00 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1f2a6f2e17}} (replace OrderedDict with dict) * 13:41 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4619f8cd03}} (remove duplicate template) * 13:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9ad3addd6a}} (Malayalam verbs, and vocative case for nouns) === 2021-03-26 === * 21:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5b44b44f52}} (Malayalam verbs) * 21:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|78a5c9a10a}} (indicate optional forms) === 2021-03-25 === * 19:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|77328e559d}} (optional forms) === 2021-03-24 === * 22:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ffa45a58b1}} (minifix) * 19:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ea6928faaa}} (clarify Norwegian Bokmål adjectives) * 19:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|99257d861c}} (Portuguese adjectives) === 2021-03-23 === * 21:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|253aed283c}} (Latvian nouns) * 19:27 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c0b2c473ff}} (add language code as ID on index page, suggested by jhsoby) === 2021-03-22 === * 21:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2e4e3dca5a}} (improved Malayalam nouns [not verbs as it says in the commit message, oops] + i18n updates) === 2021-03-16 === * 19:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|547b42f25f}} (Portuguese nouns, i18n updates) === 2021-03-13 === * 16:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f389caf9b2}} (gender i18n improvements, should be a no-op) === 2021-03-12 === * 20:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9500beeed4}} (three new translations) – should be a no-op but I didn’t want to leave it lying around without a webservice restart either * 19:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|aa07bef3bd}} (i18n update) – also, previous SAL message mentioned {{Gerrit|712d262475}} but that’s still in <code>git log @..@<nowiki>{</nowiki>u<nowiki>}</nowiki></code>, so I think I forgot to rebase last time === 2021-03-10 === * 20:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|712d262475}} (restore logging for generic API errors) * 19:59 wm-bot: <lucaswerkmeister> deployed {{Gerrit|94dfecbc2a}} (generic API error handler) === 2021-03-08 === * 14:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b7b55e1b33}} (more i18n improvements) * 11:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ea7cd3ac71}} (i18n from translatewiki.net – [[phab:T272243|T272243]]) === 2021-03-05 === * 22:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|109f22a415}} (Czech verbs update) === 2021-03-04 === * 21:08 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1435d31446}} (update Swedish translations) * 20:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|15a24d63eb}} (minor Czech verbs improvement) === 2021-02-28 === * 17:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|369031b945}} (minifix) * 17:10 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0455dc20f4}} (better OAuth error handling) === 2021-02-19 === * 18:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f66f631598}} (auth improvements) === 2021-02-18 === * 20:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a0ba7b84ab}} (quickfix) * 20:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|23ccbcf6f6}} (work around [[phab:T272319|T272319]]) === 2021-02-16 === * 20:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8d96af0ec2}} (add skip link) * 19:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3e716e6d6d}} (Bootstrap update) === 2021-02-13 === * 22:27 wm-bot: <lucaswerkmeister> deployed {{Gerrit|02a2edf583}} (edit summary fixes) * 18:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a7257a065e}} (code style fixes) * 16:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4e70e759d7}} (minifix) * 13:08 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fb17f5e4ef}} (edit mode fix for forms with multiple representations) === 2021-02-11 === * 22:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|81166d5c17}} (reduce [[phab:T230833|T230833]] workaround / "und" language codes) * 22:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8e718af67e}} (JS fix) === 2021-02-10 === * 20:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0d8279ca7f}} (<script> loading improvements) * 20:00 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1fe3d3589e}} (prevent double submit) === 2021-02-04 === * 20:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|32b6b23f72}} (German adverbs) === 2021-02-01 === * 21:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f4e7ba98a7}} (stop referrer-URL comparison) * 14:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d237952e44}} (fix current_url / CSRF detection) === 2021-01-30 === * 20:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a87ce138db}} (show bulk parse errors) === 2021-01-28 === * 20:03 wm-bot: <lucaswerkmeister> deployed {{Gerrit|868bccbbe7}} (fall back to en) * 19:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cb0855af48}} (simplify current_url) === 2021-01-27 === * 22:39 wm-bot: <lucaswerkmeister> deployed fixed version of test code, oops * 22:38 wm-bot: <lucaswerkmeister> deployed another version of test code * 22:26 wm-bot: <lucaswerkmeister> deployed uncommitted test code to print current_url debug output * 20:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1bc8d4232e}} (remove long-dead code about fixing the session cookie) * 20:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|03255e1408}} (pop OAuth redirect target) === 2021-01-13 === * 20:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e5725705d1}} (fix edit mode, drop form data stashing) === 2021-01-09 === * 21:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9a604413d3}} (German toponym) === 2021-01-07 === * 14:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|00d7fe313e}} (better edit links) === 2021-01-03 === * 11:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|db1e890252}} (grab cursor for draggable links) === 2020-12-30 === * 12:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|191518cbf9}} (edit lemma when adding first form) === 2020-12-23 === * 15:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6d8bae537b}} (Esperanto verb) * 14:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|69f610af18}} (Breton noun, without mutation, collective) === 2020-12-22 === * 11:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6e1185532d}} (Basque adjective) === 2020-12-14 === * 20:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9ba55b3ad3}} (fix current_url) === 2020-12-13 === * 00:49 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bb0cbfc6cb}} (language code in parentheses) === 2020-12-12 === * 18:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0ec650ea2f}} (autonyms on index page) === 2020-12-02 === * 21:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e5291d5cda}} (more Esperanto translations) === 2020-11-29 === * 21:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|915eb4016f}} (clarify German templates) === 2020-11-24 === * 21:58 wm-bot: <lucaswerkmeister> undeployed debug code, I don’t remember what it was for anymore * 21:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|59f2c38fed}} (the previously-uncommitted JS fix, now committed; some uncommitted debug code is still there) === 2020-11-21 === * 21:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1608cc4dd9}} (gender-dependent messages) === 2020-11-05 === * 19:51 wm-bot: <lucaswerkmeister> deployed uncommitted JS fix, to be committed later if it works as intended === 2020-10-29 === * 22:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1a150904fd}} (update Italian translations) === 2020-10-26 === * 21:27 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e3c4c2e664}} (Esperanto adjective) === 2020-10-25 === * 21:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bd4c445f02}} (edit mode fix) * 21:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|782dfdabee}} (fixes for edit mode and ordia links) === 2020-10-24 === * 13:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|792db2a9f9}} (edit mode language_code parameter) === 2020-10-19 === * 20:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a7fd004ef9}} (drag’n’drop fix; submit_lexeme debug code still there) === 2020-10-17 === * 14:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|19b5bc257a}} (more durable CSRF tokens; some uncommitted debug code to print submit_lexeme errors is still there) === 2020-10-08 === * 20:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fd8c692798}} (fix a crash; debug code still in place) === 2020-09-13 === * 08:36 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9f02b375f1}} (more conventient bulk mode transition; debug code still present) * 08:17 wm-bot: <lucaswerkmeister> deployed uncommitted extra logging for submit_lexeme errors in bulk mode === 2020-09-12 === * 12:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ce943856ed}} (fix Spanish feminine noun item ID) === 2020-09-08 === * 16:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9ac796e7aa}} (Manbhumi verbs) === 2020-09-06 === * 08:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|116e4123b0}} (fix Manbhumi duplicate search) === 2020-09-01 === * 15:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ef72c06ec8}} (Manbhumi adjectives and adverbs) === 2020-08-14 === * 19:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|13282d5404}} (Bengali verb updates) === 2020-08-12 === * 19:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e3291c8796}} (Bengali adverbs, other improvements) === 2020-08-04 === * 22:43 wm-bot: <lucaswerkmeister> <em>actually</em> deployed {{Gerrit|39457a18ab}} (forgot to git rebase) * 22:36 wm-bot: <lucaswerkmeister> deployed {{Gerrit|39457a18ab}} (Bengali adjectives and verbs) === 2020-07-08 === * 21:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b65c1018ff}} (translation update) === 2020-07-05 === * 22:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f29663c2b2}} (Norwegian Bokmål nouns) === 2020-07-04 === * 16:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cbf5ad6440}} (Norwegian Bokmål) === 2020-06-17 === * 23:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9b7349c602}} (update a Bengali template) === 2020-06-15 === * 20:54 wm-bot: <lucaswerkmeister> renamed default branch from master to main === 2020-06-14 === * 12:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8d5f428c3e}} (improved duplicate warning edit links) * 10:15 wm-bot: <lucaswerkmeister> *actually* deployed {{Gerrit|2efe64f7e5}} (forgot to git rebase) * 10:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2efe64f7e5}} (link edit mode in duplicate warning) === 2020-06-13 === * 21:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b42e79e6bb}} (more sections) * 17:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cf1079fda1}} (more section improvements) * 13:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c2e6d57a29}} (improved German sections) * 11:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4cd36a71a1}} (sections in edit mode) * 11:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4e288f0106}} (sections) * 08:54 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bfa46d522b}} (Czech edit mode translations) === 2020-06-07 === * 20:53 wm-bot84: <lucaswerkmeister> deployed {{Gerrit|9e4f3a1b65}} (two translation fixes) * 13:35 wm-bot84: <lucaswerkmeister> deployed {{Gerrit|09cc2017ec}} (Bengali nouns) === 2020-05-24 === * 13:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5c6d1c6e30}} (update Breton) === 2020-05-13 === * 22:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a2deb7908c}} (update past participle item ID after merge) === 2020-05-11 === * 19:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ddac27d2e2}} (translation update) === 2020-05-10 === * 22:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b797c90917}} (Breton typofix) * 15:00 wm-bot: <lucaswerkmeister> deployed {{Gerrit|eac96e8493}} (Breton adjectives and other improvements) * 11:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fc78831f8e}} (Breton nouns) === 2020-05-09 === * 19:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b4780fa832}} (drag’n’drop unmatched forms in edit mode) === 2020-04-25 === * 20:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0dadbb4d4e}} (toolforge.org) === 2020-04-21 === * 21:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6634452b4c}} (increase uWSGI buffer) === 2020-04-18 === * 18:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c815a210bd}} (Hebrew nouns) * 17:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|33c3ac264e}} (fix english-adverb edit mode) * 11:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2959ebf637}} (fix duplicates in advanced mode) === 2020-04-14 === * 20:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|44b5df2897}} (edit mode: show lemma, show conflicts, add missing statements) === 2020-04-13 === * 22:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2fe2118d4e}} (python3.7) * 22:20 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ab7f751ba6}} (edit mode) === 2020-02-26 === * 00:22 wm-bot: <root> Migrated to 2020 Kubernetes cluster === 2020-01-28 === * 00:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|61fe7e59fb}} (typofix) * 00:08 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e0e916e0a5}} (more Persian translations and RTL fixes) === 2020-01-27 === * 23:23 wm-bot: <lucaswerkmeister> deployed {{Gerrit|54b9e37118}} (more RTL fixes) * 23:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|72ec256823}} (Persian nouns and verbs) [actually happened ~30mins ago, forgot to log] === 2020-01-15 === * 00:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bc1d49c202}} (better CSRF error handling, [[phab:T242573|T242573]]) === 2020-01-14 === * 00:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|242c25810b}} (clarify Spanish verbs) === 2020-01-12 === * 14:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|edcbc10ae9}} (Spanish verbs) === 2020-01-11 === * 17:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d9619cb473}} (Danish nouns and verbs) * 14:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4a20b4b95e}} (Czech perfective verbs) * 14:10 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8da9227b52}} (fix typos in Czech adjective template) === 2019-11-30 === * 13:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2f5a8ccc2e}} (update english-verb) === 2019-11-21 === * 22:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|13cf2696b9}} (reorder) * 22:27 wm-bot: <lucaswerkmeister> deployed {{Gerrit|89ad1e816c}} (Basque verbs) === 2019-11-11 === * 23:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cd4239904a}} (work around [[phab:T230833|T230833]]) * 21:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8b53b417c1}} (fixes to Kurdish (Kurmancî)) * 17:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fe31bd9aa6}} (message syntax fix) === 2019-11-10 === * 19:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9d736fe2f6}} (Kurdish Kurmancî nouns) * 15:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|29e549fe31}} (Malayalam nouns) === 2019-10-27 === * 22:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2fc68fabb5}} (lexeme IDs in bulk mode) === 2019-10-16 === * 22:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b480b6d07e}} (Czech translations + adjectives with more forms) === 2019-10-07 === * 22:49 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ce8ba2b234}} (add plural grammatical feature to Ukrainian plurale tantum forms) === 2019-09-30 === * 22:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|19bf4e3347}} (remove PHP_ENGINE cookie) === 2019-08-28 === * 23:10 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a053e9a36e}} (update Swedish translations) === 2019-08-22 === * 22:53 wm-bot: <lucaswerkmeister> deployed 60cf696645v (minor bulk mode improvements) * 22:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f4fd72ab72}} (bulk mode improvements) === 2019-08-20 === * 20:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|938075faf2}} (bulk mode) === 2019-08-11 === * 11:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|09a3ac6b64}} (Swedish absolute adjectives) === 2019-08-02 === * 21:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a4d699fbcb}} (fix item ID after merge) === 2019-07-24 === * 12:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f0883f1ebc}} (templates API) === 2019-07-07 === * 18:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|50a70b3590}} (Swedish verbs) * 13:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9a148c8cc5}} (add statements when editing existing lexeme) * 12:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a8242673b9}} (use jsonify) * 12:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|994b980655}} (CORS for duplicates API) === 2019-07-06 === * 22:03 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b0f39bb09b}} (API to match lexemes to templates) === 2019-06-26 === * 20:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e74ff290cc}} (duplicates API bug fix) [actually deployed 2 hours ago, forgot to log] === 2019-06-24 === * 22:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e937ff5839}} (autocapitalize="off" on form) * 22:44 wm-bot: <lucaswerkmeister> deployed uncommitted experimental change (autocapitalize="off" on form and inputs) * 22:29 wm-bot: <lucaswerkmeister> deployed uncommitted experimental change (autocapitalize="off" on form rather than inputs) * 22:14 wm-bot: <lucaswerkmeister> deployed uncommitted experimental change (autocapitalize="off" on inputs) * 21:10 wm-bot: <lucaswerkmeister> deployed {{Gerrit|07b05a6858}} (Portuguese verbs) === 2019-06-14 === * 19:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c48127f696}} (update Russian translations) * 00:38 wm-bot: <lucaswerkmeister> kubectl delete deployment lexeme-forms.purge-all-lexemes # [[phab:T225510|T225510]] done === 2019-06-12 === * 08:48 wm-bot: <lucaswerkmeister> kubectl create -f deployment-purge-all-lexemes.yaml # [[phab:T225510|T225510]] === 2019-06-10 === * 19:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|645886b3a8}} (update German translations) * 18:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|846100f8d9}} (update Czech translations) * 12:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fe6cc3a79b}} (improved forms/senses message for duplicates) === 2019-06-09 === * 23:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5c88de6348}} (number of forms/senses for duplicates) === 2019-06-08 === * 14:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f09dfd20a1}} (Dutch nouns) * 14:00 wm-bot: <lucaswerkmeister> git remote add github https://github.com/lucaswerkmeister/tool-lexeme-forms.git # work around [[phab:T224677|T224677]] * 12:17 wm-bot: <lucaswerkmeister> restarted webservice after redirect loop === 2019-05-20 === * 09:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|496a928b67}} (switch to Python 3.5), including venv rebuild * 08:52 wm-bot: <lucaswerkmeister> stopping webserver for Python 3.5 upgrade <noinclude>[[Category:SAL]]</noinclude> n5i1nfwj245g2k73fz6g6aens60qeso 2426627 2426626 2026-06-13T16:16:19Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed d153685969 (add login check) 2426627 wikitext text/x-wiki === 2026-06-13 === * 16:16 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|d153685969}} (add login check) * 16:13 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|5fde4e8ec2}} (use authenticated session for Wikifunctions calls: [[phab:T349966|T349966]], [[phab:T423542|T423542]]) === 2026-06-11 === * 19:59 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|3358ae7e8c}} (l10n updates: ar, fi, frp, it, ko, nl, sk, zh-hans; using the previously deployed {{GENDER:}} support in duplicates-instructions) === 2026-06-06 === * 19:36 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|22a11f62c4}} (add {{GENDER:}} support to duplicates-instructions message) === 2026-06-04 === * 19:56 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|3c09a6963f}} (l10n updates: nl) === 2026-06-03 === * 17:56 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fd3d67655e}} (bump mwoauth2) === 2026-05-29 === * 17:23 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|11f47f0f24}} (install mwoauth2 as package) === 2026-05-27 === * 21:34 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|994cfa51b3}} (make mwoauth2 strictly typed) * 18:34 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|73fcbaa464}} (extract mwoauth2 module) === 2026-05-25 === * 13:51 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|c163567ccd}} (l10n updates: nb) === 2026-05-21 === * 21:45 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|97d0866c92}} (treat outdated [OAuth 1] access tokens more robustly) * 19:18 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|f0d7740038}} (l10n updates: ko, ta, vi) === 2026-05-20 === * 18:55 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|dfc101f3e0}} (Python 3.14, aka 𝜋thon) * 18:50 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|5b92917005}} (upgrade dependencies) * 18:46 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|a42e65b596}} (configure Gunicorn --forwarded-allow-ips) === 2026-05-19 === * 17:59 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|43b18b8299}} (prevent OAuthLib InsecureTransportError more strongly) * 17:52 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|80583945a8}} (migrate to OAuth 2) === 2026-04-25 === * 23:34 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|4f201ca05a}} (update gunicorn config) * 19:30 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|ab0ee1c0ce}} (Russian imperfective verbs) === 2026-04-13 === * 17:46 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fe623b17c0}} (l10n updates: hi, ms-arab) === 2026-04-06 === * 12:36 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|68d0059881}} (l10n updates: ms-arab) === 2026-03-26 === * 17:21 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|a2dde3c334}} (l10n updates: pa) === 2026-03-23 === * 19:02 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|03da8bff9b}} (l10n updates: kea) === 2026-03-19 === * 13:17 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|cdc85da9aa}} (l10n updates: kea) === 2026-03-16 === * 13:26 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|83b82c6ea4}} (l10n updates: ary, ga) === 2026-03-14 === * 14:55 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|2e36547a31}} (Moroccan Arabic templates – part of Wikimedia Hackathon Northwestern Europe 2026 \o/) === 2026-03-12 === * 16:31 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|0282ad0864}} (l10n updates: kea) === 2026-03-09 === * 13:22 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|f0983e0fcf}} (l10n updates: kea) === 2026-02-10 === * 21:15 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|ca57a0da6a}} (noop – update a test) === 2026-02-09 === * 18:40 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|49f7ed4319}} (l10n updates: el) === 2026-02-02 === * 13:44 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|74bb77b1b4}} (l10n updates: el, pl) === 2026-01-22 === * 12:56 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|4bdcac2b61}} (l10n updates: pa) === 2026-01-18 === * 17:45 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|3ba43046ee}} (avoid changing lemma if not necessary) === 2026-01-05 === * 12:55 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fa52e74def}} (l10n updates: id) === 2026-01-03 === * 17:03 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|2825501536}} (l10n updates: ca, it, pl, sv) === 2025-12-22 === * 19:08 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|40105a8e8b}} (l10n updates: it, vi) === 2025-12-11 === * 21:15 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|5c7dd452be}} (l10n updates: mk) === 2025-12-01 === * 13:01 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|54c6749c45}} (l10n updates: fi) === 2025-11-19 === * 18:55 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|7f658a6675}} (update Bootstrap) === 2025-11-18 === * 20:04 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|c74254856c}} (fix skiplink visibility) === 2025-11-17 === * 13:06 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|34ee7bf7ac}} (l10n updates: cy) === 2025-11-13 === * 12:46 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|808bd196dc}} (l10n updates: anp) === 2025-11-10 === * 19:04 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|730ae77335}} (drop typing_extensions) * 18:58 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|36b8ab588c}} (upgrade dependencies) * 18:47 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|422cb1f05c}} (l10n updates: frp, ro) === 2025-10-27 === * 14:15 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|40579b07e7}} (l10n updates: el, tg) === 2025-10-20 === * 12:30 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|9d7035ced8}} (l10n updates: frp) === 2025-10-13 === * 18:10 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|ffc43e9936}} (l10n updates: lb) === 2025-09-29 === * 18:23 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|f458d2938d}} (l10n updates: ko-kp) === 2025-09-11 === * 18:34 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|afc726918e}} (l10n updates: rki) === 2025-09-04 === * 15:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c35d575859}} (l10n updates: nb) === 2025-09-01 === * 18:40 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1e9e985c5d}} (l10n updates: vi) === 2025-08-25 === * 17:12 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|457dd44066}} (l10n updates: aig, pt) === 2025-08-24 === * 22:40 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3669a5db51}} (upgrade dependencies, including PyMySQL 1.1.2 with Python 3.13 compatibility) === 2025-08-21 === * 19:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3776ee4000}} (l10n updates: yue-hant) === 2025-08-18 === * 19:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|9b5fc1cef3}} (Portuguese Wikifunctions) * 19:07 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2886086be9}} (add missing wikifunctions_intro to german-noun-masculine) * 17:57 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|243f584c59}} (l10n updates: ar, tg, yue-hant) === 2025-08-14 === * 16:25 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d4ef0aa38c}} (l10n updates: yue-hant) === 2025-08-07 === * 19:50 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ebeea040cb}} (l10n updates: yue-hant) === 2025-07-31 === * 17:24 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6b1e8c15ae}} (l10n updates: pt, pt-br, sl) === 2025-07-24 === * 19:15 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ece1469a65}} (l10n updates: pt) === 2025-07-17 === * 19:29 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|672ec5bee8}} (l10n updates: yue-hant, zh-hant) === 2025-07-13 === * 15:50 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3c977ccc7b}} (specify .python-version) * 14:46 lucaswerkmeister: disregard the previous message, wrong tool 🤦 * 14:46 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["CONSUMER_KEY"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_KEY === 2025-07-12 === * 21:01 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|336dd318ca}} (upgrade to Python 3.13) * 20:56 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|dd03e95876}} (update documentation; no-op deployment, just to test the new buildservice procedure and put it in a single command) * 20:50 wmbot~lucaswerkmeister@tools-bastion-13: cp www-unused-tool-now-runs-on-buildservice/python/src/service.template . * 20:49 wmbot~lucaswerkmeister@tools-bastion-13: mv www www-unused-tool-now-runs-on-buildservice * 20:47 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e6d259028e}} (successful migration to buildservice) * 18:09 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|0bf37ac8d8}} (tried but failed to migrate to build service [OSError: No username set in the environment], will try again later, for now running in python3.11 again) * 17:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f8033bda4a}} (read config from envvars) * 17:41 lucaswerkmeister: commented out config.yaml, should use envvars instead * 17:41 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["SECRET_KEY"]))' {{!}} toolforge envvars create TOOL_SECRET_KEY * 17:40 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["CONSUMER_SECRET"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_SECRET * 17:40 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["CONSUMER_KEY"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_KEY * 17:16 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|15261afefb}} (change config keys to uppercase to work around [[phab:T374780|T374780]]) === 2025-07-10 === * 19:59 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8c32f8b90c}} (l10n updates: hu) === 2025-07-07 === * 06:33 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|cc28ff0494}} (l10n updates: et, it, nn, pt-br, ru) === 2025-06-16 === * 17:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c7b450ab94}} (update code for newer mwapi version) === 2025-06-11 === * 12:23 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|cae8c3c341}} (upgrade dependencies, including toolforge 6.1.0; use toolforge.load_private_yaml() from [[phab:T333728|T333728]]) === 2025-05-31 === * 13:53 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|706110e863}} (l10n updates: da, lb) === 2025-05-13 === * 16:58 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8a583bf6ff}} (l10n updates: tg) === 2025-05-06 === * 17:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|7349295f62}} (l10n updates: el) * 17:24 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e969f66351}} (update absolute_construction item ID) === 2025-04-22 === * 23:10 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d9516b6b1c}} (Quechua verb Wikifunctions) === 2025-04-21 === * 18:20 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c6d552c9c3}} (upgrade dependencies, including toolforge-i18n 0.1.2) === 2025-04-19 === * 10:56 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5425b40c0f}} (l10n updates: es) === 2025-04-14 === * 19:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ae10863f8f}} (l10n updates: af) === 2025-04-07 === * 18:01 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e106b7b684}} (Quechua verbs + l10n updates: es, pa, qu, zh-hant) === 2025-04-04 === * 19:48 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a377a0be8c}} (remove unneeded CSS) === 2025-03-29 === * 21:16 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|98e408e5a6}} (Russian perfective verbs) === 2025-03-15 === * 11:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ab6621b22d}} (l10n updates: ar) === 2025-03-11 === * 20:35 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d6def84813}} (l10n updates: lb) === 2025-02-21 === * 20:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|81611bc5dc}} (l10n updates: pa, tr) === 2025-02-04 === * 21:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2ccb28ad17}} (l10n updates: lb) === 2025-01-24 === * 10:21 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|223cafa209}} (l10n updates: ms) === 2025-01-09 === * 21:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|cebad0e4dd}} (l10n updates: ia, pa) === 2025-01-06 === * 20:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e7e3f2a500}} (l10n updates: cs, he) === 2024-12-21 === * 22:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|eb9d0ae3c2}} (l10n updates: lb, pa; also upgrade dependencies, including Flask 3.1.0 and Jinja2 3.1.5) === 2024-12-12 === * 22:13 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5ffdfb2c55}} (l10n updates: he, nl) === 2024-11-18 === * 19:47 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3933dbfa7f}} (l10n updates: af, ar, de, fr, gl, he, krc, mk, pa, sk, zh-hans); manually restored sh-latn ([[phab:T379188|T379188]]) === 2024-11-04 === * 17:32 wmbot~lucaswerkmeister@tools-bastion-13: webservice stop; webservice start # [[phab:T378976|T378976]] === 2024-11-02 === * 16:54 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a6768b885c}} (add setting for using Wikifunctions) * 14:01 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4fdd9491ee}} (improve Wikifunctions UI) * 09:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f28c3414ad}} (upgrade dependencies, including Werkzeug 3.1.0); also upgraded pip from 24.2 to 24.3.1 === 2024-10-25 === * 19:30 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8cdbda6ce3}} (upgrade dependencies, including Werkzeug 3.0.6) === 2024-10-13 === * 11:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|bfcaca2fa3}} (upgrade dependencies, including MarkupSafe 3.0) === 2024-10-03 === * 16:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e6377a9095}} (upgrade dependencies, including toolforge_i18n 0.1.1 and Werkzeug 3.0.4) * 13:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a81b469204}} (l10n updates: ms-arab) === 2024-09-26 === * 15:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|96f45731db}} (l10n updates: ar, ms-arab) === 2024-09-11 === * 21:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|38b3b281ed}} (fix two ZIDs for Breton templates) === 2024-09-01 === * 14:32 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|53a1efcc14}} (l10n updates: cy, uk) === 2024-08-18 === * 12:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|309b33b80b}} (l10n updates: pl, tg) * 12:02 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6deace1e36}} (Italian masculine+feminine nouns, dependency upgrades) === 2024-08-12 === * 18:19 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|246f9d26da}} (l10n updates: tg) === 2024-08-05 === * 13:34 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e3448958a0}} (upgrade toolforge_i18n to 0.0.7) === 2024-07-31 === * 19:15 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4775170045}} (upgrade toolforge_i18n to 0.0.6; also upgrade pip to 24.2) === 2024-07-26 === * 21:11 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|bb61fc3c89}} (l10n updates: vi [no actual translation changes, one addition to the authors, presumably their edit got reverted]) === 2024-07-22 === * 18:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|13c4824e3a}} (change Babel code of kaa from kk to uz) === 2024-07-21 === * 18:12 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e856c9b2d2}} (upgrade toolforge_i18n to 0.0.5; also upgrade pip to 24.1.2) === 2024-07-08 === * 18:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d6fa2d82b8}} (l10n updates: ja) === 2024-07-07 === * 18:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f3b3981ec9}} (upgrade toolforge_i18n to 0.0.2; also upgrade pip from 24.0 to 24.1.1) === 2024-07-05 === * 12:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1013a7234d}} (l10n updates: ar, de, uk) === 2024-06-18 === * 19:05 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8530f5f235}} (l10n updates: eo, fa, kaa, lb) === 2024-06-15 === * 13:58 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|9cb9b3dfde}} (install toolforge_i18n from PyPI) === 2024-06-07 === * 09:06 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|253d1b0f45}} (l10n updates: pa) === 2024-05-26 === * 13:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|48a5585566}} (support opting out of Wikifunctions mode) === 2024-05-20 === * 13:34 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4d952df88b}} (l10n updates: ms) === 2024-05-13 === * 18:19 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1c3d80a5e6}} (l10n updates: eu, zh-hans) === 2024-05-11 === * 12:50 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|bfccf1614c}} (more Hebrew verb templates) === 2024-05-09 === * 15:40 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5b88dd1ce1}} (improve toolforge_i18n and upgrade dependencies for newer Babel and Werkzeug) === 2024-05-06 === * 17:04 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|c5618f5968}} (set bot flag in bulk mode) * 15:43 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|8fa2740a72}} (README update, pulled without webservice restart) === 2024-05-05 === * 11:47 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|400cc9cb84}} (update Hebrew pa'al verbs) * 11:03 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|19c8210d68}} (Hebrew pa'al verbs) === 2024-05-04 === * 12:17 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|deb5b1c44e}} (extract toolforge_i18n library: [[phab:T363626|T363626]]) === 2024-05-03 === * 17:08 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|89c98da81f}} (upgrade dependencies for Python 3.12 compat; also upgraded pip<nowiki>{</nowiki>,-tools<nowiki>}</nowiki> and wheel while I’m at it) === 2024-04-22 === * 20:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1be060cd5c}} (l10n updates: ja) === 2024-04-18 === * 19:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f1a2cd1995}} (use public WikiLambda API) === 2024-04-17 === * 19:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e5d2281cea}} (l10n updates: krc) * 18:13 wmbot~lucaswerkmeister@tools-bastion-13: pulled {{Gerrit|fa6c094165}} (templates CC BY-SA 3.0 → 4.0; no webservice restart needed) === 2024-04-08 === * 17:58 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|559eb5bc47}} (make session permanent after login) === 2024-04-06 === * 13:35 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|1569542ce6}} (l10n updates: el, fa, zh-hant) === 2024-03-24 === * 12:21 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|b630198d56}} (l10n updates: fi, ms-arab) === 2024-03-15 === * 19:41 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|272a303c09}} (Danish adverbs) * 16:33 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|8f4985e682}} (improve tests; should have no production impact but I pulled+restarted anyway ^^) === 2024-03-10 === * 18:40 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|c62a9c1927}} (Maltese templates, including support for non-first forms to be the lemma: Maltese nouns have the third person singular as the lemma) * 12:42 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|bf88439696}} (l10n updates: fi, ko) === 2024-03-04 === * 18:12 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|e7a659802c}} (l10n updates: ar, io, lb) === 2024-03-03 === * 00:26 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|4106259494}} (l10n updates: ht, hu) === 2024-02-28 === * 18:50 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|3030faaa3c}} (health-check-path, [[phab:T341919|T341919]]) === 2024-02-23 === * 20:21 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|968078dbcd}} (l10n updates: hu, lt) [relog from 19:35 UTC, stashbot had problems] === 2024-02-17 === * 10:51 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|f88f2445fc}} (Esperanto adjective+verb Wikifunctions) === 2024-02-13 === * 18:51 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|85b6ec6534}} (l10n updates: ja, kaa) === 2024-02-07 === * 17:59 wmbot~lucaswerkmeister@tools-sgebastion-10: started webservice again (and patched the startup probe into it); took a while to come up but now it seems to be working * 17:49 wmbot~lucaswerkmeister@tools-sgebastion-10: stopped webservice, restart wasn’t working so let’s try harder * 17:45 wmbot~lucaswerkmeister@tools-sgebastion-10: restarted webservice, log was full of various errors === 2024-02-06 === * 20:39 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|344fd43224}} (update Breton noun Wikifunctions) === 2024-01-31 === * 19:13 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|604b43e316}} (l10n updates: it) === 2024-01-26 === * 19:03 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|249d9da0b7}} (l10n updates: id, kaa, ru, th) * 00:22 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|886d99636e}} (more Esperanto noun Wikifunctions) === 2024-01-22 === * 18:34 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|0b062cafa9}} (Norwegian language name templates) === 2024-01-13 === * 15:51 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|d24dc99256}} (l10n updates: ar) === 2024-01-07 === * 13:08 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a97ab796ea}} (wikifunctions: first form from lemma, if missing) === 2024-01-06 === * 16:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ea6b02ac57}} (Wikifunctions returning lists, Z11991→Z12689) === 2024-01-04 === * 12:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|82f5578b9a}} (l10n updates: ca, de, pl) === 2023-12-30 === * 15:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5baa3871d0}} (l10n updates: lb, zh-hans) === 2023-12-28 === * 10:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|45b698823a}} (update Italian adjectives) * 10:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4e68d80748}} (i18n updates: uk) === 2023-12-17 === * 18:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7611d4e980}} (l10n updates: ia, krc, sv) === 2023-12-11 === * 18:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|424615e192}} (l10n updates: de, krc, lb, nl, pnb) === 2023-12-09 === * 16:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fdc7c853c4}} (update Breton noun Wikifunctions) === 2023-12-05 === * 19:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|95ee032c68}} (l10n updates: ca, hno, io, it, pnb, sl, tr; i18n test improvements and fixes) === 2023-12-01 === * 19:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ba19a1cd5f}} (l10n updates: ja, sk, zh-hans) * 19:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7acef657d0}} (update Croation noun Wikifunctions) === 2023-11-29 === * 17:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|54a614fd41}} (fix some spacing) === 2023-11-25 === * 12:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|171fc2ea54}} (l10n updates: br) === 2023-11-19 === * 16:36 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0416376e58}} (German masculine noun Wikifunctions) * 15:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|11e7d12745}} (one more set of German neuter noun Wikifunctions) * 13:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|442f510a5b}} (German neuter noun Wikifunctions) === 2023-11-18 === * 17:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8c123e032e}} (l10n updates: br, he, ko) === 2023-11-12 === * 17:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cc2cf0ceaf}} (l10n updates: bn, fa, fr, gl, it, lb, mk, nb, vi, zh-hans, zh-hant; yue removed, existing settings are automatically replaced with zh-hant) === 2023-11-04 === * 18:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|203bc87b5b}} (more German feminine noun Wikifunctions – m/n will follow later) * 12:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bfa1ad40e0}} (first German Wikifunctions: feminine noun -(e)n plural) * 10:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|365c7e2814}} (cache Wikifunctions results) === 2023-11-01 === * 19:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|240a228f49}} (tests for Wikifunctions, pulled without webservice restart) * 18:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|92a91137e6}} (Wikifunctions for Breton nouns) === 2023-10-30 === * 19:10 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bea713bc0c}} (l10n updates: br) * 00:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f33e56597c}} (update French Wikifunctions button label) === 2023-10-29 === * 17:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cca1b1af23}} (Wikifunctions support in edit mode) * 16:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b5af35ab2b}} (fix Croatian feminine noun instrumental plural) * 16:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|052ba84de7}} (fix crash for users without Wikifunctions account) * 15:49 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5c3dc0dd6d}} (experimental Wikifunctions for Esperanto nouns, nominative plural only) * 14:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0ab3c10890}} (fix Wikifunctions buttons lang= and dir=) * 14:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5657d03fbb}} (experimental Wikifunctions for French nouns) * 14:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a64b857485}} (experimental Wikifunctions for Croatian nouns) * 14:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|40b0df49ee}} (experimental Wikifunctions support – happy birthday Wikidata 🎉) === 2023-10-28 === * 22:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c1f7a335e8}} (fix input patterns) === 2023-10-25 === * 17:59 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cdb1d34e11}} (Werkzeug 3.0.1) === 2023-10-20 === * 17:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|df7cf04757}} (i18n updates: io, ms-arab) === 2023-10-10 === * 19:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ad16425ee2}} (l10n updates: nl, uk, zh-hans) === 2023-10-06 === * 17:36 wm-bot: <lucaswerkmeister> deployed {{Gerrit|72e12c5a2c}} (l10n updates: zh-hans) + remove hardcoded support for Karai-karai now that MediaWiki has it === 2023-10-01 === * 17:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|216afb45fa}} (update dependencies, Flask+Werkzeug 3) === 2023-09-24 === * 13:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e5ae3295bb}} (Babel language code of Aragonese, to silence log warnings) * 13:29 wm-bot: <lucaswerkmeister> deployed {{Gerrit|45aa8fe43b}} (Danish proper nouns) === 2023-09-22 === * 16:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|72c20b3b3e}} (l10n updates: cs, kai [new, with temporary hacks], tr, zh ⇒ zh-hans) === 2023-09-04 === * 16:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|85d978855f}} (Italian adverbs) === 2023-08-28 === * 18:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|48e3991eb6}} (fix typo in armenian-noun-singulare-tantum) === 2023-08-27 === * 14:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ea49f8c2c7}} (update dependencies) * 13:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|05522cee84}} (update Italian) === 2023-08-24 === * 17:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c19c9624ba}} (l10n updates: ca, fa, io) === 2023-08-12 === * 11:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e0cf031e70}} (l10n updates: it) === 2023-08-08 === * 18:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|56acd0944a}} (l10n updates: tr) === 2023-07-27 === * 12:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5d374c3787}} (l10n updates: ban, de, gl) === 2023-07-19 === * 12:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4fa53fae89}} (l10n updates: pt-br) === 2023-07-18 === * 08:03 wm-bot: <lucaswerkmeister> deployed {{Gerrit|474e48d752}} (update Breton grammatical feature) === 2023-07-15 === * 12:03 wm-bot: <lucaswerkmeister> pip-sync (i.e., actually install dependencies in the new venv, which I completely forgot to do earlier) * 11:31 wm-bot: <lucaswerkmeister> kubectl patch deployment lexeme-forms --patch-file patch-add-startup-probe.yml * 11:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|02f72f81a2}} (Python 3.11) === 2023-07-13 === * 13:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d7fea069ba}} (l10n updates: pl) === 2023-07-10 === * 17:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|42679bb5dc}} (l10n updates: yue) === 2023-07-09 === * 14:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|78711ad373}} (l10n updates: ms) === 2023-07-02 === * 13:00 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4e2653cf19}} (revert recent punjabi-noun-masculine-guru change) * 12:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c9e84dfb8d}} (add separators to Dutch nouns) === 2023-06-30 === * 18:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1ed453b5d5}} (l10n updates: sh → sh-latn, tt → tt-cyrl, [[phab:T336606|T336606]]) === 2023-06-27 === * 20:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fe5983c571}} (l10n updates: ba) === 2023-06-25 === * 14:00 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3ad131b7bf}} (Aragonese common nouns) === 2023-06-24 === * 09:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|213bfabfb4}} (underline links on hover again) === 2023-06-22 === * 20:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|63c042d9b3}} (l10n updates: it) === 2023-06-20 === * 18:27 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7081d2769e}} (support language fallback and ?uselang) * 17:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3e76345eb5}} (l10n updates: ba, id, nb, xmf) === 2023-06-18 === * 11:36 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bebc116e22}} (Bootstrap 5) * 11:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d53e455ef7}} (update Malayalam nouns and add adjective template) === 2023-06-16 === * 17:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|248590aeb0}} (l10n updates: ba, id, pl) === 2023-06-13 === * 17:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e9112d022e}} (l10n updates: es) === 2023-06-11 === * 11:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fb8c4a30ff}} (update punjabi-noun-masculine-guru) === 2023-06-09 === * 16:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e059c8bbd6}} (l10n updates: fi); also, last time I forgot to git rebase, so this actually includes {{Gerrit|2035050d28}} (l10n updates: sv) as well === 2023-06-07 === * 07:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2035050d28}} (l10n updates: sv) === 2023-06-04 === * 22:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|08962e4902}} (update past transgressive item ID after merge; only affects czech-verb-perfective) === 2023-05-31 === * 20:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1ec8c72304}} (Russian adjectivse: remove compound lexical categories) === 2023-05-29 === * 15:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a5e90a0e02}} (update dependencies) === 2023-05-27 === * 19:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|07deb7a083}} (Punjabi additive double causative verbs) * 17:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|889b4ce276}} (Punjabi additive causative verbs) * 15:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|467d5b9f34}} (Punjabi transitive verbs) * 15:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6c76e2d3b5}} (fix two Punjabi placeholders) === 2023-05-25 === * 20:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a50e668166}} (l10n updates: ca, es, fa, fi, ru, tr, ur) === 2023-05-19 === * 17:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b59c2f0aad}} (l10n updates: es, hi, zh-hant) * 16:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b80c8ff9db}} (fix “logged in” indicator in several languages) === 2023-05-18 === * 08:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7f76b0f203}} (l10n updates: br, de, fr, he, hi, hno, ia, mk, pa, pnb, ru, sa, sl, ur) === 2023-05-13 === * 17:29 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7d3ab49b06}} (l10n updates: ar, bn, de, eo, fa, fi, fr, he, hy, ia, it, ja, ko, mk, ms, nb, pnb, ru, skr-arab, sl, zh-hant) * 12:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|dfcf34ed51}} (make “logged in as” translatable) * 11:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|65cb94f3c7}} (punjabi-verb-basic-intransitive templates) === 2023-05-12 === * 20:41 wm-bot: <lucaswerkmeister> deployed {{Gerrit|15b7403971}} (fix stray character) === 2023-05-08 === * 21:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|db3dd67b8a}} (make more translations available and tweak Babel language codes) * 20:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a7bf757be9}} (fix message keys broken by previous deployment) * 20:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5f01c59794}} (refactor message keys from _ to -, should make no difference) * 19:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b51f930220}} (user interface language setting) === 2023-05-05 === * 12:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|72a006c6ea}} (l10n updates: mrh, ta) === 2023-05-02 === * 00:08 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5f83647d21}} (test-only change, pulled without webservice restart) === 2023-05-01 === * 23:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|88da33ddc5}} (GitHub actions only change, pulled without webservice restart) * 17:41 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1380884cce}} (upgrade dependescies, GHSA-m2qf-hxjv-5gpq) * 15:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|75230357a4}} (l10n updates: lt) * 15:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1554678038}} (improve matching.py for upcoming templates, should make no difference at the moment) * 14:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|692e255a50}} (refactor matching.py, should make no difference) === 2023-04-30 === * 15:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|db66a9373c}} (refactor statement groups; should make no difference) === 2023-04-25 === * 21:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9059e45cda}} (update dependencies, Werkzeug 2.3.0 / Flask 2.3.1) * 18:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c6dc908e1e}} (refactoring for somevalue support, should make no difference yet) === 2023-04-24 === * 19:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d5b5c8994f}} (preparation & refactoring, no visible changes) === 2023-04-23 === * 18:41 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0f96d60736}} (Punjabi adverbs) * 18:00 wm-bot: <lucaswerkmeister> deployed {{Gerrit|75af96b851}} (Punjabi adjectives) * 15:27 wm-bot: <lucaswerkmeister> deployed {{Gerrit|934f5cffdb}} (Yoruba adjectives) === 2023-04-22 === * 16:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a074fd9c64}} (trim spaces) * 15:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fdb0552957}} (remove spaces) * 15:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b6a1268b21}} (Punjabi nouns) === 2023-04-15 === * 15:41 wm-bot: <lucaswerkmeister> deployed {{Gerrit|604df5c72e}} (two more variables) * 15:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1b999f4661}} (use variables for entity IDs; should make no difference at runtime) * 14:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|24fb20fd19}} (sort sets for JSON output) === 2023-04-12 === * 20:27 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5b07592a7e}} (two style improvements) === 2023-04-10 === * 17:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|282a7b6b18}} (l10n updates: anp; currently skipped because unsupported by Babel) === 2023-04-08 === * 11:41 wm-bot: <lucaswerkmeister> deployed {{Gerrit|994cbd48b0}} (fix typo in a Hindustani template) === 2023-04-01 === * 18:41 wm-bot: <lucaswerkmeister> deployed {{Gerrit|08ac04d468}} (fix Hindko template order) === 2023-03-22 === * 20:54 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b40cefa378}} (change Hindko templates to hno) === 2023-03-19 === * 21:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cf1e031a43}} (l10n updates: fi, tt) === 2023-03-13 === * 21:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d7ba3ddc23}} (l10n updates: hi, pa, tt, ur) === 2023-03-08 === * 22:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8da3525baf}} (fix lowercase item ID in portuguese-noun-biform) * 22:08 wm-bot: <lucaswerkmeister> deployed {{Gerrit|de17c6bdf6}} (fix hindustani-verb-additive-causative-double-ur label) * 22:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a99078e1c5}} (hindustani-verb-additive-causative-double templates) === 2023-03-06 === * 21:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8828e3269e}} (l10n updates: tt) * 21:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2cbf107d6e}} (hindustani-verb-additive-causative templates) * 20:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0f7a634e72}} (fix Hindustani verb placeholders) === 2023-03-05 === * 21:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c325634bc3}} (hindustani-verb-additive-transitive templates) * 19:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|85cbe15d08}} (hindustani-verb-basic-transitive templates) * 13:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|00f87cf139}} (hindustani-verb-basic-intransitive templates) === 2023-03-03 === * 20:59 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c310fd9d88}} (update Hindustani labels, and l10n update: tt) === 2023-02-27 === * 19:54 wm-bot: <lucaswerkmeister> deployed {{Gerrit|50aa1e2dc5}} (l10n updates: hi, hno, pa, pnb, ur) === 2023-02-26 === * 21:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1d6c0caecd}} (Hindustani non-verb templates – verbs still TBD, need more time) * 15:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2feff85812}} (use hno translations) === 2023-02-22 === * 20:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9e667986b4}} (l10n updates: hi, ur) === 2023-02-14 === * 19:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9debac9385}} (update dependencies, especially Werkzeug 2.2.3 with two security fixes; venv rebuilt from scratch to avoid NFS issues) * 19:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bfd63ebac1}} (l10n updates: hno); also, turns out I didn’t git rebase in the last deployment, so this *actually* deploys the Danish nouns update and pl l10n update === 2023-02-09 === * 20:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2912ebfa68}} (update Danish nouns, and l10n updates: pl) === 2023-01-31 === * 19:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bfaf13f447}} (update github actions; pulled without webservice restart) * 19:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f9bf85df5f}} (l10n updates: cy) === 2023-01-29 === * 12:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3ca9650fe1}} (Danish adjectives) === 2023-01-09 === * 19:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4857d874ce}} (l10n updates: pa) === 2023-01-03 === * 15:29 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5c27eaec33}} (l10n updates: pl) === 2022-12-30 === * 12:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|95b9026d22}} (l10n updates: pa, zh) === 2022-12-28 === * 15:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3c47032838}} (fix bulk result display when given lexeme ID) === 2022-12-26 === * 11:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b51ddc8c08}} (update Armenian noun templates) * 11:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bdaa43aef3}} (preserve target_hash in more places) === 2022-12-16 === * 21:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4802902384}} (l10n updates: yue) === 2022-12-08 === * 19:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3f6b15c1f0}} (l10n updates: fa, gl, pl, sl) === 2022-12-06 === * 13:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|97001e468b}} (fix missing statements) * 13:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|45a026916c}} (fix Hindko feminine noun template) === 2022-12-05 === * 21:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4d781fb933}} (Hindko noun templates) * 20:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2cb7ac792f}} (l10n updates: pnb) === 2022-12-04 === * 17:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|82a2272a2f}} (three new Norwegian Nynorsk noun templates) === 2022-11-29 === * 21:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b0ebae4629}} (l10n updates: el) === 2022-11-27 === * 19:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bb7cf271ae}} (l10n updates: fa) === 2022-11-19 === * 15:03 wm-bot: <lucaswerkmeister> deployed {{Gerrit|10af55574b}} (more Bokmål and Nynorsk templates) === 2022-11-15 === * 20:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5897fd06ee}} (Danish nouns fix) * 20:23 wm-bot: <lucaswerkmeister> ionice -c3 zstd --rm uwsgi.log.1668543276 # 8.85%, {{Gerrit|520591680}} => {{Gerrit|46091850}} bytes) * 20:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2b53b1199c}} (rotate uwsgi.log after 100 MiB) * 19:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0429d7d80b}} (update Danish nouns+verbs) === 2022-11-10 === * 13:41 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5160edb9ca}} (l10n updates: pnb) * 13:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|127e065522}} (NFC-normalize lemma for search) === 2022-11-07 === * 21:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0c7095c96d}} (Polish adjectives, positive only) * 20:54 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c653fb07e2}} (l10n updates: es, hy, pnb) === 2022-11-05 === * 14:02 wm-bot: <lucaswerkmeister> git gc (.git 19M → 1.1M) * 13:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8feb3f86d4}} (extra GitHub actions job, pulled without webservice restart) * 12:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d38d5ba55c}} (uninstall dev dependencies in production; reduces venv size from ca. 142 MB to ca. 75 MB, or about by half) * 12:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b7f4d4ba31}} (added test; pulled without webservice restart) * 11:03 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ccecd3bb87}} (l10n updates: krc, zh) === 2022-10-27 === * 12:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|03b6dd3b71}} (l10n updates: pnb) === 2022-10-26 === * 20:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2feba604c7}} (update dependencies, use PEP 655 NotRequired) * 19:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|55f9b203e5}} (l10n updates: sl) === 2022-10-23 === * 16:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3844a7df05}} (French verbs) === 2022-10-17 === * 19:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b098904d43}} (l10n updates: ja, pnb) === 2022-10-14 === * 18:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a829f83124}} (l10n updates: ca, hi, sh, sl) === 2022-10-05 === * 20:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|953b553968}} (translate Hebrew adjective template label) * 18:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|93ebb772c5}} (more Spanish templates) === 2022-10-01 === * 19:00 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b770688eb1}} (Hebrew adjectives) * 18:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8137259ca6}} (Flask 2.2) === 2022-09-23 === * 19:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c66922341d}} (l10n updates: ar, ku, sl) === 2022-09-18 === * 16:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8d996c1fa4}} (l10n updates: ar) === 2022-09-10 === * 18:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|609066f02b}} (README fix, pulled without webservice restart) * 16:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|52570991cd}} (diffusion → gitlab) === 2022-08-29 === * 20:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fa8f5d87a4}} (l10n updates) === 2022-08-25 === * 14:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|019b4ecc79}} (optimize messages with unused GENDER magic word) * 14:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|dd6cb7f08b}} (l10n updates) === 2022-08-03 === * 19:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a11b6a55f6}} (l10n updates) === 2022-07-21 === * 23:03 wm-bot: <lucaswerkmeister> deployed {{Gerrit|38141487d1}} (l10n updates) === 2022-07-17 === * 17:59 wm-bot: <lucaswerkmeister> deployed {{Gerrit|238f943e8a}} (add more typing; hopefully no functional changes) === 2022-07-13 === * 20:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d5cb20368d}} (l10n updates) === 2022-07-02 === * 19:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d3e2185bbc}} (l10n updates) === 2022-06-29 === * 19:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6ac757a997}} (Igbo verbs + pronouns) === 2022-06-16 === * 21:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|466976ba49}} (l10n updates) === 2022-06-14 === * 22:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b0143851e0}} (l10n updates) === 2022-05-26 === * 20:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|24d9b273c5}} (l10n updates) === 2022-05-17 === * 19:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8cdef0cf20}} (l10n updates) === 2022-05-03 === * 20:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d8429a8740}} (l10n updates) === 2022-04-29 === * 19:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fd45333563}} (l10n updates, extra unit test) === 2022-04-28 === * 23:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|860abb205b}} (Bokmål passive verbs) === 2022-04-27 === * 20:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c92b363387}} (Mandarin templates) === 2022-04-25 === * 19:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7b5d0d7298}} (l10n updates) === 2022-04-22 === * 11:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d769b4ed8b}} (l10n updates) === 2022-04-20 === * 19:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|89a5273967}} (l10n updates) === 2022-04-15 === * 18:16 wm-bot: <lucaswerkmeister> pulled {{Gerrit|24d5774c5f}} (test-only change, so no restart) * 18:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|54a5376631}} (update German verbs) * 16:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9a2cefe8e6}} (updated Portuguese templates) === 2022-04-04 === * 19:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|197baf2940}} (l10n updates) === 2022-03-30 === * 18:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c6001bf897}} (l10n updates; use pip-tools, includes some package updates such as Flask 2.0.2→2.1.0; clean up service.template) === 2022-03-19 === * 12:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f573b558d4}} (l10n updates) === 2022-03-11 === * 00:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d7787d7536}} (l10n updates) === 2022-03-05 === * 18:08 wm-bot: <lucaswerkmeister> deployed {{Gerrit|72f2adc394}} (l10n updates) === 2022-02-28 === * 12:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|04ba7580ab}} (l10n updates) === 2022-02-25 === * 00:49 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1506d1a9e9}} (l10n updates) === 2022-02-22 === * 00:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1fc2f98450}} (l10n updates) === 2022-02-15 === * 13:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|56e69bad1a}} (l10n updates) === 2022-02-11 === * 23:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b4624e0bbc}} (l10n updates) === 2022-02-07 === * 13:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b3c5446831}} (l10n updates) === 2022-01-30 === * 12:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c1d6a79ed2}} (update Odia nongendered adjectives) === 2022-01-22 === * 17:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b1cc42ef84}} (Odia nouns) * 16:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b62723fb6f}} (update Odia adverbs) === 2022-01-16 === * 19:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|504c5481e9}} (update Spanish verbs) * 18:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|68234bd17d}} (Odia adjectives and adverbs) === 2022-01-10 === * 18:59 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d1da801731}} (l10n updates) === 2022-01-06 === * 18:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|57dc392b8f}} (l10n updates) === 2022-01-03 === * 18:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|aacaae3cd6}} (revert update of indefinite item ID after merge, I flipped the items) * 15:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2eb6822ed2}} (l10n updates) * 15:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7312514fc8}} (update indefinite item ID after merge) === 2022-01-01 === * 23:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d6110ed631}} (l10n updates) === 2021-12-17 === * 21:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|20c4392de6}} (l10n updates) === 2021-12-02 === * 23:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2a2cb9b211}} (l10n updates) === 2021-11-25 === * 21:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|baef3a16f6}} (l10n updates) === 2021-11-18 === * 13:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e001c252c5}} (l10n updates, including initial Yoruba translations) === 2021-11-14 === * 14:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c113d4dd77}} (Yoruba nouns) === 2021-11-08 === * 22:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|85719cf3ae}} (update Portuguese idioms) * 22:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e58c43ab3e}} (Portuguese idioms quickfix) === 2021-11-07 === * 19:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|91216ed64b}} (Portuguese idioms) === 2021-11-06 === * 12:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7ef5eb34a3}} (fix Manbhumi bulk mode link) === 2021-11-04 === * 12:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d649d7a24a}} (l10n updates) === 2021-10-25 === * 19:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0f5b5de66a}} (bump startupProbe failureThreshold 3→10) * 19:34 wm-bot: <lucaswerkmeister> deployment was successful after all 🤷 * 19:31 wm-bot: <lucaswerkmeister> belay that, the new pod hasn’t actually started properly. investigating * 19:29 wm-bot: <lucaswerkmeister> deployed {{Gerrit|754342b9a3}} (language name for bn-x-Q6747180) === 2021-10-18 === * 12:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|eae6c8d594}} (l10n updates) === 2021-10-16 === * 14:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1903c3d0eb}} (don’t show duplicate warning errors) * 12:09 wm-bot: <lucaswerkmeister> pulled {{Gerrit|8700382f98}} (rename confusingly named deplyoment patch file) without webservice restart * 12:04 wm-bot: <lucaswerkmeister> (correction on that last message, it’s a startup probe now, not a readiness probe) * 12:03 wm-bot: <lucaswerkmeister> patched readiness probe into deployment again * 12:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|19fb8c90ee}} (findDuplicates fix) with full stop/start to pick up label changes === 2021-10-13 === * 23:31 wm-bot: <lucaswerkmeister> fully restarted webservice (stop/start) to avoid label issues * 17:49 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e5c87ff53c}} (remove type ignore comments) and updated dependencies, including Flask 2.0.2 === 2021-10-11 === * 12:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fb32d04132}} (l10n updates) === 2021-10-10 === * 11:20 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bf2834c472}} (improve error handling) === 2021-10-04 === * 19:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1697521bf5}} (l10n updates) === 2021-09-25 === * 14:45 wm-bot: <lucaswerkmeister> removed old venv-3.7 * 13:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6f9e530018}} (mobile-friendly navbar) * 13:41 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ea93caf2ee}} (l10n updates) === 2021-09-19 === * 13:49 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3c1b6e0810}} (readinessProbe → startupProbe to avoid bloating access log); deployed by adding readinessProbe: null to the patch file and patching the deployment with that === 2021-09-14 === * 20:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c36ae4154a}} (l10n updates) * 19:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|902156ddb8}} (Croatian item ID fix) === 2021-09-12 === * 21:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4da7f64c4b}} (updates without downtime) * 20:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f21554ab71}} (refactoring, noop) * 15:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a4b05045d6}} (Croatian nouns) === 2021-09-08 === * 20:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2aa32a0f7f}} (l10n updates) === 2021-09-03 === * 15:54 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3698f0b79c}} (add passive forms to Norwegian Bokmal verbs) * 15:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8051248b60}} (l10n updates) === 2021-08-30 === * 18:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|dfc0838301}} (l10n updates) === 2021-08-25 === * 20:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|237a5414d5}} (l10n updates) === 2021-08-19 === * 20:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bcc4c3aa63}} (l10n updates) === 2021-08-17 === * 21:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0ca42b7cdb}} (more types) * 18:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2382c30c01}} (initial mypy setup) * 17:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c66572938e}} (python3.9) === 2021-08-16 === * 12:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|92e5e0d70c}} (l10n updates) === 2021-08-14 === * 12:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7a1980f4e2}} (l10n updates) === 2021-08-11 === * 19:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|37acc67c90}} (l10n updates) === 2021-08-02 === * 19:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|de5ab0e740}} (l10n updates) === 2021-07-19 === * 18:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0c9f1015c0}} (work around Firefox bug) === 2021-07-18 === * 18:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fa64f7e021}} (refuse to load non-user-readable config file, guard against recurrence of [[phab:T286414|T286414]]) * 13:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|61b1d0fd93}} (Igbo adjectives and fix nouns) === 2021-07-17 === * 11:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0d1f3d924e}} (load config file differently) === 2021-07-16 === * 19:23 wm-bot: <lucaswerkmeister> deployed {{Gerrit|37766a8002}} (l10n updates) === 2021-07-11 === * 20:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5dbc39eb5e}} (l10n update) * 17:03 wm-bot: <lucaswerkmeister> restarted webservice to pick up 1.3 version of OAuth consumer ([[phab:T286414|T286414]]) * 13:36 wm-bot: <lucaswerkmeister> chmod go-rwx www/python/src/config.yaml # [[phab:T286414|T286414]] === 2021-07-01 === * 23:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ac8779515d}} (l10n updates) * 23:37 wm-bot: <lucaswerkmeister> unlink ~/services.template # new version of webservice doesn’t like the symlink :( === 2021-06-28 === * 17:54 wm-bot: <lucaswerkmeister> deployed {{Gerrit|64c5584c9d}} (remove workaround for [[phab:T241422|T241422]]) * 17:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5565da07e5}} (l10n updates, especially Igbo translations) === 2021-06-22 === * 19:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c88b1962fa}} (Igbo nouns) === 2021-06-21 === * 20:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|19098277f4}} (l10n updates) === 2021-06-20 === * 12:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|afc6f6f242}} (update German verbs) === 2021-06-19 === * 19:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c5b12d5dc1}} (Malayalam proper nouns) * 19:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|05cd31e9bd}} (update Malayalam noun) === 2021-06-15 === * 20:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0b6fed0054}} (even more optional grammatical features) * 19:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d8eadd1cae}} (more optional grammatical features) * 18:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|61a5e0fc18}} (optional grammatical features) === 2021-06-14 === * 23:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|626b73a005}} (l10n updates) * 23:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|70efbdc1a7}} (update volitive item ID) === 2021-06-10 === * 20:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1f94df1209}} (l10n updates) === 2021-06-07 === * 21:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|547231388b}} (add create link for duplicates in bulk mode) * 20:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|daf88503e0}} (l10n updates) === 2021-06-06 === * 14:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2040a7497e}} (target_hash URL parameter) === 2021-06-05 === * 20:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fcf67b1016}} (improve title) === 2021-06-04 === * 23:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|16c0cd2606}} (improve batch mode results page) === 2021-05-31 === * 20:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|43a29c4369}} (replace deprecated function) * 20:00 wm-bot: <lucaswerkmeister> pip upgrade (Flask 2.0.1 and other updates) * 19:59 wm-bot: <lucaswerkmeister> briefly stopping tool to upgrade venv * 18:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|148dafa60b}} (l10n updates) === 2021-05-30 === * 14:59 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3c047f6aca}} (l10n updates) === 2021-05-24 === * 18:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6ffd1a2c1b}} (update Esperanto verb) * 16:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7d43094e56}} (l10n updates) * 11:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e0099e68d5}} (Swedish adjective) === 2021-05-22 === * 09:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|31e85bafcf}} (l10n updates) * 09:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|44812d4446}} (add Portuguese modal adverb) === 2021-05-15 === * 14:01 wm-bot: <lucaswerkmeister> tool should be back up (uwsgi.log went from 181M to 77M after moving pre-2021 data to separate files) * 13:56 wm-bot: <lucaswerkmeister> briefly stopping tool (few minutes) to cycle the uwsgi.log === 2021-05-13 === * 23:03 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3e2ceb0513}} (l10n updates) * 14:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|67e7cf3dfb}} (rename Swedish adjective template) * 13:49 wm-bot: <lucaswerkmeister> deployed {{Gerrit|95f40ac9d5}} (Norwegian Bokmål masculine/neuter nouns) === 2021-05-10 === * 16:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|248527544d}} (l10n updates) === 2021-05-09 === * 13:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5951b46450}} (fix lang= and dir= on index) === 2021-05-03 === * 19:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b159dd1060}} (l10n updates) === 2021-05-02 === * 11:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4c9a5f0ebf}} (duplicate check JS fixes) === 2021-05-01 === * 14:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|61744950f0}} (l10n updates) === 2021-04-26 === * 19:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|abf6719d31}} (Python 3.7 fix) * 19:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d15d0c5f2d}} (rename Dutch templates) * 18:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|868ee95cf2}} (l10n updates) === 2021-04-22 === * 19:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8ab4ceb62a}} (l10n updates) === 2021-04-19 === * 20:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2f8f589a62}} (Swedish proper nouns) * 20:23 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4effbc2a36}} (l10n updates) === 2021-04-17 === * 10:20 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1d10ab467e}} (fix bulk mode) === 2021-04-15 === * 19:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|051e3789a2}} (l10n updates) === 2021-04-14 === * 20:59 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b17ed175fe}} (move login hint up) * 20:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0006696173}} (remove automatic login redirect) * 12:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|30c561955f}} (login link in navbar) === 2021-04-12 === * 18:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e4682a00bd}} (Breton noun fixes) * 18:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a3a81d0c4b}} (l10n updates) === 2021-04-09 === * 18:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|18bb25abd0}} (l10n updates) === 2021-04-05 === * 13:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f5439f66a2}} (l10n updates) === 2021-04-04 === * 13:49 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9507991400}} (Malayalam verb fix) === 2021-04-03 === * 19:08 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3e2bc5b577}} (language code refactorings; should not result in any observable changes) * 18:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8416f8d861}} (more Breton nouns + adverbs) * 16:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|21201880f5}} (MarkupSafe-aware formatters; should not result in any observable changes) * 15:08 wm-bot: <lucaswerkmeister> deployed {{Gerrit|615bba5934}} (better bulk mode errors) === 2021-04-02 === * 19:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|be73b49e29}} (better language code handling) === 2021-04-01 === * 18:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f2b128273d}} (l10n updates) === 2021-03-30 === * 21:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7ff57d504e}} (l10n updates) === 2021-03-28 === * 19:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|43d0c29996}} (update Portuguese nouns) * 14:16 wm-bot: <lucaswerkmeister> <em>actually</em> deployed {{Gerrit|2ece3adc91}} (this time I did the <code>git rebase</code> but forgot the <code>webservice restart</code>, how’s that for a change) * 13:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2ece3adc91}} (Portuguese updates) === 2021-03-27 === * 14:00 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1f2a6f2e17}} (replace OrderedDict with dict) * 13:41 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4619f8cd03}} (remove duplicate template) * 13:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9ad3addd6a}} (Malayalam verbs, and vocative case for nouns) === 2021-03-26 === * 21:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5b44b44f52}} (Malayalam verbs) * 21:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|78a5c9a10a}} (indicate optional forms) === 2021-03-25 === * 19:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|77328e559d}} (optional forms) === 2021-03-24 === * 22:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ffa45a58b1}} (minifix) * 19:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ea6928faaa}} (clarify Norwegian Bokmål adjectives) * 19:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|99257d861c}} (Portuguese adjectives) === 2021-03-23 === * 21:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|253aed283c}} (Latvian nouns) * 19:27 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c0b2c473ff}} (add language code as ID on index page, suggested by jhsoby) === 2021-03-22 === * 21:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2e4e3dca5a}} (improved Malayalam nouns [not verbs as it says in the commit message, oops] + i18n updates) === 2021-03-16 === * 19:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|547b42f25f}} (Portuguese nouns, i18n updates) === 2021-03-13 === * 16:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f389caf9b2}} (gender i18n improvements, should be a no-op) === 2021-03-12 === * 20:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9500beeed4}} (three new translations) – should be a no-op but I didn’t want to leave it lying around without a webservice restart either * 19:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|aa07bef3bd}} (i18n update) – also, previous SAL message mentioned {{Gerrit|712d262475}} but that’s still in <code>git log @..@<nowiki>{</nowiki>u<nowiki>}</nowiki></code>, so I think I forgot to rebase last time === 2021-03-10 === * 20:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|712d262475}} (restore logging for generic API errors) * 19:59 wm-bot: <lucaswerkmeister> deployed {{Gerrit|94dfecbc2a}} (generic API error handler) === 2021-03-08 === * 14:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b7b55e1b33}} (more i18n improvements) * 11:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ea7cd3ac71}} (i18n from translatewiki.net – [[phab:T272243|T272243]]) === 2021-03-05 === * 22:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|109f22a415}} (Czech verbs update) === 2021-03-04 === * 21:08 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1435d31446}} (update Swedish translations) * 20:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|15a24d63eb}} (minor Czech verbs improvement) === 2021-02-28 === * 17:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|369031b945}} (minifix) * 17:10 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0455dc20f4}} (better OAuth error handling) === 2021-02-19 === * 18:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f66f631598}} (auth improvements) === 2021-02-18 === * 20:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a0ba7b84ab}} (quickfix) * 20:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|23ccbcf6f6}} (work around [[phab:T272319|T272319]]) === 2021-02-16 === * 20:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8d96af0ec2}} (add skip link) * 19:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3e716e6d6d}} (Bootstrap update) === 2021-02-13 === * 22:27 wm-bot: <lucaswerkmeister> deployed {{Gerrit|02a2edf583}} (edit summary fixes) * 18:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a7257a065e}} (code style fixes) * 16:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4e70e759d7}} (minifix) * 13:08 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fb17f5e4ef}} (edit mode fix for forms with multiple representations) === 2021-02-11 === * 22:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|81166d5c17}} (reduce [[phab:T230833|T230833]] workaround / "und" language codes) * 22:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8e718af67e}} (JS fix) === 2021-02-10 === * 20:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0d8279ca7f}} (<script> loading improvements) * 20:00 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1fe3d3589e}} (prevent double submit) === 2021-02-04 === * 20:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|32b6b23f72}} (German adverbs) === 2021-02-01 === * 21:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f4e7ba98a7}} (stop referrer-URL comparison) * 14:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d237952e44}} (fix current_url / CSRF detection) === 2021-01-30 === * 20:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a87ce138db}} (show bulk parse errors) === 2021-01-28 === * 20:03 wm-bot: <lucaswerkmeister> deployed {{Gerrit|868bccbbe7}} (fall back to en) * 19:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cb0855af48}} (simplify current_url) === 2021-01-27 === * 22:39 wm-bot: <lucaswerkmeister> deployed fixed version of test code, oops * 22:38 wm-bot: <lucaswerkmeister> deployed another version of test code * 22:26 wm-bot: <lucaswerkmeister> deployed uncommitted test code to print current_url debug output * 20:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1bc8d4232e}} (remove long-dead code about fixing the session cookie) * 20:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|03255e1408}} (pop OAuth redirect target) === 2021-01-13 === * 20:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e5725705d1}} (fix edit mode, drop form data stashing) === 2021-01-09 === * 21:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9a604413d3}} (German toponym) === 2021-01-07 === * 14:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|00d7fe313e}} (better edit links) === 2021-01-03 === * 11:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|db1e890252}} (grab cursor for draggable links) === 2020-12-30 === * 12:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|191518cbf9}} (edit lemma when adding first form) === 2020-12-23 === * 15:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6d8bae537b}} (Esperanto verb) * 14:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|69f610af18}} (Breton noun, without mutation, collective) === 2020-12-22 === * 11:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6e1185532d}} (Basque adjective) === 2020-12-14 === * 20:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9ba55b3ad3}} (fix current_url) === 2020-12-13 === * 00:49 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bb0cbfc6cb}} (language code in parentheses) === 2020-12-12 === * 18:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0ec650ea2f}} (autonyms on index page) === 2020-12-02 === * 21:35 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e5291d5cda}} (more Esperanto translations) === 2020-11-29 === * 21:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|915eb4016f}} (clarify German templates) === 2020-11-24 === * 21:58 wm-bot: <lucaswerkmeister> undeployed debug code, I don’t remember what it was for anymore * 21:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|59f2c38fed}} (the previously-uncommitted JS fix, now committed; some uncommitted debug code is still there) === 2020-11-21 === * 21:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1608cc4dd9}} (gender-dependent messages) === 2020-11-05 === * 19:51 wm-bot: <lucaswerkmeister> deployed uncommitted JS fix, to be committed later if it works as intended === 2020-10-29 === * 22:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1a150904fd}} (update Italian translations) === 2020-10-26 === * 21:27 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e3c4c2e664}} (Esperanto adjective) === 2020-10-25 === * 21:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bd4c445f02}} (edit mode fix) * 21:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|782dfdabee}} (fixes for edit mode and ordia links) === 2020-10-24 === * 13:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|792db2a9f9}} (edit mode language_code parameter) === 2020-10-19 === * 20:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a7fd004ef9}} (drag’n’drop fix; submit_lexeme debug code still there) === 2020-10-17 === * 14:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|19b5bc257a}} (more durable CSRF tokens; some uncommitted debug code to print submit_lexeme errors is still there) === 2020-10-08 === * 20:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fd8c692798}} (fix a crash; debug code still in place) === 2020-09-13 === * 08:36 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9f02b375f1}} (more conventient bulk mode transition; debug code still present) * 08:17 wm-bot: <lucaswerkmeister> deployed uncommitted extra logging for submit_lexeme errors in bulk mode === 2020-09-12 === * 12:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ce943856ed}} (fix Spanish feminine noun item ID) === 2020-09-08 === * 16:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9ac796e7aa}} (Manbhumi verbs) === 2020-09-06 === * 08:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|116e4123b0}} (fix Manbhumi duplicate search) === 2020-09-01 === * 15:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ef72c06ec8}} (Manbhumi adjectives and adverbs) === 2020-08-14 === * 19:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|13282d5404}} (Bengali verb updates) === 2020-08-12 === * 19:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e3291c8796}} (Bengali adverbs, other improvements) === 2020-08-04 === * 22:43 wm-bot: <lucaswerkmeister> <em>actually</em> deployed {{Gerrit|39457a18ab}} (forgot to git rebase) * 22:36 wm-bot: <lucaswerkmeister> deployed {{Gerrit|39457a18ab}} (Bengali adjectives and verbs) === 2020-07-08 === * 21:48 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b65c1018ff}} (translation update) === 2020-07-05 === * 22:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f29663c2b2}} (Norwegian Bokmål nouns) === 2020-07-04 === * 16:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cbf5ad6440}} (Norwegian Bokmål) === 2020-06-17 === * 23:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9b7349c602}} (update a Bengali template) === 2020-06-15 === * 20:54 wm-bot: <lucaswerkmeister> renamed default branch from master to main === 2020-06-14 === * 12:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8d5f428c3e}} (improved duplicate warning edit links) * 10:15 wm-bot: <lucaswerkmeister> *actually* deployed {{Gerrit|2efe64f7e5}} (forgot to git rebase) * 10:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2efe64f7e5}} (link edit mode in duplicate warning) === 2020-06-13 === * 21:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b42e79e6bb}} (more sections) * 17:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cf1079fda1}} (more section improvements) * 13:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c2e6d57a29}} (improved German sections) * 11:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4cd36a71a1}} (sections in edit mode) * 11:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4e288f0106}} (sections) * 08:54 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bfa46d522b}} (Czech edit mode translations) === 2020-06-07 === * 20:53 wm-bot84: <lucaswerkmeister> deployed {{Gerrit|9e4f3a1b65}} (two translation fixes) * 13:35 wm-bot84: <lucaswerkmeister> deployed {{Gerrit|09cc2017ec}} (Bengali nouns) === 2020-05-24 === * 13:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5c6d1c6e30}} (update Breton) === 2020-05-13 === * 22:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a2deb7908c}} (update past participle item ID after merge) === 2020-05-11 === * 19:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ddac27d2e2}} (translation update) === 2020-05-10 === * 22:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b797c90917}} (Breton typofix) * 15:00 wm-bot: <lucaswerkmeister> deployed {{Gerrit|eac96e8493}} (Breton adjectives and other improvements) * 11:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fc78831f8e}} (Breton nouns) === 2020-05-09 === * 19:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b4780fa832}} (drag’n’drop unmatched forms in edit mode) === 2020-04-25 === * 20:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0dadbb4d4e}} (toolforge.org) === 2020-04-21 === * 21:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6634452b4c}} (increase uWSGI buffer) === 2020-04-18 === * 18:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c815a210bd}} (Hebrew nouns) * 17:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|33c3ac264e}} (fix english-adverb edit mode) * 11:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2959ebf637}} (fix duplicates in advanced mode) === 2020-04-14 === * 20:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|44b5df2897}} (edit mode: show lemma, show conflicts, add missing statements) === 2020-04-13 === * 22:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2fe2118d4e}} (python3.7) * 22:20 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ab7f751ba6}} (edit mode) === 2020-02-26 === * 00:22 wm-bot: <root> Migrated to 2020 Kubernetes cluster === 2020-01-28 === * 00:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|61fe7e59fb}} (typofix) * 00:08 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e0e916e0a5}} (more Persian translations and RTL fixes) === 2020-01-27 === * 23:23 wm-bot: <lucaswerkmeister> deployed {{Gerrit|54b9e37118}} (more RTL fixes) * 23:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|72ec256823}} (Persian nouns and verbs) [actually happened ~30mins ago, forgot to log] === 2020-01-15 === * 00:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bc1d49c202}} (better CSRF error handling, [[phab:T242573|T242573]]) === 2020-01-14 === * 00:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|242c25810b}} (clarify Spanish verbs) === 2020-01-12 === * 14:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|edcbc10ae9}} (Spanish verbs) === 2020-01-11 === * 17:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d9619cb473}} (Danish nouns and verbs) * 14:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|4a20b4b95e}} (Czech perfective verbs) * 14:10 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8da9227b52}} (fix typos in Czech adjective template) === 2019-11-30 === * 13:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2f5a8ccc2e}} (update english-verb) === 2019-11-21 === * 22:31 wm-bot: <lucaswerkmeister> deployed {{Gerrit|13cf2696b9}} (reorder) * 22:27 wm-bot: <lucaswerkmeister> deployed {{Gerrit|89ad1e816c}} (Basque verbs) === 2019-11-11 === * 23:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cd4239904a}} (work around [[phab:T230833|T230833]]) * 21:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8b53b417c1}} (fixes to Kurdish (Kurmancî)) * 17:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fe31bd9aa6}} (message syntax fix) === 2019-11-10 === * 19:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9d736fe2f6}} (Kurdish Kurmancî nouns) * 15:56 wm-bot: <lucaswerkmeister> deployed {{Gerrit|29e549fe31}} (Malayalam nouns) === 2019-10-27 === * 22:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|2fc68fabb5}} (lexeme IDs in bulk mode) === 2019-10-16 === * 22:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b480b6d07e}} (Czech translations + adjectives with more forms) === 2019-10-07 === * 22:49 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ce8ba2b234}} (add plural grammatical feature to Ukrainian plurale tantum forms) === 2019-09-30 === * 22:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|19bf4e3347}} (remove PHP_ENGINE cookie) === 2019-08-28 === * 23:10 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a053e9a36e}} (update Swedish translations) === 2019-08-22 === * 22:53 wm-bot: <lucaswerkmeister> deployed 60cf696645v (minor bulk mode improvements) * 22:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f4fd72ab72}} (bulk mode improvements) === 2019-08-20 === * 20:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|938075faf2}} (bulk mode) === 2019-08-11 === * 11:25 wm-bot: <lucaswerkmeister> deployed {{Gerrit|09a3ac6b64}} (Swedish absolute adjectives) === 2019-08-02 === * 21:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a4d699fbcb}} (fix item ID after merge) === 2019-07-24 === * 12:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f0883f1ebc}} (templates API) === 2019-07-07 === * 18:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|50a70b3590}} (Swedish verbs) * 13:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9a148c8cc5}} (add statements when editing existing lexeme) * 12:38 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a8242673b9}} (use jsonify) * 12:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|994b980655}} (CORS for duplicates API) === 2019-07-06 === * 22:03 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b0f39bb09b}} (API to match lexemes to templates) === 2019-06-26 === * 20:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e74ff290cc}} (duplicates API bug fix) [actually deployed 2 hours ago, forgot to log] === 2019-06-24 === * 22:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e937ff5839}} (autocapitalize="off" on form) * 22:44 wm-bot: <lucaswerkmeister> deployed uncommitted experimental change (autocapitalize="off" on form and inputs) * 22:29 wm-bot: <lucaswerkmeister> deployed uncommitted experimental change (autocapitalize="off" on form rather than inputs) * 22:14 wm-bot: <lucaswerkmeister> deployed uncommitted experimental change (autocapitalize="off" on inputs) * 21:10 wm-bot: <lucaswerkmeister> deployed {{Gerrit|07b05a6858}} (Portuguese verbs) === 2019-06-14 === * 19:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c48127f696}} (update Russian translations) * 00:38 wm-bot: <lucaswerkmeister> kubectl delete deployment lexeme-forms.purge-all-lexemes # [[phab:T225510|T225510]] done === 2019-06-12 === * 08:48 wm-bot: <lucaswerkmeister> kubectl create -f deployment-purge-all-lexemes.yaml # [[phab:T225510|T225510]] === 2019-06-10 === * 19:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|645886b3a8}} (update German translations) * 18:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|846100f8d9}} (update Czech translations) * 12:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|fe6cc3a79b}} (improved forms/senses message for duplicates) === 2019-06-09 === * 23:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|5c88de6348}} (number of forms/senses for duplicates) === 2019-06-08 === * 14:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f09dfd20a1}} (Dutch nouns) * 14:00 wm-bot: <lucaswerkmeister> git remote add github https://github.com/lucaswerkmeister/tool-lexeme-forms.git # work around [[phab:T224677|T224677]] * 12:17 wm-bot: <lucaswerkmeister> restarted webservice after redirect loop === 2019-05-20 === * 09:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|496a928b67}} (switch to Python 3.5), including venv rebuild * 08:52 wm-bot: <lucaswerkmeister> stopping webserver for Python 3.5 upgrade <noinclude>[[Category:SAL]]</noinclude> rrn6qhbrmpca59258dnwt5k2vxvtka5 0 449160 2426643 2426603 2026-06-14T00:02:08Z Dexbot 30554 Bot: Updating the report 2426643 wikitext text/x-wiki {{/Header}} == Today (2026-06-14) == == Yesterday (2026-06-13) == == Last seven days == {| class="wikitable" |+ eqiad |- ! Section !! Work |- | es3 || [[phab:T428050|Migrate es3 section to Debian Trixie (T428050)]] (marostegui) |- | es4 || [[phab:T428386|Migrate es4 section to Debian Trixie (T428386)]] (marostegui) |- | s1 || [[phab:T426083|Switchover s1 master (db1163 -&gt; db1184) (T426083)]] (fceratto) |- | s4 || * [[phab:T426086|Switchover s4 master (db1160 -&gt; db1244) (T426086)]] (fceratto) * [[phab:T428386|Migrate es4 section to Debian Trixie (T428386)]] (marostegui) |- | x1 || [[phab:T428158|Switchover x1 master (db1237 -&gt; db1220) (T428158)]] (marostegui) |- |} {| class="wikitable" |+ codfw |- ! Section !! Work |- | es4 || [[phab:T428386|Migrate es4 section to Debian Trixie (T428386)]] (marostegui) |- |} [[Category:MariaDB]] r8wuevcmi2jiht7i7l3947xhtcavbn9 498 452890 2426625 2426621 2026-06-13T13:03:18Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27467463970 (https://github.com/cluebotng/component-configs/commits/3dc535380a54d2290621b9d585a5018fdc4669a2) 2426625 wikitext text/x-wiki === 2026-06-13 === * 13:03 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27467463970 (https://github.com/cluebotng/component-configs/commits/3dc535380a54d2290621b9d585a5018fdc4669a2) * 11:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27464936848 (https://github.com/cluebotng/component-configs/commits/d19537391f153a63eba67672cf3aecb76bff362e) === 2026-06-11 === * 13:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27350281127 (https://github.com/cluebotng/component-configs/commits/41e666d399f22121043fa8132a3e8bf6368e5181) === 2026-06-10 === * 15:01 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27285125877 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b) * 12:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277789050 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884) * 12:40 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27276698071 (https://github.com/cluebotng/component-configs/commits/8be9293c4b541d74d39482efd21163eb36cda6bd) * 12:37 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27276511398 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1) * 12:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27275411814 (https://github.com/cluebotng/component-configs/commits/869dfb8d1487914e36184a9d1c5aae1e26dbba01) * 12:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27275339348 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11) === 2026-06-09 === * 17:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27225356873 (https://github.com/cluebotng/component-configs/commits/9534cf81437fb2c268eb00e4145978dddbf6322e) === 2026-06-08 === * 12:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27138156886 (https://github.com/cluebotng/component-configs/commits/4677023bc60821948b76a89b2968d9fa3db267d4) === 2026-06-05 === * 14:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27021601200 (https://github.com/cluebotng/component-configs/commits/d4efd5a504c17f41f2d280dabcb635f9c4f07000) * 14:04 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27019408986 (https://github.com/cluebotng/component-configs/commits/4de86ac4b524f394ec49dc167d0c75457e981af1) * 12:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27016160066 (https://github.com/cluebotng/component-configs/commits/3c617707fe44f00c5ac81aa315bfc989a7bd4d00) * 11:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27012814543 (https://github.com/cluebotng/component-configs/commits/48e7143fc26338766785004996ec66dea5876b0e) * 11:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27011683531 (https://github.com/cluebotng/component-configs/commits/7b3ba4bae58717e47510c281fdb60f00be6f7c92) === 2026-06-01 === * 15:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26762978339 (https://github.com/cluebotng/component-configs/commits/9a088c9b8375555c696948825fff7700458b4254) * 13:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26757971428 (https://github.com/cluebotng/component-configs/commits/4790ebea51ebfbd67e51894987e6273e5940cbf1) === 2026-05-31 === * 17:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864796 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3) * 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719862181 (https://github.com/cluebotng/component-configs/commits/8e921d9dd24ae32755f893363b9dfa897cf71c25) * 17:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719681637 (https://github.com/cluebotng/component-configs/commits/02759bc368e1f9901ca399942295a807038321cb) * 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719575507 (https://github.com/cluebotng/component-configs/commits/9bb586c7c04b2a5848b2ebc287497a81506f2d1d) === 2026-05-30 === * 18:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26691257350 (https://github.com/cluebotng/component-configs/commits/1daccd14ff5fe952e32175ee2cf249f2312d99ae) === 2026-05-29 === * 00:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26609639469 (https://github.com/cluebotng/component-configs/commits/8c2fccaaae357774084389157d9a305e72eccb20) === 2026-05-28 === * 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26592792853 (https://github.com/cluebotng/component-configs/commits/a7971b7e286e177862e5318c40b0d4d868efc7c8) === 2026-05-21 === * 20:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26251627549 (https://github.com/cluebotng/component-configs/commits/96f9184e66a6e4b35a49f02940a213125945b056) === 2026-05-19 === * 00:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26068076396 (https://github.com/cluebotng/component-configs/commits/f7db7f6fff0d4d6dd451b5f92e75ba755a74129c) === 2026-05-17 === * 08:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25986406220 (https://github.com/cluebotng/component-configs/commits/be3bb145d2803394cd0b7dbd8ae1775ac9b7cd09) === 2026-05-14 === * 18:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25878752531 (https://github.com/cluebotng/component-configs/commits/21e928fa1870ddaf5fae15afc6f92aa3cb3fb970) === 2026-05-13 === * 02:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773616037 (https://github.com/cluebotng/component-configs/commits/0fd601991775a24b437113d09438e74b996c991b) === 2026-05-12 === * 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25727821525 (https://github.com/cluebotng/component-configs/commits/91aefb7d53013ad152bb721f71980dd26170f297) * 09:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842068 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c) * 09:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724562210 (https://github.com/cluebotng/component-configs/commits/bd0e188642746ab949ec3762676ac730afff1c17) * 08:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25723480598 (https://github.com/cluebotng/component-configs/commits/25c0a1035daa67c2225c0f7f7a414ff5cfb6ed2a) * 08:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723280529 (https://github.com/cluebotng/component-configs/commits/51d7c1919958a7672895885cbb3a1061934d2788) === 2026-05-06 === * 18:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25453754970 (https://github.com/cluebotng/component-configs/commits/92f164d1ab158aea1f76cd0a787f33ffe4017e85) === 2026-05-02 === * 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25252392660 (https://github.com/cluebotng/component-configs/commits/7352cd4f730ca9f5c276772f0b338230989feef4) === 2026-04-24 === * 22:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914278528 (https://github.com/cluebotng/component-configs/commits/23a4b53f3d291b0c750d44a2c0a661333307786d) === 2026-04-20 === * 23:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24695320410 (https://github.com/cluebotng/component-configs/commits/279edf060f43353ea66e6d057773bfdb883b16a1) === 2026-04-17 === * 00:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24540678937 (https://github.com/cluebotng/component-configs/commits/26849735bbefbe218cbe0ce41db5a35941798c7b) === 2026-04-14 === * 21:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24422836848 (https://github.com/cluebotng/component-configs/commits/10f4f0f81e169fac55d056176a273966c8160078) === 2026-04-11 === * 10:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24280508671 (https://github.com/cluebotng/component-configs/commits/5953d3fb9c5e414df6995740382b7bd3be49ced2) * 10:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24280372486 (https://github.com/cluebotng/component-configs/commits/5b34645dff3f37bc9f974635e03cd6b8436f37d1) === 2026-04-10 === * 23:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24268430819 (https://github.com/cluebotng/component-configs/commits/3652893dce02243971055a6ab740363f103ce104) * 23:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24268031378 (https://github.com/cluebotng/component-configs/commits/31367659ada078f50022f1df4b16b6139db27c09) * 22:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24267603054 (https://github.com/cluebotng/component-configs/commits/ba252b54cec9387b47dd4ac4a347d4a9c5118c3e) * 16:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24251572443 (https://github.com/cluebotng/component-configs/commits/2426c8db99c6d44c954ced07c9f41fcaa9e8e549) * 15:53 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24251572443 (https://github.com/cluebotng/component-configs/commits/2426c8db99c6d44c954ced07c9f41fcaa9e8e549) * 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442439 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f) * 15:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249898021 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3) * 14:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247620609 (https://github.com/cluebotng/component-configs/commits/e63a941f5b83d97a9751af731c869062ceef4519) * 14:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247581365 (https://github.com/cluebotng/component-configs/commits/49becfde53d5f960c8e4df0484cebb2bb4d4c5aa) * 13:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24246395413 (https://github.com/cluebotng/component-configs/commits/945fa198e64a0e63b777bb570d57d68ef0ce3f69) * 13:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245816710 (https://github.com/cluebotng/component-configs/commits/6181fdda40150d3535541f3084ac7ff245f19536) * 13:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245353001 (https://github.com/cluebotng/component-configs/commits/97eebf1bcdf5be901e0d3fd82c1b3ea6a8668163) * 13:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245099622 (https://github.com/cluebotng/component-configs/commits/251c10040c01caf2ba9b855050c318d5d2fd8e81) * 13:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24244959273 (https://github.com/cluebotng/component-configs/commits/2a6605ee2d07c0ff0d690aaa8aabed0ca35bab72) * 04:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24226442413 (https://github.com/cluebotng/component-configs/commits/4f895f83dae3f356cae2a1bbcfea51dd9d18bd15) * 01:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24221486227 (https://github.com/cluebotng/component-configs/commits/d96804861818d7786153d18d47be075a4dbbb6f2) === 2026-04-09 === * 20:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24212235180 (https://github.com/cluebotng/component-configs/commits/1cd21afab7312bd0122c0e735f8f4dca03019011) * 19:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24208407251 (https://github.com/cluebotng/component-configs/commits/6cd680dd209bf7fbb01cf24cb6cca82f0fab716d) * 18:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206712296 (https://github.com/cluebotng/component-configs/commits/e7d5ec988541b9d441a5c565f624b7e88e11204f) * 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206093220 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253) * 14:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24195131449 (https://github.com/cluebotng/component-configs/commits/6b512f6db7cc4e49078b135e437185906821ae81) === 2026-04-08 === * 05:03 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24118645397 (https://github.com/cluebotng/component-configs/commits/908dd70b5972cca0c0dafbe50a0020547b833a4e) === 2026-04-07 === * 22:08 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24106708812 (https://github.com/cluebotng/component-configs/commits/b85f56b6997ccf41cc8ea32f33a61809b68b9bc5) === 2026-04-02 === * 22:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23924337776 (https://github.com/cluebotng/component-configs/commits/266152f9f673810b0c9460b5828cb86e7aee31d9) === 2026-03-31 === * 06:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23782957628 (https://github.com/cluebotng/component-configs/commits/7888bbd75773dc064d78ad2ee8949f1540eab0fd) * 01:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23775778726 (https://github.com/cluebotng/component-configs/commits/47f8e20c39e29b952f6dbbd04917970802ce1a0b) === 2026-03-27 === * 17:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656811 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4) * 17:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537327 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b) === 2026-03-23 === * 10:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23433070550 (https://github.com/cluebotng/component-configs/commits/dd92649311ee430b4225d5c6db5d6e6b16d10a86) * 10:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23433054689 (https://github.com/cluebotng/component-configs/commits/c895b3d11b8546e54e4cca5ba350c0a5ca9c5917) === 2026-03-21 === * 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383565693 (https://github.com/cluebotng/component-configs/commits/48390b500ab2b65905e09987c12a3e42c3f69778) * 16:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640335 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa) * 16:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383565693 (https://github.com/cluebotng/component-configs/commits/48390b500ab2b65905e09987c12a3e42c3f69778) * 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383551619 (https://github.com/cluebotng/component-configs/commits/ffff74b90a37a0c6bdd565128d3c11ae195e0763) === 2026-03-20 === * 04:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23329334324 (https://github.com/cluebotng/component-configs/commits/bd7700c30291bfab3a656aa8f257292e287a71ca) === 2026-03-19 === * 09:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23287809902 (https://github.com/cluebotng/component-configs/commits/0976850451c9fbb8c4afb773cc70b91cd7c6fdeb) === 2026-03-17 === * 21:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23217273642 (https://github.com/cluebotng/component-configs/commits/4fe6cb3d8ad39b60746b3b6bd2f83c4d05a82d6b) === 2026-03-13 === * 01:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23031229526 (https://github.com/cluebotng/component-configs/commits/ebd67e60183f161276bf0e13daab55ceb2463eb2) === 2026-03-10 === * 00:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22881598726 (https://github.com/cluebotng/component-configs/commits/bc32d8044077ff83db8b985b87df029ff564ad29) === 2026-03-07 === * 00:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22788149115 (https://github.com/cluebotng/component-configs/commits/b3731fab9a7f4f225ecbe318fa80808de6c904b0) === 2026-03-06 === * 09:08 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22756671151 (https://github.com/cluebotng/component-configs/commits/397fc33968a3c4795b97b1791a0b991ebeb81430) === 2026-03-04 === * 09:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22662750872 (https://github.com/cluebotng/component-configs/commits/01746ef8804c30c85963ea888a75887ebe879e3b) * 01:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22650516003 (https://github.com/cluebotng/component-configs/commits/e7a1e2e06f2ccf038c06cb203369f336c298cf6c) === 2026-03-03 === * 21:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22642685900 (https://github.com/cluebotng/component-configs/commits/3cbfb68b3c0e7d97130ede1be762389f300234d2) === 2026-03-02 === * 01:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22557173037 (https://github.com/cluebotng/component-configs/commits/a414cf552e0a0c0d2c9e9817f922d56a4c899bf6) === 2026-02-27 === * 01:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22468544596 (https://github.com/cluebotng/component-configs/commits/b961f37db0544196a7206882b8e3f2292b7e0894) === 2026-02-25 === * 21:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22415923749 (https://github.com/cluebotng/component-configs/commits/a2a4f5ecffad1b49c96c33b5045430a5b75f71bc) * 11:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22395760068 (https://github.com/cluebotng/component-configs/commits/c6093c4ed72aba8fa453b2f67e48d1effeaabb4b) === 2026-02-24 === * 21:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22370719324 (https://github.com/cluebotng/component-configs/commits/7170b95a5f9b6be3c928684f1e9c436deb3ddd1f) * 00:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22331436693 (https://github.com/cluebotng/component-configs/commits/372e84511fdcb0893755ac22f399d3f24f438f7b) === 2026-02-21 === * 05:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22251332525 (https://github.com/cluebotng/component-configs/commits/29401fbb166f71e375eca7254fe841cb01836d2f) === 2026-02-20 === * 13:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22226385604 (https://github.com/cluebotng/component-configs/commits/fbd7d861a7062a2c09fd2117cbf569beb53916f4) * 08:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22216991615 (https://github.com/cluebotng/component-configs/commits/64d521535aa35454c28900f70009efc0e9ff4a10) * 05:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22212024170 (https://github.com/cluebotng/component-configs/commits/9b0508c1c5a875dd795c865e67f2a93d4f247597) === 2026-02-19 === * 13:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22183076688 (https://github.com/cluebotng/component-configs/commits/919ebb8860a93b9d071da361cad56448a3b1f2b4) * 00:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22163899784 (https://github.com/cluebotng/component-configs/commits/0ce51e9bda73cc3ee0df647f7ba8dcfd02eb97e6) === 2026-02-16 === * 02:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22047780823 (https://github.com/cluebotng/component-configs/commits/3a1f6b151d38aab4ce1a62509b108ac9afc5230b) === 2026-02-15 === * 08:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22032600290 (https://github.com/cluebotng/component-configs/commits/80b9cda20d3f21e2f901db6ccbd168bfffb6b063) === 2026-02-14 === * 20:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22023900270 (https://github.com/cluebotng/component-configs/commits/e8878c3f7a08aa1712126c1b6490f6db41621f44) * 20:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22023724371 (https://github.com/cluebotng/component-configs/commits/de3779f7adea66769077e2380d7b0ce25f3d9e82) * 20:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22023714306 (https://github.com/cluebotng/component-configs/commits/7cd082664340738b5c6cc46d0a195f3814672a3a) * 19:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c) * 19:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c) * 19:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c) * 19:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022827162 (https://github.com/cluebotng/component-configs/commits/0e1693c2b662aaa0c9264ceef355bcbfbc162ea7) * 19:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022587196 (https://github.com/cluebotng/component-configs/commits/57dcc675b3ed54fc17f697d6c7b9554b5d06aab0) * 18:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022444685 (https://github.com/cluebotng/component-configs/commits/1e0b17c59284d25ea8ac39a455abb9921ee6608a) === 2025-11-26 === * 19:58 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19715882591 (https://github.com/cluebotng/component-configs/commits/18c2bc79b5f0023e682a9245197cf87c5cc76943) === 2025-11-11 === * 15:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642915 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c) * 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270263370 (https://github.com/cluebotng/component-configs/commits/d1674e8f4f6cec3b48e848137ce42585278d4a67) === 2025-11-09 === * 22:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19215285730 (https://github.com/cluebotng/component-configs/commits/bf77359fc102b05a026ea8b66dc01ff16a936804) * 22:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19215085734 (https://github.com/cluebotng/component-configs/commits/6d8f2491239fbe29d19544922253d9930a88e7a0) * 20:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213996489 (https://github.com/cluebotng/component-configs/commits/38bc77281c9dbd1100915d95ba68705d8a7392a7) * 20:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213940731 (https://github.com/cluebotng/component-configs/commits/c01b89b7b0455d4f1cc63a2eb002f9c55c0a663f) === 2025-11-05 === * 19:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19114484110 (https://github.com/cluebotng/component-configs/commits/fae01bfaeaeca0cf7676ece10cbd39948560086f) * 16:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19108874377 (https://github.com/cluebotng/component-configs/commits/3f51ec3aa53d1378883a9dc973716e57c283d26c) === 2025-10-29 === * 15:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18912872633 (https://github.com/cluebotng/component-configs/commits/3281794d8d1d2e17d9e9859c6f6f7ae3c5216eda) === 2025-10-23 === * 12:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18748267282 (https://github.com/cluebotng/component-configs/commits/bc8f1b883d0d53edf08bea5e5319ee7ee0b4fb82) === 2025-10-07 === * 06:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18304366702 (https://github.com/cluebotng/component-configs/commits/5b83bca0e9293029698d7f3a1b2764727ae7f971) === 2025-10-06 === * 06:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18272390911 (https://github.com/cluebotng/component-configs/commits/49abbdd5dd7066314199c213043305ceed2b54f7) === 2025-10-05 === * 06:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18255184611 (https://github.com/cluebotng/component-configs/commits/7fe1a04069d9d0b4b11019443c85885c202852d4) === 2025-10-03 === * 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18215013747 (https://github.com/cluebotng/component-configs/commits/7ab2bbe022e2513dc81a13a7055c4c7736e5f876) === 2025-09-29 === * 16:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101417 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5) * 15:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18102721922 (https://github.com/cluebotng/component-configs/commits/87ddcf2fce928fde2ba91ecdba3561b12b8de1d2) * 14:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18099932067 (https://github.com/cluebotng/component-configs/commits/0de901e1203dd61656503ef2127efe360e9ed6cc) * 09:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18091994310 (https://github.com/cluebotng/component-configs/commits/ff3951fa5af87196929a9a864f8189b7a7436ac8) * 09:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18091898160 (https://github.com/cluebotng/component-configs/commits/ff3951fa5af87196929a9a864f8189b7a7436ac8) === 2025-09-27 === * 13:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060157048 (https://github.com/cluebotng/component-configs/commits/3aa079ed0cb7aa29f9ece46a47ad96203e53f242) === 2025-09-26 === * 22:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18050560602 (https://github.com/cluebotng/component-configs/commits/886ded0824a9ce7b27c852949f3530bda15bef14) * 11:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18036958606 (https://github.com/cluebotng/component-configs/commits/a51fe109bfad3e2df5aa8e89b837a951bf8ad2cf) * 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18030114498 (https://github.com/cluebotng/component-configs/commits/ea47ef95beb4cf8a1b7d439a83af7b2d4cf168ce) === 2025-09-25 === * 17:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18016018062 (https://github.com/cluebotng/component-configs/commits/150020d96f0c95173ba88c382221223a0c1f7a8d) * 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998409 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c) * 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801858 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c) * 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014528718 (https://github.com/cluebotng/component-configs/commits/96654b441f84901e1a607ced407eb9babb8fdbfc) * 16:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014528718 (https://github.com/cluebotng/component-configs/commits/96654b441f84901e1a607ced407eb9babb8fdbfc) * 16:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221959 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d) * 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782292 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e) * 16:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677351 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a) * 16:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531088 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f) * 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18012641376 (https://github.com/cluebotng/component-configs/commits/87c176492b1f1fb18570dbb70687258843c5773c) * 14:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18008240418 (https://github.com/cluebotng/component-configs/commits/9949a4a5acff374c1edd7b6e21959a28721e02d0) === 2025-09-24 === * 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198575 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4) * 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820837 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960) * 17:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009157 (https://github.com/cluebotng/component-configs/commits/refs/heads/main) * 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983740752 (https://github.com/cluebotng/component-configs/commits/refs/heads/main) * 12:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17977211158 (https://github.com/cluebotng/component-configs/commits/refs/heads/main) * 12:11 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17976182663 * 12:06 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17976062312 * 06:46 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17968643293 === 2025-09-22 === * 20:43 wmbot~component-configs@tools-bastion: Test migrating log to feed channel * 20:43 wmbot~damian-scripts@tools-bastion-15: Test migrating log to feed channel * 19:12 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925793114 === 2025-08-24 === * 18:02 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.3.0 === 2025-08-23 === * 19:44 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.5 * 18:37 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.4 * 18:10 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.3 === 2025-08-14 === * 13:39 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.2 * 13:27 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.1 * 13:20 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.0 === 2025-08-13 === * 20:07 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.10 * 18:53 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.9 === 2025-08-11 === * 19:49 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.7 === 2025-08-10 === * 18:19 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.6 * 15:06 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.5 === 2025-08-07 === * 15:38 wmbot~damian@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.2 === 2023-06-15 === * 18:43 wm-bot: <root> webservice restart, checked pods <noinclude>[[Category:SAL]]</noinclude> 4a3u9qvbco2du67vq7bic2occs2su34 2426628 2426625 2026-06-13T16:52:57Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27472937497 (https://github.com/cluebotng/component-configs/commits/38317c72706d30dee2f423e689b46826fa0cbb9a) 2426628 wikitext text/x-wiki === 2026-06-13 === * 16:52 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27472937497 (https://github.com/cluebotng/component-configs/commits/38317c72706d30dee2f423e689b46826fa0cbb9a) * 13:03 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27467463970 (https://github.com/cluebotng/component-configs/commits/3dc535380a54d2290621b9d585a5018fdc4669a2) * 11:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27464936848 (https://github.com/cluebotng/component-configs/commits/d19537391f153a63eba67672cf3aecb76bff362e) === 2026-06-11 === * 13:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27350281127 (https://github.com/cluebotng/component-configs/commits/41e666d399f22121043fa8132a3e8bf6368e5181) === 2026-06-10 === * 15:01 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27285125877 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b) * 12:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277789050 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884) * 12:40 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27276698071 (https://github.com/cluebotng/component-configs/commits/8be9293c4b541d74d39482efd21163eb36cda6bd) * 12:37 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27276511398 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1) * 12:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27275411814 (https://github.com/cluebotng/component-configs/commits/869dfb8d1487914e36184a9d1c5aae1e26dbba01) * 12:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27275339348 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11) === 2026-06-09 === * 17:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27225356873 (https://github.com/cluebotng/component-configs/commits/9534cf81437fb2c268eb00e4145978dddbf6322e) === 2026-06-08 === * 12:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27138156886 (https://github.com/cluebotng/component-configs/commits/4677023bc60821948b76a89b2968d9fa3db267d4) === 2026-06-05 === * 14:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27021601200 (https://github.com/cluebotng/component-configs/commits/d4efd5a504c17f41f2d280dabcb635f9c4f07000) * 14:04 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27019408986 (https://github.com/cluebotng/component-configs/commits/4de86ac4b524f394ec49dc167d0c75457e981af1) * 12:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27016160066 (https://github.com/cluebotng/component-configs/commits/3c617707fe44f00c5ac81aa315bfc989a7bd4d00) * 11:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27012814543 (https://github.com/cluebotng/component-configs/commits/48e7143fc26338766785004996ec66dea5876b0e) * 11:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27011683531 (https://github.com/cluebotng/component-configs/commits/7b3ba4bae58717e47510c281fdb60f00be6f7c92) === 2026-06-01 === * 15:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26762978339 (https://github.com/cluebotng/component-configs/commits/9a088c9b8375555c696948825fff7700458b4254) * 13:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26757971428 (https://github.com/cluebotng/component-configs/commits/4790ebea51ebfbd67e51894987e6273e5940cbf1) === 2026-05-31 === * 17:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864796 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3) * 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719862181 (https://github.com/cluebotng/component-configs/commits/8e921d9dd24ae32755f893363b9dfa897cf71c25) * 17:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719681637 (https://github.com/cluebotng/component-configs/commits/02759bc368e1f9901ca399942295a807038321cb) * 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719575507 (https://github.com/cluebotng/component-configs/commits/9bb586c7c04b2a5848b2ebc287497a81506f2d1d) === 2026-05-30 === * 18:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26691257350 (https://github.com/cluebotng/component-configs/commits/1daccd14ff5fe952e32175ee2cf249f2312d99ae) === 2026-05-29 === * 00:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26609639469 (https://github.com/cluebotng/component-configs/commits/8c2fccaaae357774084389157d9a305e72eccb20) === 2026-05-28 === * 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26592792853 (https://github.com/cluebotng/component-configs/commits/a7971b7e286e177862e5318c40b0d4d868efc7c8) === 2026-05-21 === * 20:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26251627549 (https://github.com/cluebotng/component-configs/commits/96f9184e66a6e4b35a49f02940a213125945b056) === 2026-05-19 === * 00:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26068076396 (https://github.com/cluebotng/component-configs/commits/f7db7f6fff0d4d6dd451b5f92e75ba755a74129c) === 2026-05-17 === * 08:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25986406220 (https://github.com/cluebotng/component-configs/commits/be3bb145d2803394cd0b7dbd8ae1775ac9b7cd09) === 2026-05-14 === * 18:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25878752531 (https://github.com/cluebotng/component-configs/commits/21e928fa1870ddaf5fae15afc6f92aa3cb3fb970) === 2026-05-13 === * 02:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773616037 (https://github.com/cluebotng/component-configs/commits/0fd601991775a24b437113d09438e74b996c991b) === 2026-05-12 === * 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25727821525 (https://github.com/cluebotng/component-configs/commits/91aefb7d53013ad152bb721f71980dd26170f297) * 09:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842068 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c) * 09:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724562210 (https://github.com/cluebotng/component-configs/commits/bd0e188642746ab949ec3762676ac730afff1c17) * 08:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25723480598 (https://github.com/cluebotng/component-configs/commits/25c0a1035daa67c2225c0f7f7a414ff5cfb6ed2a) * 08:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723280529 (https://github.com/cluebotng/component-configs/commits/51d7c1919958a7672895885cbb3a1061934d2788) === 2026-05-06 === * 18:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25453754970 (https://github.com/cluebotng/component-configs/commits/92f164d1ab158aea1f76cd0a787f33ffe4017e85) === 2026-05-02 === * 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25252392660 (https://github.com/cluebotng/component-configs/commits/7352cd4f730ca9f5c276772f0b338230989feef4) === 2026-04-24 === * 22:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914278528 (https://github.com/cluebotng/component-configs/commits/23a4b53f3d291b0c750d44a2c0a661333307786d) === 2026-04-20 === * 23:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24695320410 (https://github.com/cluebotng/component-configs/commits/279edf060f43353ea66e6d057773bfdb883b16a1) === 2026-04-17 === * 00:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24540678937 (https://github.com/cluebotng/component-configs/commits/26849735bbefbe218cbe0ce41db5a35941798c7b) === 2026-04-14 === * 21:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24422836848 (https://github.com/cluebotng/component-configs/commits/10f4f0f81e169fac55d056176a273966c8160078) === 2026-04-11 === * 10:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24280508671 (https://github.com/cluebotng/component-configs/commits/5953d3fb9c5e414df6995740382b7bd3be49ced2) * 10:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24280372486 (https://github.com/cluebotng/component-configs/commits/5b34645dff3f37bc9f974635e03cd6b8436f37d1) === 2026-04-10 === * 23:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24268430819 (https://github.com/cluebotng/component-configs/commits/3652893dce02243971055a6ab740363f103ce104) * 23:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24268031378 (https://github.com/cluebotng/component-configs/commits/31367659ada078f50022f1df4b16b6139db27c09) * 22:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24267603054 (https://github.com/cluebotng/component-configs/commits/ba252b54cec9387b47dd4ac4a347d4a9c5118c3e) * 16:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24251572443 (https://github.com/cluebotng/component-configs/commits/2426c8db99c6d44c954ced07c9f41fcaa9e8e549) * 15:53 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24251572443 (https://github.com/cluebotng/component-configs/commits/2426c8db99c6d44c954ced07c9f41fcaa9e8e549) * 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442439 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f) * 15:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249898021 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3) * 14:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247620609 (https://github.com/cluebotng/component-configs/commits/e63a941f5b83d97a9751af731c869062ceef4519) * 14:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247581365 (https://github.com/cluebotng/component-configs/commits/49becfde53d5f960c8e4df0484cebb2bb4d4c5aa) * 13:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24246395413 (https://github.com/cluebotng/component-configs/commits/945fa198e64a0e63b777bb570d57d68ef0ce3f69) * 13:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245816710 (https://github.com/cluebotng/component-configs/commits/6181fdda40150d3535541f3084ac7ff245f19536) * 13:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245353001 (https://github.com/cluebotng/component-configs/commits/97eebf1bcdf5be901e0d3fd82c1b3ea6a8668163) * 13:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245099622 (https://github.com/cluebotng/component-configs/commits/251c10040c01caf2ba9b855050c318d5d2fd8e81) * 13:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24244959273 (https://github.com/cluebotng/component-configs/commits/2a6605ee2d07c0ff0d690aaa8aabed0ca35bab72) * 04:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24226442413 (https://github.com/cluebotng/component-configs/commits/4f895f83dae3f356cae2a1bbcfea51dd9d18bd15) * 01:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24221486227 (https://github.com/cluebotng/component-configs/commits/d96804861818d7786153d18d47be075a4dbbb6f2) === 2026-04-09 === * 20:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24212235180 (https://github.com/cluebotng/component-configs/commits/1cd21afab7312bd0122c0e735f8f4dca03019011) * 19:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24208407251 (https://github.com/cluebotng/component-configs/commits/6cd680dd209bf7fbb01cf24cb6cca82f0fab716d) * 18:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206712296 (https://github.com/cluebotng/component-configs/commits/e7d5ec988541b9d441a5c565f624b7e88e11204f) * 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206093220 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253) * 14:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24195131449 (https://github.com/cluebotng/component-configs/commits/6b512f6db7cc4e49078b135e437185906821ae81) === 2026-04-08 === * 05:03 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24118645397 (https://github.com/cluebotng/component-configs/commits/908dd70b5972cca0c0dafbe50a0020547b833a4e) === 2026-04-07 === * 22:08 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24106708812 (https://github.com/cluebotng/component-configs/commits/b85f56b6997ccf41cc8ea32f33a61809b68b9bc5) === 2026-04-02 === * 22:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23924337776 (https://github.com/cluebotng/component-configs/commits/266152f9f673810b0c9460b5828cb86e7aee31d9) === 2026-03-31 === * 06:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23782957628 (https://github.com/cluebotng/component-configs/commits/7888bbd75773dc064d78ad2ee8949f1540eab0fd) * 01:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23775778726 (https://github.com/cluebotng/component-configs/commits/47f8e20c39e29b952f6dbbd04917970802ce1a0b) === 2026-03-27 === * 17:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656811 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4) * 17:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537327 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b) === 2026-03-23 === * 10:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23433070550 (https://github.com/cluebotng/component-configs/commits/dd92649311ee430b4225d5c6db5d6e6b16d10a86) * 10:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23433054689 (https://github.com/cluebotng/component-configs/commits/c895b3d11b8546e54e4cca5ba350c0a5ca9c5917) === 2026-03-21 === * 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383565693 (https://github.com/cluebotng/component-configs/commits/48390b500ab2b65905e09987c12a3e42c3f69778) * 16:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640335 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa) * 16:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383565693 (https://github.com/cluebotng/component-configs/commits/48390b500ab2b65905e09987c12a3e42c3f69778) * 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383551619 (https://github.com/cluebotng/component-configs/commits/ffff74b90a37a0c6bdd565128d3c11ae195e0763) === 2026-03-20 === * 04:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23329334324 (https://github.com/cluebotng/component-configs/commits/bd7700c30291bfab3a656aa8f257292e287a71ca) === 2026-03-19 === * 09:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23287809902 (https://github.com/cluebotng/component-configs/commits/0976850451c9fbb8c4afb773cc70b91cd7c6fdeb) === 2026-03-17 === * 21:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23217273642 (https://github.com/cluebotng/component-configs/commits/4fe6cb3d8ad39b60746b3b6bd2f83c4d05a82d6b) === 2026-03-13 === * 01:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23031229526 (https://github.com/cluebotng/component-configs/commits/ebd67e60183f161276bf0e13daab55ceb2463eb2) === 2026-03-10 === * 00:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22881598726 (https://github.com/cluebotng/component-configs/commits/bc32d8044077ff83db8b985b87df029ff564ad29) === 2026-03-07 === * 00:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22788149115 (https://github.com/cluebotng/component-configs/commits/b3731fab9a7f4f225ecbe318fa80808de6c904b0) === 2026-03-06 === * 09:08 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22756671151 (https://github.com/cluebotng/component-configs/commits/397fc33968a3c4795b97b1791a0b991ebeb81430) === 2026-03-04 === * 09:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22662750872 (https://github.com/cluebotng/component-configs/commits/01746ef8804c30c85963ea888a75887ebe879e3b) * 01:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22650516003 (https://github.com/cluebotng/component-configs/commits/e7a1e2e06f2ccf038c06cb203369f336c298cf6c) === 2026-03-03 === * 21:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22642685900 (https://github.com/cluebotng/component-configs/commits/3cbfb68b3c0e7d97130ede1be762389f300234d2) === 2026-03-02 === * 01:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22557173037 (https://github.com/cluebotng/component-configs/commits/a414cf552e0a0c0d2c9e9817f922d56a4c899bf6) === 2026-02-27 === * 01:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22468544596 (https://github.com/cluebotng/component-configs/commits/b961f37db0544196a7206882b8e3f2292b7e0894) === 2026-02-25 === * 21:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22415923749 (https://github.com/cluebotng/component-configs/commits/a2a4f5ecffad1b49c96c33b5045430a5b75f71bc) * 11:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22395760068 (https://github.com/cluebotng/component-configs/commits/c6093c4ed72aba8fa453b2f67e48d1effeaabb4b) === 2026-02-24 === * 21:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22370719324 (https://github.com/cluebotng/component-configs/commits/7170b95a5f9b6be3c928684f1e9c436deb3ddd1f) * 00:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22331436693 (https://github.com/cluebotng/component-configs/commits/372e84511fdcb0893755ac22f399d3f24f438f7b) === 2026-02-21 === * 05:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22251332525 (https://github.com/cluebotng/component-configs/commits/29401fbb166f71e375eca7254fe841cb01836d2f) === 2026-02-20 === * 13:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22226385604 (https://github.com/cluebotng/component-configs/commits/fbd7d861a7062a2c09fd2117cbf569beb53916f4) * 08:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22216991615 (https://github.com/cluebotng/component-configs/commits/64d521535aa35454c28900f70009efc0e9ff4a10) * 05:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22212024170 (https://github.com/cluebotng/component-configs/commits/9b0508c1c5a875dd795c865e67f2a93d4f247597) === 2026-02-19 === * 13:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22183076688 (https://github.com/cluebotng/component-configs/commits/919ebb8860a93b9d071da361cad56448a3b1f2b4) * 00:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22163899784 (https://github.com/cluebotng/component-configs/commits/0ce51e9bda73cc3ee0df647f7ba8dcfd02eb97e6) === 2026-02-16 === * 02:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22047780823 (https://github.com/cluebotng/component-configs/commits/3a1f6b151d38aab4ce1a62509b108ac9afc5230b) === 2026-02-15 === * 08:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22032600290 (https://github.com/cluebotng/component-configs/commits/80b9cda20d3f21e2f901db6ccbd168bfffb6b063) === 2026-02-14 === * 20:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22023900270 (https://github.com/cluebotng/component-configs/commits/e8878c3f7a08aa1712126c1b6490f6db41621f44) * 20:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22023724371 (https://github.com/cluebotng/component-configs/commits/de3779f7adea66769077e2380d7b0ce25f3d9e82) * 20:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22023714306 (https://github.com/cluebotng/component-configs/commits/7cd082664340738b5c6cc46d0a195f3814672a3a) * 19:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c) * 19:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c) * 19:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c) * 19:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022827162 (https://github.com/cluebotng/component-configs/commits/0e1693c2b662aaa0c9264ceef355bcbfbc162ea7) * 19:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022587196 (https://github.com/cluebotng/component-configs/commits/57dcc675b3ed54fc17f697d6c7b9554b5d06aab0) * 18:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022444685 (https://github.com/cluebotng/component-configs/commits/1e0b17c59284d25ea8ac39a455abb9921ee6608a) === 2025-11-26 === * 19:58 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19715882591 (https://github.com/cluebotng/component-configs/commits/18c2bc79b5f0023e682a9245197cf87c5cc76943) === 2025-11-11 === * 15:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642915 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c) * 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270263370 (https://github.com/cluebotng/component-configs/commits/d1674e8f4f6cec3b48e848137ce42585278d4a67) === 2025-11-09 === * 22:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19215285730 (https://github.com/cluebotng/component-configs/commits/bf77359fc102b05a026ea8b66dc01ff16a936804) * 22:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19215085734 (https://github.com/cluebotng/component-configs/commits/6d8f2491239fbe29d19544922253d9930a88e7a0) * 20:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213996489 (https://github.com/cluebotng/component-configs/commits/38bc77281c9dbd1100915d95ba68705d8a7392a7) * 20:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213940731 (https://github.com/cluebotng/component-configs/commits/c01b89b7b0455d4f1cc63a2eb002f9c55c0a663f) === 2025-11-05 === * 19:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19114484110 (https://github.com/cluebotng/component-configs/commits/fae01bfaeaeca0cf7676ece10cbd39948560086f) * 16:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19108874377 (https://github.com/cluebotng/component-configs/commits/3f51ec3aa53d1378883a9dc973716e57c283d26c) === 2025-10-29 === * 15:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18912872633 (https://github.com/cluebotng/component-configs/commits/3281794d8d1d2e17d9e9859c6f6f7ae3c5216eda) === 2025-10-23 === * 12:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18748267282 (https://github.com/cluebotng/component-configs/commits/bc8f1b883d0d53edf08bea5e5319ee7ee0b4fb82) === 2025-10-07 === * 06:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18304366702 (https://github.com/cluebotng/component-configs/commits/5b83bca0e9293029698d7f3a1b2764727ae7f971) === 2025-10-06 === * 06:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18272390911 (https://github.com/cluebotng/component-configs/commits/49abbdd5dd7066314199c213043305ceed2b54f7) === 2025-10-05 === * 06:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18255184611 (https://github.com/cluebotng/component-configs/commits/7fe1a04069d9d0b4b11019443c85885c202852d4) === 2025-10-03 === * 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18215013747 (https://github.com/cluebotng/component-configs/commits/7ab2bbe022e2513dc81a13a7055c4c7736e5f876) === 2025-09-29 === * 16:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101417 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5) * 15:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18102721922 (https://github.com/cluebotng/component-configs/commits/87ddcf2fce928fde2ba91ecdba3561b12b8de1d2) * 14:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18099932067 (https://github.com/cluebotng/component-configs/commits/0de901e1203dd61656503ef2127efe360e9ed6cc) * 09:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18091994310 (https://github.com/cluebotng/component-configs/commits/ff3951fa5af87196929a9a864f8189b7a7436ac8) * 09:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18091898160 (https://github.com/cluebotng/component-configs/commits/ff3951fa5af87196929a9a864f8189b7a7436ac8) === 2025-09-27 === * 13:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060157048 (https://github.com/cluebotng/component-configs/commits/3aa079ed0cb7aa29f9ece46a47ad96203e53f242) === 2025-09-26 === * 22:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18050560602 (https://github.com/cluebotng/component-configs/commits/886ded0824a9ce7b27c852949f3530bda15bef14) * 11:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18036958606 (https://github.com/cluebotng/component-configs/commits/a51fe109bfad3e2df5aa8e89b837a951bf8ad2cf) * 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18030114498 (https://github.com/cluebotng/component-configs/commits/ea47ef95beb4cf8a1b7d439a83af7b2d4cf168ce) === 2025-09-25 === * 17:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18016018062 (https://github.com/cluebotng/component-configs/commits/150020d96f0c95173ba88c382221223a0c1f7a8d) * 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998409 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c) * 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801858 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c) * 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014528718 (https://github.com/cluebotng/component-configs/commits/96654b441f84901e1a607ced407eb9babb8fdbfc) * 16:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014528718 (https://github.com/cluebotng/component-configs/commits/96654b441f84901e1a607ced407eb9babb8fdbfc) * 16:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221959 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d) * 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782292 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e) * 16:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677351 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a) * 16:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531088 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f) * 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18012641376 (https://github.com/cluebotng/component-configs/commits/87c176492b1f1fb18570dbb70687258843c5773c) * 14:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18008240418 (https://github.com/cluebotng/component-configs/commits/9949a4a5acff374c1edd7b6e21959a28721e02d0) === 2025-09-24 === * 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198575 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4) * 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820837 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960) * 17:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009157 (https://github.com/cluebotng/component-configs/commits/refs/heads/main) * 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983740752 (https://github.com/cluebotng/component-configs/commits/refs/heads/main) * 12:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17977211158 (https://github.com/cluebotng/component-configs/commits/refs/heads/main) * 12:11 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17976182663 * 12:06 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17976062312 * 06:46 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17968643293 === 2025-09-22 === * 20:43 wmbot~component-configs@tools-bastion: Test migrating log to feed channel * 20:43 wmbot~damian-scripts@tools-bastion-15: Test migrating log to feed channel * 19:12 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925793114 === 2025-08-24 === * 18:02 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.3.0 === 2025-08-23 === * 19:44 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.5 * 18:37 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.4 * 18:10 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.3 === 2025-08-14 === * 13:39 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.2 * 13:27 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.1 * 13:20 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.0 === 2025-08-13 === * 20:07 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.10 * 18:53 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.9 === 2025-08-11 === * 19:49 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.7 === 2025-08-10 === * 18:19 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.6 * 15:06 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.5 === 2025-08-07 === * 15:38 wmbot~damian@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.2 === 2023-06-15 === * 18:43 wm-bot: <root> webservice restart, checked pods <noinclude>[[Category:SAL]]</noinclude> q8tr2vlg6r950ofr9pxs9z3aekysb48 2426649 2426628 2026-06-14T10:15:42Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27495607670 (https://github.com/cluebotng/component-configs/commits/63eaa8007e638829714e8661427aa311d662b341) 2426649 wikitext text/x-wiki === 2026-06-14 === * 10:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27495607670 (https://github.com/cluebotng/component-configs/commits/63eaa8007e638829714e8661427aa311d662b341) === 2026-06-13 === * 16:52 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27472937497 (https://github.com/cluebotng/component-configs/commits/38317c72706d30dee2f423e689b46826fa0cbb9a) * 13:03 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27467463970 (https://github.com/cluebotng/component-configs/commits/3dc535380a54d2290621b9d585a5018fdc4669a2) * 11:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27464936848 (https://github.com/cluebotng/component-configs/commits/d19537391f153a63eba67672cf3aecb76bff362e) === 2026-06-11 === * 13:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27350281127 (https://github.com/cluebotng/component-configs/commits/41e666d399f22121043fa8132a3e8bf6368e5181) === 2026-06-10 === * 15:01 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27285125877 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b) * 12:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277789050 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884) * 12:40 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27276698071 (https://github.com/cluebotng/component-configs/commits/8be9293c4b541d74d39482efd21163eb36cda6bd) * 12:37 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27276511398 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1) * 12:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27275411814 (https://github.com/cluebotng/component-configs/commits/869dfb8d1487914e36184a9d1c5aae1e26dbba01) * 12:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27275339348 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11) === 2026-06-09 === * 17:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27225356873 (https://github.com/cluebotng/component-configs/commits/9534cf81437fb2c268eb00e4145978dddbf6322e) === 2026-06-08 === * 12:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27138156886 (https://github.com/cluebotng/component-configs/commits/4677023bc60821948b76a89b2968d9fa3db267d4) === 2026-06-05 === * 14:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27021601200 (https://github.com/cluebotng/component-configs/commits/d4efd5a504c17f41f2d280dabcb635f9c4f07000) * 14:04 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27019408986 (https://github.com/cluebotng/component-configs/commits/4de86ac4b524f394ec49dc167d0c75457e981af1) * 12:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27016160066 (https://github.com/cluebotng/component-configs/commits/3c617707fe44f00c5ac81aa315bfc989a7bd4d00) * 11:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27012814543 (https://github.com/cluebotng/component-configs/commits/48e7143fc26338766785004996ec66dea5876b0e) * 11:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27011683531 (https://github.com/cluebotng/component-configs/commits/7b3ba4bae58717e47510c281fdb60f00be6f7c92) === 2026-06-01 === * 15:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26762978339 (https://github.com/cluebotng/component-configs/commits/9a088c9b8375555c696948825fff7700458b4254) * 13:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26757971428 (https://github.com/cluebotng/component-configs/commits/4790ebea51ebfbd67e51894987e6273e5940cbf1) === 2026-05-31 === * 17:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864796 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3) * 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719862181 (https://github.com/cluebotng/component-configs/commits/8e921d9dd24ae32755f893363b9dfa897cf71c25) * 17:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719681637 (https://github.com/cluebotng/component-configs/commits/02759bc368e1f9901ca399942295a807038321cb) * 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719575507 (https://github.com/cluebotng/component-configs/commits/9bb586c7c04b2a5848b2ebc287497a81506f2d1d) === 2026-05-30 === * 18:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26691257350 (https://github.com/cluebotng/component-configs/commits/1daccd14ff5fe952e32175ee2cf249f2312d99ae) === 2026-05-29 === * 00:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26609639469 (https://github.com/cluebotng/component-configs/commits/8c2fccaaae357774084389157d9a305e72eccb20) === 2026-05-28 === * 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26592792853 (https://github.com/cluebotng/component-configs/commits/a7971b7e286e177862e5318c40b0d4d868efc7c8) === 2026-05-21 === * 20:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26251627549 (https://github.com/cluebotng/component-configs/commits/96f9184e66a6e4b35a49f02940a213125945b056) === 2026-05-19 === * 00:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26068076396 (https://github.com/cluebotng/component-configs/commits/f7db7f6fff0d4d6dd451b5f92e75ba755a74129c) === 2026-05-17 === * 08:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25986406220 (https://github.com/cluebotng/component-configs/commits/be3bb145d2803394cd0b7dbd8ae1775ac9b7cd09) === 2026-05-14 === * 18:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25878752531 (https://github.com/cluebotng/component-configs/commits/21e928fa1870ddaf5fae15afc6f92aa3cb3fb970) === 2026-05-13 === * 02:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773616037 (https://github.com/cluebotng/component-configs/commits/0fd601991775a24b437113d09438e74b996c991b) === 2026-05-12 === * 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25727821525 (https://github.com/cluebotng/component-configs/commits/91aefb7d53013ad152bb721f71980dd26170f297) * 09:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842068 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c) * 09:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724562210 (https://github.com/cluebotng/component-configs/commits/bd0e188642746ab949ec3762676ac730afff1c17) * 08:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25723480598 (https://github.com/cluebotng/component-configs/commits/25c0a1035daa67c2225c0f7f7a414ff5cfb6ed2a) * 08:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723280529 (https://github.com/cluebotng/component-configs/commits/51d7c1919958a7672895885cbb3a1061934d2788) === 2026-05-06 === * 18:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25453754970 (https://github.com/cluebotng/component-configs/commits/92f164d1ab158aea1f76cd0a787f33ffe4017e85) === 2026-05-02 === * 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25252392660 (https://github.com/cluebotng/component-configs/commits/7352cd4f730ca9f5c276772f0b338230989feef4) === 2026-04-24 === * 22:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914278528 (https://github.com/cluebotng/component-configs/commits/23a4b53f3d291b0c750d44a2c0a661333307786d) === 2026-04-20 === * 23:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24695320410 (https://github.com/cluebotng/component-configs/commits/279edf060f43353ea66e6d057773bfdb883b16a1) === 2026-04-17 === * 00:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24540678937 (https://github.com/cluebotng/component-configs/commits/26849735bbefbe218cbe0ce41db5a35941798c7b) === 2026-04-14 === * 21:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24422836848 (https://github.com/cluebotng/component-configs/commits/10f4f0f81e169fac55d056176a273966c8160078) === 2026-04-11 === * 10:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24280508671 (https://github.com/cluebotng/component-configs/commits/5953d3fb9c5e414df6995740382b7bd3be49ced2) * 10:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24280372486 (https://github.com/cluebotng/component-configs/commits/5b34645dff3f37bc9f974635e03cd6b8436f37d1) === 2026-04-10 === * 23:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24268430819 (https://github.com/cluebotng/component-configs/commits/3652893dce02243971055a6ab740363f103ce104) * 23:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24268031378 (https://github.com/cluebotng/component-configs/commits/31367659ada078f50022f1df4b16b6139db27c09) * 22:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24267603054 (https://github.com/cluebotng/component-configs/commits/ba252b54cec9387b47dd4ac4a347d4a9c5118c3e) * 16:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24251572443 (https://github.com/cluebotng/component-configs/commits/2426c8db99c6d44c954ced07c9f41fcaa9e8e549) * 15:53 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24251572443 (https://github.com/cluebotng/component-configs/commits/2426c8db99c6d44c954ced07c9f41fcaa9e8e549) * 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442439 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f) * 15:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249898021 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3) * 14:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247620609 (https://github.com/cluebotng/component-configs/commits/e63a941f5b83d97a9751af731c869062ceef4519) * 14:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247581365 (https://github.com/cluebotng/component-configs/commits/49becfde53d5f960c8e4df0484cebb2bb4d4c5aa) * 13:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24246395413 (https://github.com/cluebotng/component-configs/commits/945fa198e64a0e63b777bb570d57d68ef0ce3f69) * 13:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245816710 (https://github.com/cluebotng/component-configs/commits/6181fdda40150d3535541f3084ac7ff245f19536) * 13:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245353001 (https://github.com/cluebotng/component-configs/commits/97eebf1bcdf5be901e0d3fd82c1b3ea6a8668163) * 13:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245099622 (https://github.com/cluebotng/component-configs/commits/251c10040c01caf2ba9b855050c318d5d2fd8e81) * 13:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24244959273 (https://github.com/cluebotng/component-configs/commits/2a6605ee2d07c0ff0d690aaa8aabed0ca35bab72) * 04:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24226442413 (https://github.com/cluebotng/component-configs/commits/4f895f83dae3f356cae2a1bbcfea51dd9d18bd15) * 01:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24221486227 (https://github.com/cluebotng/component-configs/commits/d96804861818d7786153d18d47be075a4dbbb6f2) === 2026-04-09 === * 20:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24212235180 (https://github.com/cluebotng/component-configs/commits/1cd21afab7312bd0122c0e735f8f4dca03019011) * 19:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24208407251 (https://github.com/cluebotng/component-configs/commits/6cd680dd209bf7fbb01cf24cb6cca82f0fab716d) * 18:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206712296 (https://github.com/cluebotng/component-configs/commits/e7d5ec988541b9d441a5c565f624b7e88e11204f) * 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206093220 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253) * 14:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24195131449 (https://github.com/cluebotng/component-configs/commits/6b512f6db7cc4e49078b135e437185906821ae81) === 2026-04-08 === * 05:03 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24118645397 (https://github.com/cluebotng/component-configs/commits/908dd70b5972cca0c0dafbe50a0020547b833a4e) === 2026-04-07 === * 22:08 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24106708812 (https://github.com/cluebotng/component-configs/commits/b85f56b6997ccf41cc8ea32f33a61809b68b9bc5) === 2026-04-02 === * 22:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23924337776 (https://github.com/cluebotng/component-configs/commits/266152f9f673810b0c9460b5828cb86e7aee31d9) === 2026-03-31 === * 06:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23782957628 (https://github.com/cluebotng/component-configs/commits/7888bbd75773dc064d78ad2ee8949f1540eab0fd) * 01:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23775778726 (https://github.com/cluebotng/component-configs/commits/47f8e20c39e29b952f6dbbd04917970802ce1a0b) === 2026-03-27 === * 17:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656811 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4) * 17:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537327 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b) === 2026-03-23 === * 10:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23433070550 (https://github.com/cluebotng/component-configs/commits/dd92649311ee430b4225d5c6db5d6e6b16d10a86) * 10:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23433054689 (https://github.com/cluebotng/component-configs/commits/c895b3d11b8546e54e4cca5ba350c0a5ca9c5917) === 2026-03-21 === * 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383565693 (https://github.com/cluebotng/component-configs/commits/48390b500ab2b65905e09987c12a3e42c3f69778) * 16:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640335 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa) * 16:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383565693 (https://github.com/cluebotng/component-configs/commits/48390b500ab2b65905e09987c12a3e42c3f69778) * 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383551619 (https://github.com/cluebotng/component-configs/commits/ffff74b90a37a0c6bdd565128d3c11ae195e0763) === 2026-03-20 === * 04:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23329334324 (https://github.com/cluebotng/component-configs/commits/bd7700c30291bfab3a656aa8f257292e287a71ca) === 2026-03-19 === * 09:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23287809902 (https://github.com/cluebotng/component-configs/commits/0976850451c9fbb8c4afb773cc70b91cd7c6fdeb) === 2026-03-17 === * 21:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23217273642 (https://github.com/cluebotng/component-configs/commits/4fe6cb3d8ad39b60746b3b6bd2f83c4d05a82d6b) === 2026-03-13 === * 01:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23031229526 (https://github.com/cluebotng/component-configs/commits/ebd67e60183f161276bf0e13daab55ceb2463eb2) === 2026-03-10 === * 00:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22881598726 (https://github.com/cluebotng/component-configs/commits/bc32d8044077ff83db8b985b87df029ff564ad29) === 2026-03-07 === * 00:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22788149115 (https://github.com/cluebotng/component-configs/commits/b3731fab9a7f4f225ecbe318fa80808de6c904b0) === 2026-03-06 === * 09:08 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22756671151 (https://github.com/cluebotng/component-configs/commits/397fc33968a3c4795b97b1791a0b991ebeb81430) === 2026-03-04 === * 09:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22662750872 (https://github.com/cluebotng/component-configs/commits/01746ef8804c30c85963ea888a75887ebe879e3b) * 01:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22650516003 (https://github.com/cluebotng/component-configs/commits/e7a1e2e06f2ccf038c06cb203369f336c298cf6c) === 2026-03-03 === * 21:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22642685900 (https://github.com/cluebotng/component-configs/commits/3cbfb68b3c0e7d97130ede1be762389f300234d2) === 2026-03-02 === * 01:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22557173037 (https://github.com/cluebotng/component-configs/commits/a414cf552e0a0c0d2c9e9817f922d56a4c899bf6) === 2026-02-27 === * 01:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22468544596 (https://github.com/cluebotng/component-configs/commits/b961f37db0544196a7206882b8e3f2292b7e0894) === 2026-02-25 === * 21:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22415923749 (https://github.com/cluebotng/component-configs/commits/a2a4f5ecffad1b49c96c33b5045430a5b75f71bc) * 11:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22395760068 (https://github.com/cluebotng/component-configs/commits/c6093c4ed72aba8fa453b2f67e48d1effeaabb4b) === 2026-02-24 === * 21:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22370719324 (https://github.com/cluebotng/component-configs/commits/7170b95a5f9b6be3c928684f1e9c436deb3ddd1f) * 00:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22331436693 (https://github.com/cluebotng/component-configs/commits/372e84511fdcb0893755ac22f399d3f24f438f7b) === 2026-02-21 === * 05:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22251332525 (https://github.com/cluebotng/component-configs/commits/29401fbb166f71e375eca7254fe841cb01836d2f) === 2026-02-20 === * 13:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22226385604 (https://github.com/cluebotng/component-configs/commits/fbd7d861a7062a2c09fd2117cbf569beb53916f4) * 08:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22216991615 (https://github.com/cluebotng/component-configs/commits/64d521535aa35454c28900f70009efc0e9ff4a10) * 05:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22212024170 (https://github.com/cluebotng/component-configs/commits/9b0508c1c5a875dd795c865e67f2a93d4f247597) === 2026-02-19 === * 13:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22183076688 (https://github.com/cluebotng/component-configs/commits/919ebb8860a93b9d071da361cad56448a3b1f2b4) * 00:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22163899784 (https://github.com/cluebotng/component-configs/commits/0ce51e9bda73cc3ee0df647f7ba8dcfd02eb97e6) === 2026-02-16 === * 02:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22047780823 (https://github.com/cluebotng/component-configs/commits/3a1f6b151d38aab4ce1a62509b108ac9afc5230b) === 2026-02-15 === * 08:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22032600290 (https://github.com/cluebotng/component-configs/commits/80b9cda20d3f21e2f901db6ccbd168bfffb6b063) === 2026-02-14 === * 20:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22023900270 (https://github.com/cluebotng/component-configs/commits/e8878c3f7a08aa1712126c1b6490f6db41621f44) * 20:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22023724371 (https://github.com/cluebotng/component-configs/commits/de3779f7adea66769077e2380d7b0ce25f3d9e82) * 20:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22023714306 (https://github.com/cluebotng/component-configs/commits/7cd082664340738b5c6cc46d0a195f3814672a3a) * 19:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c) * 19:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c) * 19:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c) * 19:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022827162 (https://github.com/cluebotng/component-configs/commits/0e1693c2b662aaa0c9264ceef355bcbfbc162ea7) * 19:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022587196 (https://github.com/cluebotng/component-configs/commits/57dcc675b3ed54fc17f697d6c7b9554b5d06aab0) * 18:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022444685 (https://github.com/cluebotng/component-configs/commits/1e0b17c59284d25ea8ac39a455abb9921ee6608a) === 2025-11-26 === * 19:58 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19715882591 (https://github.com/cluebotng/component-configs/commits/18c2bc79b5f0023e682a9245197cf87c5cc76943) === 2025-11-11 === * 15:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642915 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c) * 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270263370 (https://github.com/cluebotng/component-configs/commits/d1674e8f4f6cec3b48e848137ce42585278d4a67) === 2025-11-09 === * 22:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19215285730 (https://github.com/cluebotng/component-configs/commits/bf77359fc102b05a026ea8b66dc01ff16a936804) * 22:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19215085734 (https://github.com/cluebotng/component-configs/commits/6d8f2491239fbe29d19544922253d9930a88e7a0) * 20:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213996489 (https://github.com/cluebotng/component-configs/commits/38bc77281c9dbd1100915d95ba68705d8a7392a7) * 20:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213940731 (https://github.com/cluebotng/component-configs/commits/c01b89b7b0455d4f1cc63a2eb002f9c55c0a663f) === 2025-11-05 === * 19:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19114484110 (https://github.com/cluebotng/component-configs/commits/fae01bfaeaeca0cf7676ece10cbd39948560086f) * 16:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19108874377 (https://github.com/cluebotng/component-configs/commits/3f51ec3aa53d1378883a9dc973716e57c283d26c) === 2025-10-29 === * 15:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18912872633 (https://github.com/cluebotng/component-configs/commits/3281794d8d1d2e17d9e9859c6f6f7ae3c5216eda) === 2025-10-23 === * 12:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18748267282 (https://github.com/cluebotng/component-configs/commits/bc8f1b883d0d53edf08bea5e5319ee7ee0b4fb82) === 2025-10-07 === * 06:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18304366702 (https://github.com/cluebotng/component-configs/commits/5b83bca0e9293029698d7f3a1b2764727ae7f971) === 2025-10-06 === * 06:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18272390911 (https://github.com/cluebotng/component-configs/commits/49abbdd5dd7066314199c213043305ceed2b54f7) === 2025-10-05 === * 06:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18255184611 (https://github.com/cluebotng/component-configs/commits/7fe1a04069d9d0b4b11019443c85885c202852d4) === 2025-10-03 === * 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18215013747 (https://github.com/cluebotng/component-configs/commits/7ab2bbe022e2513dc81a13a7055c4c7736e5f876) === 2025-09-29 === * 16:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101417 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5) * 15:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18102721922 (https://github.com/cluebotng/component-configs/commits/87ddcf2fce928fde2ba91ecdba3561b12b8de1d2) * 14:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18099932067 (https://github.com/cluebotng/component-configs/commits/0de901e1203dd61656503ef2127efe360e9ed6cc) * 09:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18091994310 (https://github.com/cluebotng/component-configs/commits/ff3951fa5af87196929a9a864f8189b7a7436ac8) * 09:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18091898160 (https://github.com/cluebotng/component-configs/commits/ff3951fa5af87196929a9a864f8189b7a7436ac8) === 2025-09-27 === * 13:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060157048 (https://github.com/cluebotng/component-configs/commits/3aa079ed0cb7aa29f9ece46a47ad96203e53f242) === 2025-09-26 === * 22:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18050560602 (https://github.com/cluebotng/component-configs/commits/886ded0824a9ce7b27c852949f3530bda15bef14) * 11:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18036958606 (https://github.com/cluebotng/component-configs/commits/a51fe109bfad3e2df5aa8e89b837a951bf8ad2cf) * 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18030114498 (https://github.com/cluebotng/component-configs/commits/ea47ef95beb4cf8a1b7d439a83af7b2d4cf168ce) === 2025-09-25 === * 17:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18016018062 (https://github.com/cluebotng/component-configs/commits/150020d96f0c95173ba88c382221223a0c1f7a8d) * 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998409 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c) * 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801858 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c) * 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014528718 (https://github.com/cluebotng/component-configs/commits/96654b441f84901e1a607ced407eb9babb8fdbfc) * 16:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014528718 (https://github.com/cluebotng/component-configs/commits/96654b441f84901e1a607ced407eb9babb8fdbfc) * 16:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221959 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d) * 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782292 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e) * 16:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677351 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a) * 16:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531088 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f) * 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18012641376 (https://github.com/cluebotng/component-configs/commits/87c176492b1f1fb18570dbb70687258843c5773c) * 14:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18008240418 (https://github.com/cluebotng/component-configs/commits/9949a4a5acff374c1edd7b6e21959a28721e02d0) === 2025-09-24 === * 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198575 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4) * 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820837 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960) * 17:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009157 (https://github.com/cluebotng/component-configs/commits/refs/heads/main) * 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983740752 (https://github.com/cluebotng/component-configs/commits/refs/heads/main) * 12:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17977211158 (https://github.com/cluebotng/component-configs/commits/refs/heads/main) * 12:11 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17976182663 * 12:06 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17976062312 * 06:46 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17968643293 === 2025-09-22 === * 20:43 wmbot~component-configs@tools-bastion: Test migrating log to feed channel * 20:43 wmbot~damian-scripts@tools-bastion-15: Test migrating log to feed channel * 19:12 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925793114 === 2025-08-24 === * 18:02 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.3.0 === 2025-08-23 === * 19:44 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.5 * 18:37 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.4 * 18:10 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.3 === 2025-08-14 === * 13:39 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.2 * 13:27 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.1 * 13:20 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.0 === 2025-08-13 === * 20:07 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.10 * 18:53 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.9 === 2025-08-11 === * 19:49 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.7 === 2025-08-10 === * 18:19 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.6 * 15:06 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.5 === 2025-08-07 === * 15:38 wmbot~damian@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.2 === 2023-06-15 === * 18:43 wm-bot: <root> webservice restart, checked pods <noinclude>[[Category:SAL]]</noinclude> 3v6q0mnjb98ajbn3nkcyx6ldj7gy63c 116 453906 2426648 2426177 2026-06-14T07:24:14Z Gitlabaccountapprovalbot 37332 malahimhaseeb was rejected. 2426648 wikitext text/x-wiki <noinclude>'''Audit log of approvals''' made by [[gitlab:gitlabaccountapprovalbot|@gitlabaccountapprovalbot]]. __NOTOC__</noinclude> === 2026-06-14 === * 07:24 "malahimhaseeb" was rejected (pending since 2026-03-15T07:21:57.748Z). === 2026-06-11 === * 11:48 [[gitlab:cadddr|@cadddr]] was approved. * 11:18 "wikipiggy" was rejected (pending since 2026-03-12T11:16:09.335Z). * 07:15 [[gitlab:vesihiisi|@vesihiisi]] was approved. === 2026-06-10 === * 07:03 [[gitlab:dmiranda|@dmiranda]] was approved. === 2026-06-09 === * 14:21 [[gitlab:linkgenetic|@linkgenetic]] was approved. * 14:03 [[gitlab:sjones-ctr|@sjones-ctr]] was approved. * 12:51 [[gitlab:ekrem|@ekrem]] was approved. === 2026-06-08 === * 17:48 "jmprax" was rejected (pending since 2026-03-09T17:46:38.807Z). * 16:15 [[gitlab:ahonc|@ahonc]] was approved. * 12:57 [[gitlab:rainmonger|@rainmonger]] was approved. === 2026-06-07 === * 23:03 "shadowthewuff" was rejected (pending since 2026-03-08T23:00:53.442Z). * 11:45 "wiki-pavan" was rejected (pending since 2026-03-08T11:45:11.116Z). * 02:39 [[gitlab:launchpad|@launchpad]] was approved. === 2026-06-06 === * 14:54 "unicord" was rejected (pending since 2026-03-07T14:52:04.992Z). * 12:48 "chien" was rejected (pending since 2026-03-07T12:48:11.669Z). === 2026-06-04 === * 14:33 "only-vikas" was rejected (pending since 2026-03-05T14:32:09.186Z). === 2026-06-03 === * 15:00 [[gitlab:anafibnshahibul|@anafibnshahibul]] was approved. === 2026-06-02 === * 21:21 "mgagat" was rejected (pending since 2026-03-03T21:18:37.223Z). * 13:57 "prasunaenumarthy" was rejected (pending since 2026-03-03T13:57:14.847Z). * 05:48 [[gitlab:tmoney|@tmoney]] was approved. === 2026-06-01 === * 14:57 "vikram2101" was rejected (pending since 2026-03-02T14:54:26.550Z). * 12:03 "watshell" was rejected (pending since 2026-03-02T12:03:09.329Z). === 2026-05-29 === * 12:48 "mounikapotladurthi" was rejected (pending since 2026-02-27T12:45:38.609Z). === 2026-05-27 === * 20:00 "vinitha" was rejected (pending since 2026-02-25T19:58:43.524Z). * 16:30 "codeurluce" was rejected (pending since 2026-02-25T16:28:53.973Z). * 14:33 [[gitlab:thilio|@thilio]] was approved. === 2026-05-26 === * 12:09 "charisad" was rejected (pending since 2026-02-24T12:07:21.881Z). === 2026-05-25 === * 22:54 "ddshelto" was rejected (pending since 2026-02-23T22:52:44.427Z). * 19:51 "lakz-99" was rejected (pending since 2026-02-23T19:47:00.263Z). * 19:48 "lakz-99" was rejected (pending since 2026-02-23T19:47:00.263Z). === 2026-05-24 === * 18:45 "jiyagupta-cs" was rejected (pending since 2026-02-22T18:43:33.176Z). === 2026-05-23 === * 13:09 [[gitlab:gauthammohanraj|@gauthammohanraj]] was approved. * 04:21 [[gitlab:staraction|@staraction]] was approved. === 2026-05-22 === * 19:03 "i-horich" was rejected (pending since 2026-02-20T19:00:43.519Z). * 01:48 "50323233" was rejected (pending since 2026-02-20T01:48:05.555Z). === 2026-05-21 === * 18:51 "kartikeyg0104" was rejected (pending since 2026-02-19T18:48:39.707Z). * 16:27 [[gitlab:renovatebot|@renovatebot]] was approved. * 16:06 [[gitlab:gkm563|@gkm563]] was approved. === 2026-05-20 === * 01:21 "beedellrokejulianlockhart" was rejected (pending since 2026-02-18T01:19:13.284Z). === 2026-05-18 === * 23:18 "wladek92" was rejected (pending since 2026-02-16T23:16:22.939Z). * 16:36 [[gitlab:effeietsanders|@effeietsanders]] was approved. === 2026-05-14 === * 21:00 [[gitlab:nehemienathan|@nehemienathan]] was approved. === 2026-05-13 === * 10:51 "ssssaaaa" was rejected (pending since 2026-02-11T10:50:36.975Z). === 2026-05-12 === * 18:06 [[gitlab:psubhashish|@psubhashish]] was approved. * 08:12 "khan" was rejected (pending since 2026-02-10T08:11:48.776Z). * 04:27 "galaxysh" was rejected (pending since 2026-02-10T04:24:59.440Z). === 2026-05-11 === * 12:18 "peterxy12" was rejected (pending since 2026-02-09T12:18:01.982Z). === 2026-05-10 === * 11:09 "yalihupokn" was rejected (pending since 2026-02-08T11:06:51.336Z). * 05:12 "wobadha" was rejected (pending since 2026-02-08T05:11:00.569Z). === 2026-05-09 === * 13:45 "bwiki" was rejected (pending since 2026-02-07T13:43:38.177Z). === 2026-05-08 === * 09:24 [[gitlab:cwilliams|@cwilliams]] was approved. === 2026-05-07 === * 14:15 "rehankhan78" was rejected (pending since 2026-02-05T14:13:37.754Z). === 2026-05-06 === * 11:24 "ari" was rejected (pending since 2026-02-04T11:24:11.760Z). * 08:09 [[gitlab:neriah|@neriah]] was approved. * 06:27 [[gitlab:status401|@status401]] was approved. === 2026-05-03 === * 09:54 [[gitlab:anilk|@anilk]] was approved. === 2026-05-02 === * 17:54 [[gitlab:sweil|@sweil]] was approved. * 17:00 [[gitlab:aoppo|@aoppo]] was approved. === 2026-05-01 === * 21:18 [[gitlab:dawalda|@dawalda]] was approved. === 2026-04-30 === * 21:42 "merohibine" was rejected (pending since 2026-01-29T21:40:00.756Z). * 20:54 [[gitlab:tfmorris|@tfmorris]] was approved. * 17:33 [[gitlab:uyen|@uyen]] was approved. * 07:39 [[gitlab:mahveotm|@mahveotm]] was approved. * 06:36 [[gitlab:leo321|@leo321]] was approved. === 2026-04-29 === * 02:27 [[gitlab:dw31415|@dw31415]] was approved. === 2026-04-28 === * 23:09 [[gitlab:dtorsani|@dtorsani]] was approved. === 2026-04-27 === * 23:42 [[gitlab:quinlan|@quinlan]] was approved. * 05:00 [[gitlab:matthewyeager|@matthewyeager]] was approved. === 2026-04-26 === * 17:36 "kuba-hajnej" was rejected (pending since 2026-01-25T17:33:32.467Z). * 13:03 "jklamo" was rejected (pending since 2026-01-25T13:02:22.936Z). === 2026-04-25 === * 20:24 [[gitlab:maldaxura|@maldaxura]] was approved. * 14:33 [[gitlab:sirtobi|@sirtobi]] was approved. * 04:18 "ice5678" was rejected (pending since 2026-01-24T04:15:30.008Z). === 2026-04-24 === * 22:06 [[gitlab:arcstur|@arcstur]] was approved. === 2026-04-22 === * 23:06 "dtorsani" was rejected (pending since 2026-01-21T23:03:25.843Z). * 22:18 [[gitlab:egezort|@egezort]] was approved. * 16:45 "nexpectarpit" was rejected (pending since 2026-01-21T16:43:21.045Z). === 2026-04-20 === * 19:15 "fitch" was rejected (pending since 2026-01-19T19:12:35.644Z). === 2026-04-19 === * 02:54 [[gitlab:neoact|@neoact]] was approved. === 2026-04-18 === * 07:06 [[gitlab:kockaadmiralac|@kockaadmiralac]] was approved. === 2026-04-17 === * 13:42 "liselot" was rejected (pending since 2026-01-16T13:39:41.909Z). === 2026-04-15 === * 17:03 "lahari" was rejected (pending since 2026-01-14T17:02:06.275Z). === 2026-04-14 === * 13:00 "surajseth520" was rejected (pending since 2026-01-13T12:59:45.906Z). * 04:51 [[gitlab:canley|@canley]] was approved. * 01:03 "bshizzle" was rejected (pending since 2026-01-13T01:00:48.120Z). === 2026-04-13 === * 15:30 [[gitlab:passimacopoulos|@passimacopoulos]] was approved. === 2026-04-11 === * 12:30 "krithash" was rejected (pending since 2026-01-10T12:27:24.731Z). === 2026-04-10 === * 15:30 "raunak1709" was rejected (pending since 2026-01-09T15:29:10.901Z). === 2026-04-07 === * 17:03 [[gitlab:supnabla|@supnabla]] was approved. === 2026-04-06 === * 20:00 [[gitlab:laerdon|@laerdon]] was approved. * 19:21 [[gitlab:ljq3|@ljq3]] was approved. === 2026-04-04 === * 11:06 "mixcc" was rejected (pending since 2026-01-03T11:03:33.922Z). === 2026-04-02 === * 05:30 [[gitlab:mbh1|@mbh1]] was approved. === 2026-04-01 === * 18:21 "yuvrajpatil17" was rejected (pending since 2025-12-31T18:20:27.991Z). * 12:12 [[gitlab:amorii0|@amorii0]] was approved. === 2026-03-31 === * 11:00 "krrishsehgal" was rejected (pending since 2025-12-30T11:00:16.384Z). === 2026-03-30 === * 15:36 [[gitlab:atsuko|@atsuko]] was approved. === 2026-03-29 === * 11:36 [[gitlab:giftcup|@giftcup]] was approved. === 2026-03-28 === * 14:51 [[gitlab:janeeva1|@janeeva1]] was approved. === 2026-03-26 === * 13:36 [[gitlab:saiphani02|@saiphani02]] was approved. * 11:48 [[gitlab:valerioboz-wmch|@valerioboz-wmch]] was approved. === 2026-03-25 === * 09:45 "quansi" was rejected (pending since 2025-12-24T09:42:13.451Z). * 02:18 [[gitlab:viztor|@viztor]] was approved. === 2026-03-24 === * 23:18 [[gitlab:maryyann|@maryyann]] was approved. * 23:01 [[gitlab:codenamenoreste|@codenamenoreste]] was approved. * 13:36 [[gitlab:marc-maillard-wmse|@marc-maillard-wmse]] was approved. * 07:39 "fred2675" was rejected (pending since 2025-12-23T07:39:11.380Z). === 2026-03-23 === * 14:51 [[gitlab:komla|@komla]] was approved. * 05:51 "lunachuck43" was rejected (pending since 2025-12-22T05:50:17.862Z). * 04:06 "reza110011" was rejected (pending since 2025-12-22T04:05:25.117Z). === 2026-03-20 === * 21:54 "mertgor" was rejected (pending since 2025-12-19T21:51:51.419Z). * 20:57 "autanmahmah" was rejected (pending since 2025-12-19T20:54:51.678Z). * 09:57 [[gitlab:nethahussain|@nethahussain]] was approved. * 09:27 [[gitlab:piewriter|@piewriter]] was approved. * 08:15 [[gitlab:dondersmooi|@dondersmooi]] was approved. === 2026-03-19 === * 21:03 "sayvhior" was rejected (pending since 2025-12-18T21:02:31.699Z). === 2026-03-18 === * 20:15 [[gitlab:martinmystere|@martinmystere]] was approved. === 2026-03-17 === * 02:51 "louperivois" was rejected (pending since 2025-12-16T02:50:48.197Z). === 2026-03-16 === * 12:54 "mokayaj857" was rejected (pending since 2025-12-15T12:53:39.015Z). * 06:18 "roamer15" was rejected (pending since 2025-12-15T06:16:38.042Z). === 2026-03-14 === * 11:12 "umaramuhammad" was rejected (pending since 2025-12-13T11:10:44.004Z). * 09:33 "akuma19" was rejected (pending since 2025-12-13T09:31:39.044Z). * 07:06 [[gitlab:syunsyunminmin|@syunsyunminmin]] was approved. === 2026-03-12 === * 20:24 [[gitlab:11wb|@11wb]] was approved. * 09:54 [[gitlab:bcxfu75k|@bcxfu75k]] was approved. === 2026-03-10 === * 09:12 [[gitlab:viktoriahillerudwmse|@viktoriahillerudwmse]] was approved. === 2026-03-06 === * 08:09 "vazhayilnewone" was rejected (pending since 2025-12-05T08:07:02.184Z). === 2026-03-04 === * 20:54 [[gitlab:elphie|@elphie]] was approved. * 11:39 "ronaldahmed" was rejected (pending since 2025-12-03T11:37:47.492Z). * 02:12 "ltslw" was rejected (pending since 2025-12-03T02:11:52.040Z). === 2026-03-02 === * 19:21 "dlopez350" was rejected (pending since 2025-12-01T19:20:38.918Z). * 18:15 [[gitlab:lsandergreen|@lsandergreen]] was approved. === 2026-03-01 === * 10:51 [[gitlab:clintacc|@clintacc]] was approved. === 2026-02-28 === * 09:24 "cardboardlamp" was rejected (pending since 2025-11-29T09:22:03.947Z). * 08:18 "wiki-pavan" was rejected (pending since 2025-11-29T08:16:24.184Z). === 2026-02-27 === * 20:45 "thisisrick25" was rejected (pending since 2025-11-28T20:42:24.454Z). === 2026-02-26 === * 13:57 "chuiimuiiofc" was rejected (pending since 2025-11-27T13:57:02.794Z). * 13:54 "steffpro" was rejected (pending since 2025-11-27T13:52:10.859Z). === 2026-02-25 === * 21:24 "abubakarhabibudayyabu" was rejected (pending since 2025-11-26T21:22:37.776Z). === 2026-02-24 === * 05:00 "playboi" was rejected (pending since 2025-11-25T05:00:30.762Z). === 2026-02-23 === * 14:00 "alph65" was rejected (pending since 2025-11-24T13:59:00.797Z). * 12:33 [[gitlab:robertsky|@robertsky]] was approved. === 2026-02-22 === * 00:30 "hp8p" was rejected (pending since 2025-11-23T00:29:24.741Z). === 2026-02-19 === * 16:45 "clayjar" was rejected (pending since 2025-11-20T16:44:48.380Z). === 2026-02-18 === * 22:18 "nexus" was rejected (pending since 2025-11-19T22:16:48.818Z). * 12:00 "bernsteinnn" was rejected (pending since 2025-11-19T11:59:04.427Z). === 2026-02-17 === * 11:36 "jason2000-cpu" was rejected (pending since 2025-11-18T11:34:00.314Z). === 2026-02-16 === * 14:54 "smaurya" was rejected (pending since 2025-11-17T14:52:06.906Z). === 2026-02-15 === * 16:51 "kra-79" was rejected (pending since 2025-11-16T16:50:41.375Z). === 2026-02-14 === * 15:15 [[gitlab:mess|@mess]] was approved. === 2026-02-13 === * 13:57 "sopalsuemae957" was rejected (pending since 2025-11-14T13:55:16.921Z). * 13:30 [[gitlab:wyslijp16-toolforge|@wyslijp16-toolforge]] was approved. === 2026-02-12 === * 16:30 "kristinagligoric" was rejected (pending since 2025-11-13T16:29:21.646Z). * 03:33 [[gitlab:anyehansen|@anyehansen]] was approved. * 02:21 [[gitlab:thejoyfultentmaker|@thejoyfultentmaker]] was approved. === 2026-02-10 === * 13:18 [[gitlab:db111|@db111]] was approved. === 2026-02-09 === * 19:06 "squirrel289" was rejected (pending since 2025-11-10T19:04:27.831Z). === 2026-02-06 === * 20:54 [[gitlab:gillux|@gillux]] was approved. * 09:09 [[gitlab:lih|@lih]] was approved. === 2026-01-31 === * 16:21 [[gitlab:taxonbot1|@taxonbot1]] was approved. === 2026-01-28 === * 14:30 [[gitlab:ademola|@ademola]] was approved. * 10:51 "watshell" was rejected (pending since 2025-10-29T10:51:01.521Z). === 2026-01-26 === * 23:06 "tavaresgmg" was rejected (pending since 2025-10-27T23:04:42.140Z). === 2026-01-25 === * 06:03 "cata" was rejected (pending since 2025-10-26T06:01:26.155Z). === 2026-01-24 === * 21:15 [[gitlab:wiegels|@wiegels]] was approved. * 06:30 [[gitlab:blaquans|@blaquans]] was approved. === 2026-01-23 === * 16:27 [[gitlab:lerickson|@lerickson]] was approved. * 10:15 "fran0035g" was rejected (pending since 2025-10-24T10:12:17.732Z). === 2026-01-22 === * 21:00 "hacksyn" was rejected (pending since 2025-10-23T20:59:15.982Z). === 2026-01-21 === * 17:30 [[gitlab:otcenas11|@otcenas11]] was approved. === 2026-01-19 === * 21:48 [[gitlab:amdrel|@amdrel]] was approved. * 04:36 "rayalexa" was rejected (pending since 2025-10-20T04:35:02.094Z). === 2026-01-18 === * 15:45 "somya" was rejected (pending since 2025-10-19T15:43:43.701Z). * 06:54 "sergg001" was rejected (pending since 2025-10-19T06:54:12.296Z). === 2026-01-16 === * 11:57 "zeejohsy" was rejected (pending since 2025-10-17T11:56:22.372Z). * 04:45 "rocky25" was rejected (pending since 2025-10-17T04:43:33.180Z). === 2026-01-15 === * 16:39 "tiisu" was rejected (pending since 2025-10-16T16:37:18.438Z). * 12:00 "noahalorwu" was rejected (pending since 2025-10-16T11:58:26.133Z). * 10:39 "prjayaiuedu" was rejected (pending since 2025-10-16T10:37:16.947Z). === 2026-01-13 === * 17:21 [[gitlab:lwilson-ctr|@lwilson-ctr]] was approved. === 2026-01-12 === * 17:03 "stagietechs" was rejected (pending since 2025-10-13T17:02:25.281Z). === 2026-01-10 === * 19:06 "keerthisr" was rejected (pending since 2025-10-11T19:05:01.758Z). === 2026-01-09 === * 20:36 "lightb" was rejected (pending since 2025-10-10T20:34:20.264Z). === 2026-01-08 === * 19:42 [[gitlab:tbodt|@tbodt]] was approved. * 13:57 [[gitlab:martynranyard|@martynranyard]] was approved. === 2026-01-07 === * 17:48 [[gitlab:santanuwiki25|@santanuwiki25]] was approved. * 14:27 "dipanshu" was rejected (pending since 2025-10-08T14:26:10.794Z). * 12:30 "adeolaadesina" was rejected (pending since 2025-10-08T12:29:49.592Z). * 09:21 "tony-kamande" was rejected (pending since 2025-10-08T09:20:28.421Z). * 06:18 "hninwuttyi" was rejected (pending since 2025-10-08T06:17:28.006Z). * 05:09 "andume" was rejected (pending since 2025-10-08T05:07:18.582Z). * 02:00 "mosope" was rejected (pending since 2025-10-08T01:59:54.800Z). * 01:15 [[gitlab:tungstalite|@tungstalite]] was approved. === 2026-01-06 === * 18:24 "leerensucher" was rejected (pending since 2025-10-07T18:21:41.253Z). * 14:54 "leonidlednev" was rejected (pending since 2025-10-07T14:53:07.273Z). * 12:57 "alexandre-tingaud" was rejected (pending since 2025-10-07T12:54:27.206Z). === 2026-01-04 === * 21:33 [[gitlab:matr1x-101|@matr1x-101]] was approved. * 15:18 "makjr" was rejected (pending since 2025-10-05T15:16:31.558Z). * 14:09 "dakshq" was rejected (pending since 2025-10-05T14:08:40.608Z). === 2026-01-03 === * 20:42 [[gitlab:apehitkey|@apehitkey]] was approved. * 18:00 [[gitlab:jeremyb|@jeremyb]] was approved. * 14:09 [[gitlab:twelephant|@twelephant]] was approved. === 2026-01-01 === * 11:30 "shellstanislav" was rejected (pending since 2025-10-02T11:29:10.150Z). === 2025-12-30 === * 19:51 "camilojdiaz" was rejected (pending since 2025-09-30T19:49:24.913Z). === 2025-12-29 === * 16:03 "zied" was rejected (pending since 2025-09-29T16:01:30.415Z). * 08:18 "rahulsidpradhan" was rejected (pending since 2025-09-29T08:17:02.849Z). === 2025-12-26 === * 09:48 "thembo42" was rejected (pending since 2025-09-26T09:45:15.033Z). === 2025-12-25 === * 14:03 "196936074751" was rejected (pending since 2025-09-25T14:02:31.367Z). === 2025-12-23 === * 16:21 "ngarnsworthy" was rejected (pending since 2025-09-23T16:20:41.211Z). === 2025-12-22 === * 12:39 "aza555" was rejected (pending since 2025-09-22T12:38:02.622Z). === 2025-12-20 === * 23:45 "saph" was rejected (pending since 2025-09-20T23:45:01.222Z). === 2025-12-19 === * 10:15 "vladdymoses" was rejected (pending since 2025-09-19T10:15:00.999Z). * 07:15 "dirtylittlepoobah" was rejected (pending since 2025-09-19T07:13:55.537Z). === 2025-12-18 === * 16:24 [[gitlab:guyfawcus|@guyfawcus]] was approved. === 2025-12-17 === * 21:39 [[gitlab:holdyourhorses|@holdyourhorses]] was approved. * 18:30 "prudencia" was rejected (pending since 2025-09-17T18:27:18.860Z). * 02:24 "lottie" was rejected (pending since 2025-09-17T02:21:21.744Z). === 2025-12-16 === * 09:39 [[gitlab:melcatherine|@melcatherine]] was approved. * 08:54 [[gitlab:leila237|@leila237]] was approved. === 2025-12-15 === * 18:27 [[gitlab:royalsailor|@royalsailor]] was approved. * 09:39 [[gitlab:olaf8940|@olaf8940]] was approved. * 09:39 "brianbybyby" was rejected (pending since 2025-09-15T09:37:45.430Z). === 2025-12-14 === * 20:21 [[gitlab:essa237|@essa237]] was approved. * 16:42 [[gitlab:bovimacoco|@bovimacoco]] was approved. === 2025-12-13 === * 21:54 "mmns21" was rejected (pending since 2025-09-13T21:52:24.017Z). * 20:33 "bugcrawler" was rejected (pending since 2025-09-13T20:31:09.211Z). === 2025-12-12 === * 14:39 "ruvchoudhary" was rejected (pending since 2025-09-12T14:36:16.167Z). * 06:54 "rezadress" was rejected (pending since 2025-09-12T06:52:21.749Z). === 2025-12-10 === * 17:30 [[gitlab:itsmoon|@itsmoon]] was approved. === 2025-12-09 === * 15:42 [[gitlab:mercy-o|@mercy-o]] was approved. === 2025-12-06 === * 16:45 "jacquesradjabu" was rejected (pending since 2025-09-06T16:45:17.969Z). * 11:27 [[gitlab:ikhitron|@ikhitron]] was approved. === 2025-12-01 === * 08:12 "halconmilenario21" was rejected (pending since 2025-09-01T08:12:10.262Z). === 2025-11-30 === * 21:06 [[gitlab:habs|@habs]] was approved. === 2025-11-29 === * 16:36 "bovimacoco" was rejected (pending since 2025-08-30T16:34:39.712Z). * 00:45 [[gitlab:jjpmaster|@jjpmaster]] was approved. === 2025-11-24 === * 10:30 "alph65" was rejected (pending since 2025-08-25T10:28:40.957Z). * 02:24 [[gitlab:yaron|@yaron]] was approved. === 2025-11-20 === * 16:06 "clayjar" was rejected (pending since 2025-08-21T16:04:54.450Z). === 2025-11-17 === * 21:09 [[gitlab:ankita97531|@ankita97531]] was approved. === 2025-11-16 === * 14:15 "commanderkefir" was rejected (pending since 2025-08-17T14:13:14.791Z). * 08:21 "rehankhan78" was rejected (pending since 2025-08-17T08:19:44.896Z). === 2025-11-15 === * 14:36 "cyberscribe" was rejected (pending since 2025-08-16T14:34:27.230Z). === 2025-11-13 === * 04:21 "waddie96" was rejected (pending since 2025-08-14T04:19:27.461Z). === 2025-11-11 === * 06:42 [[gitlab:seanhoyland|@seanhoyland]] was approved. === 2025-11-10 === * 00:06 [[gitlab:jaredblumer|@jaredblumer]] was approved. === 2025-11-09 === * 22:36 "heinxiety" was rejected (pending since 2025-08-10T22:33:12.041Z). === 2025-11-07 === * 22:00 [[gitlab:forzagreen|@forzagreen]] was approved. === 2025-11-06 === * 16:57 [[gitlab:rsilvola|@rsilvola]] was approved. === 2025-11-04 === * 21:24 [[gitlab:devdoingdev|@devdoingdev]] was approved. === 2025-11-03 === * 17:48 "joewaleed98" was rejected (pending since 2025-08-04T17:46:12.191Z). === 2025-11-01 === * 18:00 "eliasempresas" was rejected (pending since 2025-08-02T17:58:04.412Z). === 2025-10-31 === * 18:51 [[gitlab:chaoticenby|@chaoticenby]] was approved. * 04:33 "3ch310n" was rejected (pending since 2025-08-01T04:32:21.982Z). === 2025-10-30 === * 10:03 [[gitlab:tausheefhassan|@tausheefhassan]] was approved. === 2025-10-29 === * 14:54 "theap" was rejected (pending since 2025-07-30T14:52:12.066Z). === 2025-10-28 === * 06:06 [[gitlab:tanbiruzzaman|@tanbiruzzaman]] was approved. === 2025-10-27 === * 07:51 [[gitlab:jmoore111|@jmoore111]] was approved. === 2025-10-25 === * 21:09 [[gitlab:valor|@valor]] was approved. * 21:03 [[gitlab:booksmurf|@booksmurf]] was approved. * 02:48 "mystyc1" was rejected (pending since 2025-07-26T02:46:19.373Z). === 2025-10-24 === * 05:12 "aadarshmahesh" was rejected (pending since 2025-07-25T05:09:38.264Z). === 2025-10-22 === * 20:54 [[gitlab:janewanga|@janewanga]] was approved. * 17:27 "abeljeevan" was rejected (pending since 2025-07-23T17:26:46.884Z). * 16:12 "shrimpnaur" was rejected (pending since 2025-07-23T16:10:37.864Z). === 2025-10-21 === * 18:51 "jrmuizel" was rejected (pending since 2025-07-22T18:50:07.315Z). * 09:33 [[gitlab:dpogorzelski|@dpogorzelski]] was approved. === 2025-10-17 === * 13:21 [[gitlab:blegodwin|@blegodwin]] was approved. === 2025-10-16 === * 14:51 [[gitlab:bahago|@bahago]] was approved. * 14:12 "harikrishna0005" was rejected (pending since 2025-07-17T14:10:48.385Z). * 14:09 "gauthammohanraj" was rejected (pending since 2025-07-17T14:08:47.643Z). === 2025-10-15 === * 13:48 [[gitlab:adwivedii|@adwivedii]] was approved. * 13:18 [[gitlab:kimbrenekakande|@kimbrenekakande]] was approved. * 13:03 "childmnajennifer" was rejected (pending since 2025-07-16T13:01:50.236Z). * 05:06 "vssb4214" was rejected (pending since 2025-07-16T05:05:33.985Z). === 2025-10-14 === * 19:39 [[gitlab:afanyulionel|@afanyulionel]] was approved. * 15:33 [[gitlab:sadrettin|@sadrettin]] was approved. * 14:18 [[gitlab:tmwyk|@tmwyk]] was approved. * 08:42 "yasu0796" was rejected (pending since 2025-07-15T08:41:26.453Z). === 2025-10-13 === * 16:09 [[gitlab:atlas0007|@atlas0007]] was approved. === 2025-10-11 === * 17:42 [[gitlab:techwizzie|@techwizzie]] was approved. === 2025-10-10 === * 19:03 [[gitlab:miiswom|@miiswom]] was approved. * 16:06 [[gitlab:ninatakang|@ninatakang]] was approved. === 2025-10-09 === * 15:42 [[gitlab:jaykaneki|@jaykaneki]] was approved. * 14:21 [[gitlab:lebogang|@lebogang]] was approved. * 14:15 [[gitlab:kimondorose|@kimondorose]] was approved. * 13:48 [[gitlab:joyakinyi|@joyakinyi]] was approved. * 13:48 [[gitlab:dikshyashahi|@dikshyashahi]] was approved. * 13:45 [[gitlab:obediobadiah|@obediobadiah]] was approved. * 13:45 [[gitlab:system625|@system625]] was approved. * 13:45 [[gitlab:rolalove|@rolalove]] was approved. * 13:39 [[gitlab:olatundeawo|@olatundeawo]] was approved. * 13:36 [[gitlab:danielchristlight|@danielchristlight]] was approved. * 13:36 [[gitlab:dipanshu1223|@dipanshu1223]] was approved. * 13:36 [[gitlab:aradhya|@aradhya]] was approved. * 09:57 "bognd" was rejected (pending since 2025-07-10T09:55:48.661Z). === 2025-10-08 === * 23:36 [[gitlab:sopzy|@sopzy]] was approved. * 23:03 [[gitlab:oluwatumininu|@oluwatumininu]] was approved. * 19:39 [[gitlab:levon003|@levon003]] was approved. * 15:24 [[gitlab:ritika-bhambri11|@ritika-bhambri11]] was approved. * 13:45 [[gitlab:anbanguyen|@anbanguyen]] was approved. * 13:36 [[gitlab:chumzine|@chumzine]] was approved. * 13:27 [[gitlab:shr0x-ya|@shr0x-ya]] was approved. * 12:45 [[gitlab:nurahwakili|@nurahwakili]] was approved. * 03:42 "nazhiba" was rejected (pending since 2025-07-09T03:40:12.625Z). * 02:12 "mafennel" was rejected (pending since 2025-07-09T02:11:40.598Z). === 2025-10-07 === * 22:54 [[gitlab:olusegunfaj|@olusegunfaj]] was approved. * 21:30 [[gitlab:rona|@rona]] was approved. * 21:09 [[gitlab:sandijigs|@sandijigs]] was approved. * 13:36 "xisbajao" was rejected (pending since 2025-07-08T13:33:35.018Z). * 01:36 "areczek94" was rejected (pending since 2025-07-08T01:35:40.633Z). === 2025-10-06 === * 19:21 "wmcarter2017" was rejected (pending since 2025-07-07T19:21:12.899Z). === 2025-10-05 === * 14:15 "meetmendapara" was rejected (pending since 2025-07-06T14:14:16.726Z). === 2025-10-04 === * 20:51 "nftbaee" was rejected (pending since 2025-07-05T20:50:57.688Z). === 2025-10-03 === * 06:12 [[gitlab:javiermonton|@javiermonton]] was approved. === 2025-10-02 === * 20:15 "talaqalotaibipmp" was rejected (pending since 2025-07-03T20:13:05.164Z). === 2025-10-01 === * 10:54 "bjensen" was rejected (pending since 2025-07-02T10:53:46.574Z). * 02:45 "kowal1984" was rejected (pending since 2025-07-02T02:44:56.946Z). === 2025-09-30 === * 21:21 [[gitlab:kavaljeetsingh|@kavaljeetsingh]] was approved. * 00:24 "adium" was rejected (pending since 2025-07-01T00:23:43.807Z). === 2025-09-28 === * 08:54 [[gitlab:pexerik|@pexerik]] was approved. === 2025-09-27 === * 13:57 [[gitlab:rubahhitamvukova|@rubahhitamvukova]] was approved. === 2025-09-26 === * 16:57 "algorithmic" was rejected (pending since 2025-06-27T16:56:17.480Z). * 13:54 [[gitlab:shadabgdg|@shadabgdg]] was approved. * 13:12 [[gitlab:spushpit|@spushpit]] was approved. === 2025-09-20 === * 14:06 "bwiki" was rejected (pending since 2025-06-21T13:59:14.749Z). === 2025-09-16 === * 05:39 [[gitlab:deepchirp|@deepchirp]] was approved. === 2025-09-15 === * 22:00 [[gitlab:noisk8|@noisk8]] was approved. * 11:03 "ahonc" was rejected (pending since 2025-06-16T11:00:54.843Z). === 2025-09-13 === * 18:24 "a-ssh22" was rejected (pending since 2025-06-14T18:23:33.937Z). * 12:36 [[gitlab:rajashreetalukdar|@rajashreetalukdar]] was approved. * 00:45 [[gitlab:sumitsurai|@sumitsurai]] was approved. === 2025-09-12 === * 17:12 [[gitlab:suyash23|@suyash23]] was approved. * 00:46 "remotetravel" was rejected (pending since 2025-06-13T00:44:08.171Z). === 2025-09-10 === * 21:09 "jancborchardt" was rejected (pending since 2025-06-11T21:06:30.759Z). === 2025-09-09 === * 17:03 [[gitlab:vwf|@vwf]] was approved. * 06:36 [[gitlab:cactusisme|@cactusisme]] was approved. === 2025-09-08 === * 18:09 "birushandegeya" was rejected (pending since 2025-06-09T18:08:00.087Z). * 16:27 "ngarnsworthy" was rejected (pending since 2025-06-09T16:24:37.213Z). * 12:33 "zolgoyo" was rejected (pending since 2025-06-09T12:31:34.199Z). === 2025-09-06 === * 23:09 [[gitlab:jaishsingh913|@jaishsingh913]] was approved. === 2025-09-05 === * 21:45 [[gitlab:sakshi2|@sakshi2]] was approved. * 20:42 "abdukhaliq1" was rejected (pending since 2025-06-06T20:40:42.023Z). * 14:27 "beubsamy" was rejected (pending since 2025-06-06T14:27:06.781Z). === 2025-09-04 === * 23:27 "sdhehua" was rejected (pending since 2025-06-05T23:24:45.777Z). * 19:00 [[gitlab:perry|@perry]] was approved. * 11:24 "saintwolf" was rejected (pending since 2025-06-05T11:21:20.176Z). === 2025-09-02 === * 05:48 [[gitlab:aliu|@aliu]] was approved. === 2025-08-29 === * 13:30 "kksurendran066" was rejected (pending since 2025-05-30T13:27:48.755Z). === 2025-08-28 === * 22:18 "tauraamuix" was rejected (pending since 2025-05-29T22:16:08.228Z). === 2025-08-26 === * 19:03 [[gitlab:dikkulah|@dikkulah]] was approved. === 2025-08-22 === * 23:51 [[gitlab:khoroshun_mike|@khoroshun_mike]] was approved. === 2025-08-21 === * 07:39 [[gitlab:yuka|@yuka]] was approved. === 2025-08-19 === * 07:48 [[gitlab:zhaofjx|@zhaofjx]] was approved. === 2025-08-17 === * 14:27 "madhan13k" was rejected (pending since 2025-05-18T14:26:08.973Z). === 2025-08-15 === * 10:15 "mohammed_abukhadra" was rejected (pending since 2025-05-16T10:14:48.403Z). === 2025-08-11 === * 11:48 "hmmyesbro" was rejected (pending since 2025-05-12T11:45:24.350Z). === 2025-08-10 === * 13:15 [[gitlab:dactyl|@dactyl]] was approved. === 2025-08-09 === * 04:39 "xxxx100000" was rejected (pending since 2025-05-10T04:37:44.949Z). === 2025-08-08 === * 14:33 [[gitlab:josefanthony|@josefanthony]] was approved. === 2025-08-07 === * 23:42 [[gitlab:robins7|@robins7]] was approved. * 21:42 [[gitlab:pols12|@pols12]] was approved. * 17:15 "sbronson" was rejected (pending since 2025-05-08T17:15:08.834Z). * 14:57 [[gitlab:alvindulle|@alvindulle]] was approved. * 14:45 [[gitlab:xentos|@xentos]] was approved. * 06:27 "jamesboste" was rejected (pending since 2025-05-08T06:25:14.793Z). * 03:57 "ysun" was rejected (pending since 2025-05-08T03:55:07.348Z). === 2025-08-06 === * 21:51 "pols12" was rejected (pending since 2025-05-07T21:49:13.598Z). * 01:51 "okeamah" was rejected (pending since 2025-05-07T01:48:50.114Z). === 2025-08-05 === * 09:15 "mobashir-2013" was rejected (pending since 2025-05-06T09:14:24.069Z). === 2025-08-01 === * 08:00 "douginamug" was rejected (pending since 2025-05-02T07:57:38.317Z). === 2025-07-31 === * 02:30 [[gitlab:ads|@ads]] was approved. === 2025-07-27 === * 13:15 "mrico2703" was rejected (pending since 2025-04-27T13:13:12.346Z). * 10:17 [[gitlab:josephfrancis12|@josephfrancis12]] was approved. * 10:17 [[gitlab:fuzzew|@fuzzew]] was approved. * 05:57 [[gitlab:biscuitbobby|@biscuitbobby]] was approved. * 05:48 [[gitlab:ecoholic|@ecoholic]] was approved. === 2025-07-26 === * 11:48 [[gitlab:chimnayyyy|@chimnayyyy]] was approved. * 11:48 [[gitlab:alwinalbert|@alwinalbert]] was approved. * 11:48 [[gitlab:hridyakk|@hridyakk]] was approved. * 11:45 [[gitlab:gaurigupta21|@gaurigupta21]] was approved. * 11:45 [[gitlab:binetaa|@binetaa]] was approved. * 10:21 [[gitlab:jyothikat22|@jyothikat22]] was approved. * 10:21 [[gitlab:zobotrombie|@zobotrombie]] was approved. * 10:21 [[gitlab:flykrth|@flykrth]] was approved. * 10:21 [[gitlab:mehrinshamim|@mehrinshamim]] was approved. * 10:21 [[gitlab:aadhi13|@aadhi13]] was approved. * 10:21 [[gitlab:malavikam05|@malavikam05]] was approved. * 10:18 [[gitlab:nf609|@nf609]] was approved. * 05:48 [[gitlab:nazalnihad|@nazalnihad]] was approved. * 05:48 [[gitlab:naveen28204280|@naveen28204280]] was approved. === 2025-07-25 === * 09:49 [[gitlab:kasyap9|@kasyap9]] was approved. * 09:30 [[gitlab:swayamagrahari|@swayamagrahari]] was approved. === 2025-07-24 === * 19:36 [[gitlab:madutgn|@madutgn]] was approved. === 2025-07-23 === * 20:09 [[gitlab:somerandomdeveloper|@somerandomdeveloper]] was approved. === 2025-07-22 === * 00:15 [[gitlab:iagoqnsi|@iagoqnsi]] was approved. === 2025-07-21 === * 17:30 [[gitlab:asadiqui|@asadiqui]] was approved. * 16:39 [[gitlab:tryvix1509|@tryvix1509]] was approved. * 04:27 [[gitlab:damian|@damian]] was approved. === 2025-07-20 === * 09:42 "mike-khoroshun" was rejected (pending since 2025-04-20T09:42:22.732Z). === 2025-07-17 === * 17:57 [[gitlab:haroldkrabs|@haroldkrabs]] was approved. * 13:45 [[gitlab:envlh|@envlh]] was approved. === 2025-07-14 === * 10:24 [[gitlab:missguru|@missguru]] was approved. * 00:57 "clarfonthey" was rejected (pending since 2025-04-14T00:56:32.626Z). === 2025-07-13 === * 01:01 [[gitlab:l235|@l235]] was approved. === 2025-07-11 === * 03:06 "rodavlas" was rejected (pending since 2025-04-11T03:05:45.590Z). === 2025-07-06 === * 00:09 "lakasa" was rejected (pending since 2025-04-06T00:06:28.469Z). === 2025-07-05 === * 21:54 "ctrlzvi" was rejected (pending since 2025-04-05T21:54:12.542Z). * 14:30 "aminualiyu" was rejected (pending since 2025-04-05T14:27:22.617Z). === 2025-07-04 === * 03:15 [[gitlab:galstar|@galstar]] was approved. === 2025-07-02 === * 11:27 "vicolas11" was rejected (pending since 2025-04-02T11:25:12.682Z). === 2025-06-29 === * 23:12 "naomi723" was rejected (pending since 2025-03-30T23:09:24.630Z). === 2025-06-28 === * 16:21 "mudeh2372" was rejected (pending since 2025-03-29T16:18:27.057Z). === 2025-06-27 === * 23:18 "rony143" was rejected (pending since 2025-03-28T23:16:13.671Z). * 22:21 [[gitlab:rluts|@rluts]] was approved. === 2025-06-26 === * 13:54 "creativegurus" was rejected (pending since 2025-03-27T13:52:41.706Z). === 2025-06-24 === * 17:42 [[gitlab:devjadiya|@devjadiya]] was approved. * 14:00 "dominic-r" was rejected (pending since 2025-03-25T14:00:07.307Z). === 2025-06-21 === * 00:48 [[gitlab:vriaa|@vriaa]] was approved. === 2025-06-18 === * 15:21 "ayushkhati1" was rejected (pending since 2025-03-19T15:18:50.062Z). === 2025-06-17 === * 20:45 "chiomavero" was rejected (pending since 2025-03-18T20:44:13.967Z). * 00:27 [[gitlab:eggroll97|@eggroll97]] was approved. === 2025-06-14 === * 20:57 "volvox" was rejected (pending since 2025-03-15T20:56:34.018Z). === 2025-06-13 === * 16:09 [[gitlab:supergrey|@supergrey]] was approved. * 11:03 "chqaz" was rejected (pending since 2025-03-14T11:01:09.600Z). * 10:24 [[gitlab:slong-wmf|@slong-wmf]] was approved. * 10:15 "hearvox" was rejected (pending since 2025-03-14T10:13:13.112Z). === 2025-06-12 === * 15:18 "jlam" was rejected (pending since 2025-03-13T15:17:54.099Z). === 2025-06-09 === * 20:48 "dipanjansengupta" was rejected (pending since 2025-03-10T20:48:03.545Z). * 19:27 [[gitlab:reggycelly|@reggycelly]] was approved. * 14:51 "arendpieter" was rejected (pending since 2025-03-10T14:51:01.445Z). * 13:21 [[gitlab:greenreaper|@greenreaper]] was approved. * 09:33 [[gitlab:mmta|@mmta]] was approved. * 08:03 "a-ssh22" was rejected (pending since 2025-03-10T08:03:08.111Z). === 2025-06-08 === * 21:06 "mm-episodenlistedlvaupdater" was rejected (pending since 2025-03-09T21:04:06.323Z). === 2025-06-06 === * 11:06 [[gitlab:olea|@olea]] was approved. === 2025-06-05 === * 20:33 [[gitlab:encodedwp|@encodedwp]] was approved. * 15:00 [[gitlab:toluayo|@toluayo]] was approved. * 13:51 [[gitlab:arnold_lup|@arnold_lup]] was approved. * 11:54 "sdhehua" was rejected (pending since 2025-03-06T11:51:48.241Z). === 2025-06-03 === * 21:27 [[gitlab:wewakey|@wewakey]] was approved. * 12:36 "hunsimon2" was rejected (pending since 2025-03-04T12:34:56.520Z). * 11:54 "hunsimon" was rejected (pending since 2025-03-04T11:53:54.652Z). === 2025-06-02 === * 12:01 [[gitlab:jaimedes|@jaimedes]] was approved. === 2025-05-30 === * 18:00 "sathvik9105" was rejected (pending since 2025-02-28T17:59:42.867Z). * 11:21 [[gitlab:tonythomas01|@tonythomas01]] was approved. * 10:06 [[gitlab:gpsleo|@gpsleo]] was approved. === 2025-05-29 === * 22:12 [[gitlab:codynguyen1116|@codynguyen1116]] was approved. === 2025-05-28 === * 02:57 [[gitlab:saper|@saper]] was approved. === 2025-05-27 === * 21:06 [[gitlab:mohammed_qays|@mohammed_qays]] was approved. * 15:33 "satanluimm" was rejected (pending since 2025-02-25T15:32:48.101Z). === 2025-05-26 === * 23:57 "seyedali220" was rejected (pending since 2025-02-24T23:56:17.621Z). === 2025-05-21 === * 11:12 [[gitlab:guilherme|@guilherme]] was approved. === 2025-05-19 === * 13:24 [[gitlab:emojiwiki|@emojiwiki]] was approved. === 2025-05-18 === * 00:00 "xidme" was rejected (pending since 2025-02-15T23:58:56.796Z). === 2025-05-17 === * 02:39 "kdh8219" was rejected (pending since 2025-02-15T02:36:32.237Z). === 2025-05-16 === * 15:09 [[gitlab:maxbinderwmf|@maxbinderwmf]] was approved. === 2025-05-15 === * 04:30 "inspectorzer0" was rejected (pending since 2025-02-13T04:27:33.179Z). === 2025-05-14 === * 17:42 [[gitlab:llugo|@llugo]] was approved. === 2025-05-13 === * 20:18 "mmta" was rejected (pending since 2025-02-11T20:17:23.407Z). === 2025-05-11 === * 20:51 "jad" was rejected (pending since 2025-02-09T20:49:07.333Z). * 17:54 "nishchalsundan" was rejected (pending since 2025-02-09T17:52:25.761Z). * 16:39 "mohammed_abukhadra" was rejected (pending since 2025-02-09T16:39:03.730Z). === 2025-05-09 === * 09:12 [[gitlab:sirchanmp|@sirchanmp]] was approved. === 2025-05-08 === * 08:18 [[gitlab:mengeditch|@mengeditch]] was approved. === 2025-05-07 === * 03:45 "xluffy" was rejected (pending since 2025-02-05T03:45:14.181Z). === 2025-05-06 === * 16:54 "punhaniabhishek" was rejected (pending since 2025-02-04T16:53:50.758Z). * 09:36 [[gitlab:bmartinezcalvo|@bmartinezcalvo]] was approved. === 2025-05-02 === * 12:24 [[gitlab:tohaomg|@tohaomg]] was approved. * 11:48 [[gitlab:mavrikant|@mavrikant]] was approved. * 11:45 [[gitlab:daanvr|@daanvr]] was approved. === 2025-05-01 === * 09:09 "mjoerg" was rejected (pending since 2025-01-30T09:09:04.204Z). === 2025-04-30 === * 23:06 "sanskardubey" was rejected (pending since 2025-01-29T23:03:25.489Z). === 2025-04-29 === * 16:00 "geyslein" was rejected (pending since 2025-01-28T16:00:01.510Z). === 2025-04-26 === * 09:30 "anjali9027" was rejected (pending since 2025-01-25T09:28:07.064Z). === 2025-04-25 === * 18:00 "salahhazaa" was rejected (pending since 2025-01-24T17:58:30.030Z). * 15:15 [[gitlab:yiming|@yiming]] was approved. * 02:06 "mrchanmp" was rejected (pending since 2025-01-24T02:03:58.308Z). === 2025-04-23 === * 17:03 "rj2904" was rejected (pending since 2025-01-22T17:03:11.207Z). * 14:21 "nischay33" was rejected (pending since 2025-01-22T14:19:21.081Z). === 2025-04-22 === * 19:27 "dj80" was rejected (pending since 2025-01-21T19:25:28.498Z). * 14:30 [[gitlab:kaimamin|@kaimamin]] was approved. * 09:57 "debo" was rejected (pending since 2025-01-21T09:54:47.955Z). === 2025-04-21 === * 12:24 "unshell" was rejected (pending since 2025-01-20T12:21:59.686Z). === 2025-04-18 === * 15:06 [[gitlab:spartanarbinger|@spartanarbinger]] was approved. === 2025-04-16 === * 03:09 "dewey" was rejected (pending since 2025-01-15T03:06:17.488Z). === 2025-04-15 === * 19:45 "emdadul" was rejected (pending since 2025-01-14T19:42:29.285Z). === 2025-04-14 === * 06:45 [[gitlab:bcampbell804|@bcampbell804]] was approved. === 2025-04-11 === * 06:27 [[gitlab:jvanderhoop|@jvanderhoop]] was approved. === 2025-04-10 === * 04:12 "bhai420" was rejected (pending since 2025-01-09T04:10:29.430Z). === 2025-04-09 === * 05:03 "austinvarshney" was rejected (pending since 2025-01-08T05:02:34.175Z). === 2025-04-06 === * 15:36 [[gitlab:elph|@elph]] was approved. === 2025-04-02 === * 10:33 [[gitlab:ozge|@ozge]] was approved. === 2025-03-31 === * 20:15 "demandkey" was rejected (pending since 2024-12-30T20:14:23.096Z). * 15:18 [[gitlab:danyya|@danyya]] was approved. === 2025-03-28 === * 15:54 [[gitlab:rutsavi09|@rutsavi09]] was approved. * 15:54 [[gitlab:ilanen1|@ilanen1]] was approved. === 2025-03-25 === * 19:27 [[gitlab:irfo|@irfo]] was approved. * 11:54 [[gitlab:kmontalva-wmf|@kmontalva-wmf]] was approved. * 04:33 [[gitlab:paul26|@paul26]] was approved. * 04:18 "as1100k" was rejected (pending since 2024-12-24T04:18:06.813Z). === 2025-03-24 === * 11:33 "amzadkhankk" was rejected (pending since 2024-12-23T11:33:14.176Z). === 2025-03-23 === * 12:24 "wolfdo" was rejected (pending since 2024-12-22T12:23:35.056Z). === 2025-03-22 === * 09:45 [[gitlab:fjmustak|@fjmustak]] was approved. === 2025-03-20 === * 18:42 "sathishkokila" was rejected (pending since 2024-12-19T18:39:35.161Z). * 17:03 [[gitlab:alien4444|@alien4444]] was approved. * 15:27 [[gitlab:davidcoronel|@davidcoronel]] was approved. === 2025-03-19 === * 22:57 [[gitlab:r1f4t|@r1f4t]] was approved. * 19:03 "daniel24ps" was rejected (pending since 2024-12-18T19:00:21.249Z). * 14:18 [[gitlab:beepbooppenguin|@beepbooppenguin]] was approved. === 2025-03-18 === * 17:48 "rahulkundu1209" was rejected (pending since 2024-12-17T17:46:41.936Z). * 08:15 "kirtisikka972" was rejected (pending since 2024-12-17T08:13:25.487Z). === 2025-03-15 === * 13:30 "tulspal_sidhu" was rejected (pending since 2024-12-14T13:29:10.606Z). * 01:39 "peacedeadc" was rejected (pending since 2024-12-14T01:37:36.579Z). === 2025-03-14 === * 03:51 [[gitlab:chuckthebuck|@chuckthebuck]] was approved. * 02:33 "yxngtrtxll" was rejected (pending since 2024-12-13T02:31:51.658Z). === 2025-03-13 === * 14:36 [[gitlab:iccander|@iccander]] was approved. === 2025-03-12 === * 23:21 "jokerchic36" was rejected (pending since 2024-12-11T23:21:00.670Z). * 15:30 [[gitlab:naomi|@naomi]] was approved. * 15:27 [[gitlab:cobi|@cobi]] was approved. === 2025-03-11 === * 12:42 "mohitvermaxx" was rejected (pending since 2024-12-10T12:40:56.967Z). === 2025-03-10 === * 16:51 [[gitlab:nanona15dobato|@nanona15dobato]] was approved. === 2025-03-09 === * 22:39 [[gitlab:jonkolbert|@jonkolbert]] was approved. * 20:45 [[gitlab:urbanecmtest2|@urbanecmtest2]] was approved. === 2025-03-07 === * 16:54 [[gitlab:hswan|@hswan]] was approved. * 14:42 [[gitlab:atitkov|@atitkov]] was approved. * 00:42 [[gitlab:infrastruktur|@infrastruktur]] was approved. === 2025-03-06 === * 17:21 "johnmann" was rejected (pending since 2024-12-05T17:19:24.995Z). === 2025-03-05 === * 07:33 [[gitlab:monx9494|@monx9494]] was approved. === 2025-03-02 === * 21:21 "paul26" was rejected (pending since 2024-12-01T21:20:19.681Z). === 2025-03-01 === * 19:15 [[gitlab:izno|@izno]] was approved. * 12:45 [[gitlab:nyerho|@nyerho]] was approved. === 2025-02-28 === * 18:27 [[gitlab:chuckonwumelu|@chuckonwumelu]] was approved. * 13:09 "ashwinpraveengo" was rejected (pending since 2024-11-29T13:07:47.240Z). * 00:18 "eduardoaugusto" was rejected (pending since 2024-11-29T00:17:43.372Z). === 2025-02-27 === * 20:39 "volkanurl" was rejected (pending since 2024-11-28T20:37:18.101Z). === 2025-02-24 === * 21:15 [[gitlab:feeglgeef|@feeglgeef]] was approved. * 20:18 [[gitlab:piaanalysis2|@piaanalysis2]] was approved. * 19:06 [[gitlab:dhardy|@dhardy]] was approved. === 2025-02-22 === * 19:27 [[gitlab:owuh|@owuh]] was approved. === 2025-02-19 === * 16:06 [[gitlab:artemkloko|@artemkloko]] was approved. * 13:03 [[gitlab:jgafnea|@jgafnea]] was approved. === 2025-02-17 === * 16:33 [[gitlab:asmartkitten|@asmartkitten]] was approved. === 2025-02-16 === * 19:12 "gaurigupta21" was rejected (pending since 2024-11-17T19:11:07.416Z). === 2025-02-15 === * 01:18 [[gitlab:mediawiki-quickstart-ci|@mediawiki-quickstart-ci]] was approved. === 2025-02-14 === * 15:21 "nathanbnm" was rejected (pending since 2024-11-15T15:18:19.632Z). === 2025-02-13 === * 16:45 [[gitlab:priyanshuchahal|@priyanshuchahal]] was approved. * 16:42 [[gitlab:ajhalili2006|@ajhalili2006]] was approved. === 2025-02-12 === * 23:21 "monkeypatch999" was rejected (pending since 2024-11-13T23:20:38.398Z). * 06:36 [[gitlab:jainlakshita28|@jainlakshita28]] was approved. === 2025-02-11 === * 19:27 [[gitlab:matthewsm2|@matthewsm2]] was approved. === 2025-02-09 === * 16:15 "mohammed_abukhadra" was rejected (pending since 2024-11-10T16:15:18.361Z). === 2025-02-07 === * 21:33 "brennan" was rejected (pending since 2024-11-08T21:31:07.351Z). === 2025-02-06 === * 08:24 "mmta" was rejected (pending since 2024-11-07T08:22:36.724Z). * 06:21 [[gitlab:bunnypranav|@bunnypranav]] was approved. === 2025-02-05 === * 22:39 "chrissteinchen" was rejected (pending since 2024-11-06T22:38:16.673Z). === 2025-02-03 === * 07:45 "edriiic" was rejected (pending since 2024-11-04T07:44:46.849Z). * 01:12 "geppy" was rejected (pending since 2024-11-04T01:10:48.710Z). === 2025-02-02 === * 13:18 "funa-enpitu" was rejected (pending since 2024-11-03T13:15:46.065Z). === 2025-01-31 === * 23:42 "nfontes" was rejected (pending since 2024-11-01T23:39:41.755Z). * 22:51 "sbronson" was rejected (pending since 2024-11-01T22:50:31.871Z). * 00:42 [[gitlab:farid|@farid]] was approved. === 2025-01-27 === * 08:15 [[gitlab:eliza189|@eliza189]] was approved. === 2025-01-25 === * 09:51 [[gitlab:pamputt|@pamputt]] was approved. === 2025-01-23 === * 14:30 [[gitlab:lubianat|@lubianat]] was approved. * 11:45 [[gitlab:bootsa|@bootsa]] was approved. === 2025-01-21 === * 05:09 "niko" was rejected (pending since 2024-07-21T16:10:01.377Z). * 05:09 "thawizkid369777" was rejected (pending since 2024-07-18T17:42:44.493Z). * 05:09 "sarthaksingh2" was rejected (pending since 2024-07-10T11:31:30.470Z). * 05:09 "shriyakt" was rejected (pending since 2024-07-06T04:54:10.248Z). * 05:09 "akshaya" was rejected (pending since 2024-07-06T04:04:51.488Z). * 05:09 "alaka03aj" was rejected (pending since 2024-07-05T18:01:54.876Z). * 05:09 "sulochanaviji-5049" was rejected (pending since 2024-07-01T05:58:00.427Z). * 05:09 "nayanjnath" was rejected (pending since 2024-07-01T02:51:57.405Z). * 05:09 "sd44" was rejected (pending since 2024-06-30T04:28:51.436Z). * 05:09 "metavalent" was rejected (pending since 2024-06-29T01:37:14.210Z). * 05:09 "wicloudx" was rejected (pending since 2024-06-28T11:51:23.335Z). * 05:09 "debo" was rejected (pending since 2024-06-28T01:44:59.845Z). * 05:09 "bwiki" was rejected (pending since 2024-06-23T14:15:38.032Z). * 05:09 "toprak" was rejected (pending since 2024-06-23T11:35:50.819Z). * 05:09 "iristeller" was rejected (pending since 2024-06-14T20:53:48.959Z). * 05:09 "jcolvin" was rejected (pending since 2024-06-12T17:29:01.238Z). * 05:09 "kalyan" was rejected (pending since 2024-06-07T07:52:46.993Z). * 05:09 "bluecrystal" was rejected (pending since 2024-06-06T19:16:20.107Z). * 05:09 "iftttrohit" was rejected (pending since 2024-06-04T12:08:50.818Z). * 05:09 "pogpotato" was rejected (pending since 2024-06-03T17:58:21.684Z). * 05:09 "cptlausebaer" was rejected (pending since 2024-05-31T18:53:27.692Z). * 05:09 "hdevine825" was rejected (pending since 2024-05-31T17:04:18.279Z). * 05:09 "anaghaa18" was rejected (pending since 2024-05-25T19:14:31.803Z). * 05:09 "atharvanair04" was rejected (pending since 2024-05-25T14:24:52.825Z). * 05:09 "anasvemmully" was rejected (pending since 2024-05-25T06:10:27.261Z). * 05:09 "abhinavmohandas" was rejected (pending since 2024-05-25T06:05:24.825Z). * 05:09 "kksurendran06" was rejected (pending since 2024-05-25T06:04:38.082Z). * 05:09 "albertmarshall8896" was rejected (pending since 2024-05-23T09:32:05.462Z). * 05:09 "akellison" was rejected (pending since 2024-05-17T02:07:24.229Z). * 05:09 "mainowill" was rejected (pending since 2024-04-16T23:30:33.881Z). * 05:09 "bzhqc" was rejected (pending since 2024-04-16T19:50:38.676Z). * 05:09 "safan41" was rejected (pending since 2024-04-16T03:34:48.942Z). * 05:09 "mgagat" was rejected (pending since 2024-04-16T03:21:51.764Z). * 05:09 "okeamah" was rejected (pending since 2024-04-16T02:49:00.143Z). * 05:09 "xuhao61" was rejected (pending since 2024-04-15T23:45:09.083Z). * 04:47 "cybel" was rejected (pending since 2024-04-15T06:46:35.791Z). === 2025-01-20 === * 14:33 [[gitlab:your1|@your1]] was approved. === 2025-01-18 === * 10:09 [[gitlab:galrach600|@galrach600]] was approved. * 02:51 [[gitlab:blankeclair|@blankeclair]] was approved. === 2025-01-17 === * 13:57 [[gitlab:dsantamaria|@dsantamaria]] was approved. === 2025-01-15 === * 17:12 [[gitlab:smartse|@smartse]] was approved. === 2025-01-14 === * 17:03 [[gitlab:naorleizer|@naorleizer]] was approved. === 2025-01-13 === * 02:45 [[gitlab:wolf20482|@wolf20482]] was approved. === 2025-01-12 === * 17:45 [[gitlab:tamzin|@tamzin]] was approved. === 2025-01-11 === * 15:24 [[gitlab:bargioni|@bargioni]] was approved. * 14:30 [[gitlab:salelya|@salelya]] was approved. * 10:15 [[gitlab:malakatshy|@malakatshy]] was approved. * 05:21 [[gitlab:newmcpee|@newmcpee]] was approved. === 2025-01-09 === * 15:30 [[gitlab:gkyziridis|@gkyziridis]] was approved. === 2025-01-08 === * 16:21 [[gitlab:ukrface|@ukrface]] was approved. === 2024-12-28 === * 03:27 [[gitlab:twonum|@twonum]] was approved. === 2024-12-25 === * 06:09 [[gitlab:harsv567|@harsv567]] was approved. === 2024-12-21 === * 11:24 [[gitlab:amutha2002|@amutha2002]] was approved. === 2024-12-20 === * 19:51 [[gitlab:hridyeshgupta|@hridyeshgupta]] was approved. * 10:00 [[gitlab:ro-shines|@ro-shines]] was approved. * 08:09 [[gitlab:kesharwaniarpita|@kesharwaniarpita]] was approved. === 2024-12-18 === * 14:45 [[gitlab:soylacarli|@soylacarli]] was approved. === 2024-12-16 === * 20:33 [[gitlab:aleyasiddika1|@aleyasiddika1]] was approved. === 2024-12-15 === * 07:33 [[gitlab:abhishek02bhardwaj|@abhishek02bhardwaj]] was approved. === 2024-12-13 === * 13:18 [[gitlab:ashmitabathre204|@ashmitabathre204]] was approved. === 2024-12-10 === * 06:39 [[gitlab:ginaan|@ginaan]] was approved. === 2024-12-09 === * 05:45 [[gitlab:kallinavya|@kallinavya]] was approved. * 00:54 [[gitlab:viserion-7|@viserion-7]] was approved. === 2024-12-08 === * 17:27 [[gitlab:wargo|@wargo]] was approved. === 2024-12-05 === * 11:15 [[gitlab:ranjithraj|@ranjithraj]] was approved. === 2024-12-02 === * 21:21 [[gitlab:a930913|@a930913]] was approved. === 2024-12-01 === * 02:39 [[gitlab:kingchristlike1|@kingchristlike1]] was approved. === 2024-11-21 === * 13:45 [[gitlab:sascha|@sascha]] was approved. === 2024-11-19 === * 16:36 [[gitlab:jly|@jly]] was approved. === 2024-11-15 === * 02:54 [[gitlab:danielyepezgarces|@danielyepezgarces]] was approved. === 2024-11-14 === * 14:15 [[gitlab:stimoroll|@stimoroll]] was approved. === 2024-11-09 === * 17:15 [[gitlab:f4udeveloper|@f4udeveloper]] was approved. === 2024-11-07 === * 19:15 [[gitlab:zulf|@zulf]] was approved. * 05:33 [[gitlab:hassanamin|@hassanamin]] was approved. === 2024-11-06 === * 19:39 [[gitlab:daniuu|@daniuu]] was approved. * 00:18 [[gitlab:rlopez-wmf|@rlopez-wmf]] was approved. === 2024-10-09 === * 14:45 [[gitlab:jtweed|@jtweed]] was approved. * 10:24 [[gitlab:ifrahkh|@ifrahkh]] was approved. * 09:06 [[gitlab:wikibayer|@wikibayer]] was approved. === 2024-10-06 === * 10:27 [[gitlab:keerthan16|@keerthan16]] was approved. === 2024-10-04 === * 07:45 [[gitlab:hakimi97|@hakimi97]] was approved. === 2024-09-30 === * 07:39 [[gitlab:ninjastrikers|@ninjastrikers]] was approved. === 2024-09-28 === * 17:30 [[gitlab:webrunner95|@webrunner95]] was approved. === 2024-09-18 === * 21:39 [[gitlab:elliottetzkorn|@elliottetzkorn]] was approved. === 2024-09-14 === * 22:06 [[gitlab:humptydumpty|@humptydumpty]] was approved. === 2024-09-06 === * 08:48 [[gitlab:mickabarber|@mickabarber]] was approved. === 2024-08-27 === * 17:36 [[gitlab:edgars|@edgars]] was approved. === 2024-08-22 === * 09:18 [[gitlab:antonkokhwmde|@antonkokhwmde]] was approved. === 2024-08-14 === * 19:21 [[gitlab:jfk|@jfk]] was approved. === 2024-08-13 === * 17:57 [[gitlab:daxserver|@daxserver]] was approved. === 2024-08-11 === * 09:57 [[gitlab:pauliesnug|@pauliesnug]] was approved. === 2024-08-10 === * 08:42 [[gitlab:ashig|@ashig]] was approved. === 2024-08-09 === * 14:09 [[gitlab:masssly|@masssly]] was approved. === 2024-08-05 === * 22:15 [[gitlab:mrtortue|@mrtortue]] was approved. === 2024-08-02 === * 16:21 [[gitlab:dsantini|@dsantini]] was approved. === 2024-07-31 === * 11:54 [[gitlab:cptviraj|@cptviraj]] was approved. === 2024-07-30 === * 19:09 [[gitlab:iniquity|@iniquity]] was approved. * 10:00 [[gitlab:collins|@collins]] was approved. === 2024-07-27 === * 15:57 [[gitlab:songnguxyz|@songnguxyz]] was approved. === 2024-07-25 === * 12:36 [[gitlab:mszabo|@mszabo]] was approved. * 09:21 [[gitlab:agarwalmahima|@agarwalmahima]] was approved. === 2024-07-24 === * 08:05 [[gitlab:dragoniez|@dragoniez]] was approved. === 2024-07-23 === * 06:54 [[gitlab:mirji|@mirji]] was approved. === 2024-07-16 === * 10:00 [[gitlab:lakejason0|@lakejason0]] was approved. === 2024-07-12 === * 11:33 [[gitlab:cn|@cn]] was approved. * 08:12 [[gitlab:unchampignon|@unchampignon]] was approved. === 2024-07-07 === * 17:12 [[gitlab:agamyasamuel|@agamyasamuel]] was approved. * 05:24 [[gitlab:kuldeepburjbhalaike|@kuldeepburjbhalaike]] was approved. === 2024-07-06 === * 11:18 [[gitlab:dibya|@dibya]] was approved. * 04:54 [[gitlab:sarthakparashar|@sarthakparashar]] was approved. === 2024-07-05 === * 18:15 [[gitlab:vanshikarathi|@vanshikarathi]] was approved. === 2024-07-02 === * 19:00 [[gitlab:ebrahim|@ebrahim]] was approved. === 2024-07-01 === * 20:12 [[gitlab:rockingpenny4|@rockingpenny4]] was approved. * 18:15 [[gitlab:balajijagadesh|@balajijagadesh]] was approved. === 2024-06-30 === * 18:24 [[gitlab:hrideshmg|@hrideshmg]] was approved. * 07:18 [[gitlab:chanakyakumardas|@chanakyakumardas]] was approved. * 06:30 [[gitlab:rihaan180|@rihaan180]] was approved. === 2024-06-27 === * 17:36 [[gitlab:driedmueller|@driedmueller]] was approved. === 2024-06-19 === * 12:57 [[gitlab:audreypenven|@audreypenven]] was approved. === 2024-06-16 === * 01:18 [[gitlab:roysmith|@roysmith]] was approved. === 2024-06-08 === * 02:45 [[gitlab:jleedev|@jleedev]] was approved. === 2024-06-03 === * 13:57 [[gitlab:afeder|@afeder]] was approved. === 2024-06-01 === * 10:54 [[gitlab:florianschmitt|@florianschmitt]] was approved. === 2024-05-30 === * 16:42 [[gitlab:krlsca|@krlsca]] was approved. === 2024-05-28 === * 11:24 [[gitlab:rickijay|@rickijay]] was approved. === 2024-05-26 === * 11:18 [[gitlab:ranjithsiji|@ranjithsiji]] was approved. === 2024-05-25 === * 07:24 [[gitlab:jony|@jony]] was approved. === 2024-05-23 === * 08:45 [[gitlab:lepticed7|@lepticed7]] was approved. === 2024-05-22 === * 20:42 [[gitlab:echecs|@echecs]] was approved. === 2024-05-21 === * 13:33 [[gitlab:mbs|@mbs]] was approved. === 2024-05-19 === * 18:06 [[gitlab:ionenlaser|@ionenlaser]] was approved. === 2024-05-18 === * 23:36 [[gitlab:mdaniels5757|@mdaniels5757]] was approved. === 2024-05-17 === * 08:54 [[gitlab:grapedog|@grapedog]] was approved. === 2024-05-08 === * 19:42 [[gitlab:kelhurd|@kelhurd]] was approved. * 19:06 [[gitlab:khurd|@khurd]] was approved. === 2024-05-06 === * 19:48 [[gitlab:j3j5|@j3j5]] was approved. * 12:06 [[gitlab:tk-999|@tk-999]] was approved. === 2024-05-05 === * 22:09 [[gitlab:pppery|@pppery]] was approved. * 20:33 [[gitlab:sakretsu|@sakretsu]] was approved. * 12:12 [[gitlab:waterquark|@waterquark]] was approved. === 2024-05-04 === * 09:03 [[gitlab:multichill|@multichill]] was approved. * 07:42 [[gitlab:abaris|@abaris]] was approved. === 2024-05-03 === * 14:57 [[gitlab:maurusian|@maurusian]] was approved. === 2024-04-24 === * 05:48 [[gitlab:wolfinux|@wolfinux]] was approved. === 2024-04-23 === * 15:48 [[gitlab:dreamrimmer|@dreamrimmer]] was approved. === 2024-04-21 === * 06:51 [[gitlab:alon|@alon]] was approved. === 2024-04-17 === * 23:33 [[gitlab:derenrich|@derenrich]] was approved. === 2024-04-16 === * 17:18 [[gitlab:valcio|@valcio]] was approved. === 2024-04-14 === * 16:51 [[gitlab:wikilucas00|@wikilucas00]] was approved. === 2024-04-06 === * 12:48 [[gitlab:theprotonade|@theprotonade]] was approved. === 2024-04-02 === * 07:30 [[gitlab:bohuizhang|@bohuizhang]] was approved. === 2024-03-30 === * 13:36 [[gitlab:lpintscher|@lpintscher]] was approved. === 2024-03-26 === * 17:09 [[gitlab:eenabulele|@eenabulele]] was approved. === 2024-03-25 === * 14:27 [[gitlab:tuukka|@tuukka]] was approved. === 2024-03-24 === * 12:24 [[gitlab:firefly|@firefly]] was approved. === 2024-03-21 === * 19:33 [[gitlab:universal-omega|@universal-omega]] was approved. === 2024-03-17 === * 10:36 [[gitlab:bisel91|@bisel91]] was approved. === 2024-03-16 === * 10:09 [[gitlab:delord|@delord]] was approved. * 00:42 [[gitlab:athulvis1|@athulvis1]] was approved. === 2024-03-15 === * 19:06 [[gitlab:ignaciorodrguez|@ignaciorodrguez]] was approved. * 08:30 [[gitlab:peachey88|@peachey88]] was approved. * 06:51 [[gitlab:derick|@derick]] was approved. === 2024-03-12 === * 15:06 [[gitlab:xiaoxiao|@xiaoxiao]] was approved. === 2024-03-06 === * 13:21 [[gitlab:desianabae1|@desianabae1]] was approved. === 2024-03-05 === * 19:21 [[gitlab:ep1c|@ep1c]] was approved. * 16:33 [[gitlab:jasmine|@jasmine]] was approved. === 2024-03-02 === * 06:42 [[gitlab:potsdamlamb|@potsdamlamb]] was approved. === 2024-02-29 === * 23:18 [[gitlab:arandomname123|@arandomname123]] was approved. * 18:03 [[gitlab:baba|@baba]] was approved. * 17:48 [[gitlab:yfdyh000|@yfdyh000]] was approved. * 03:09 [[gitlab:sds|@sds]] was approved. === 2024-02-27 === * 23:33 [[gitlab:lofhi|@lofhi]] was approved. === 2024-02-15 === * 19:45 [[gitlab:gergesshamon|@gergesshamon]] was approved. === 2024-02-14 === * 14:33 [[gitlab:philipnelson99|@philipnelson99]] was approved. === 2024-02-13 === * 13:06 [[gitlab:dringsim|@dringsim]] was approved. === 2024-02-12 === * 17:36 [[gitlab:haak|@haak]] was approved. === 2024-02-05 === * 17:33 [[gitlab:qwerfjkl|@qwerfjkl]] was approved. * 17:14 [[gitlab:ahecht|@ahecht]] was approved. === 2024-02-01 === * 09:27 [[gitlab:arinaigum|@arinaigum]] was approved. * 00:15 [[gitlab:jas42|@jas42]] was approved. * 00:15 [[gitlab:edhu|@edhu]] was approved. * 00:15 [[gitlab:marnanel|@marnanel]] was approved. * 00:15 [[gitlab:ibrahemqasim|@ibrahemqasim]] was approved. * 00:15 [[gitlab:amasotti|@amasotti]] was approved. * 00:15 [[gitlab:deni|@deni]] was approved. * 00:15 [[gitlab:cyber|@cyber]] was approved. * 00:15 [[gitlab:saroj|@saroj]] was approved. === 2024-01-29 === * 21:42 [[gitlab:rgupta|@rgupta]] was approved. === 2024-01-07 === * 09:48 [[gitlab:lutrome|@lutrome]] was approved. === 2024-01-05 === * 20:48 [[gitlab:jinoytommanjaly|@jinoytommanjaly]] was approved. * 02:51 [[gitlab:braunobruno|@braunobruno]] was approved. * 01:08 [[gitlab:amorymeltzer|@amorymeltzer]] was approved. * 01:08 [[gitlab:phi22ipus|@phi22ipus]] was approved. === 2024-01-03 === * 14:45 [[gitlab:gabina|@gabina]] was approved. === 2024-01-02 === * 13:18 [[gitlab:arthurtaylor|@arthurtaylor]] was approved. === 2023-12-23 === * 00:33 [[gitlab:aram|@aram]] was approved. === 2023-12-22 === * 16:24 [[gitlab:elpitareio|@elpitareio]] was approved. === 2023-12-21 === * 00:43 [[gitlab:bsadowski1|@bsadowski1]] was approved. * 00:43 [[gitlab:ederporto|@ederporto]] was approved. * 00:43 [[gitlab:sadraiiali|@sadraiiali]] was approved. * 00:43 [[gitlab:wasp-outis|@wasp-outis]] was approved. * 00:43 [[gitlab:bodhisattwa|@bodhisattwa]] was approved. * 00:43 [[gitlab:air7538|@air7538]] was approved. * 00:43 [[gitlab:anzx|@anzx]] was approved. * 00:43 [[gitlab:tekask1903|@tekask1903]] was approved. * 00:42 [[gitlab:kiwi-0x010c|@kiwi-0x010c]] was approved. * 00:42 [[gitlab:mpaa|@mpaa]] was approved. * 00:42 [[gitlab:kutay|@kutay]] was approved. * 00:42 [[gitlab:wattmto|@wattmto]] was approved. er4gy38yunrox5s6hfutj7vw3tat51a 0 454546 2426629 2418630 2026-06-13T18:19:01Z Addshore 138 Bitergia is now at https://development-metrics.wmcloud.org/ 2426629 wikitext text/x-wiki {{Navigation Data Platform}} This page provides links to data documentation for private and public Wikimedia data sources. Its primary audience is WMF data analysts, product teams, and researchers who have an official non-disclosure agreement with the Wikimedia Foundation. * Private data requires [[Data_Platform/Data access#Production access|production data access]]. It includes datasets in WMF's [[Data_Platform/Data_Lake|Data Lake]]: a large, analytics-oriented repository of data about Wikimedia projects. * A selection of public data sources are linked here, but public Wikimedia data is described more fully at [[meta:Research:Data]]. == Traffic data == Analytics data about wiki pageviews and site usage. {{ContentGrid |content= {{Colored box |title = Private traffic data |content = Most Data Lake traffic datasets are updated at hourly granularity, with 2-3 hours lag behind real-time. This data includes: * [[Data_Platform/Data Lake/Traffic/Webrequest|Webrequests]] * [[Data_Platform/Data Lake/Traffic/Pageviews|Pageviews]] * [[Data_Platform/Data Lake/Traffic/Unique Devices|Unique devices]] Full dataset list at [[Data_Platform/Data_Lake/Traffic | Data Lake/Traffic]]. [[File:Datahublogo.png|30x30px|link=https://datahub.wikimedia.org/search?filter__entityType___false___EQUAL___1=DATASET&filter_tags___false___EQUAL___0=urn%3Ali%3Atag%3Atraffic&page=1&query=&unionType=0|alt="DataHub logo"]] [https://datahub.wikimedia.org/search?filter__entityType___false___EQUAL___1=DATASET&filter_tags___false___EQUAL___0=urn%3Ali%3Atag%3Atraffic&page=1&query=&unionType=0 View datasets tagged with "traffic" in DataHub] (requires a [[mw:Developer_account|developer account]]) }} {{Colored box |title = Public traffic data |content = APIs: * [[wmdoc:analytics-api|Wikimedia Analytics API]] (page views, unique devices, media requests, and more) Specialized datasets: * [[meta:Differential_privacy/Completed/Country-project-page|Differentially private pageviews]] [[meta:Data_dumps|Dumps]]: * [https://dumps.wikimedia.org/other/pageview_complete/readme.html Pageview hourly & daily] * [https://archive.org/search?query=subject%3A%22d0cmf%22 Daily pageviews] (d0cmf, records grouped by local wikis) * [https://dumps.wikimedia.org/other/mediacounts/readme.html Mediacounts] * [https://dumps.wikimedia.org/other/clickstream/readme.html Clickstream] Dashboards: * [https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-os Browser statistics] * [https://analytics.wikimedia.org/dashboards/vital-signs/ Readers:Pageviews and Unique Devices] * [https://stats.wikimedia.org Wikistats] * [https://pageviews.wmcloud.org Pageviews tool] * [https://wikinav.toolforge.org/ WikiNav] }} }} == Content data == Datasets that contain full content of revisions for Wikimedia wikis. {{ContentGrid |content= {{Colored box |title = Private content data |content = * [[Data Platform/Data Lake/Content/Mediawiki content history v1|mediawiki_content_history_v1]]: full content of all revisions, past and present, from all wikis * [[Data Platform/Data Lake/Content/Mediawiki content current v1|mediawiki content current v1]]: full content of the latest revisions of all pages on all wikis * [[Data_Platform/Data Lake/Content/Wikidata entity|wikidata_entity]]: Wikidata's latest content in structured form * [[Data_Platform/Data Lake/Content/Wikidata item page link|wikidata_item_page_link]]: links between Wikidata items and corresponding Wikipedia pages in various languages {{Remark|You can [[Wiki_Replicas|access MediaWiki replica databases through Wikimedia Cloud Services]].|reminder}} }} {{Colored box |title = Public content data |content = APIs: * [[mw:Special:MyLanguage/API:Revisions|Revisions]] * [[mw:Special:MyLanguage/API:Parse|Parse]] * [[wikidata:Wikidata:REST_API|Wikidata:REST_API]] {{Remark|Code running on Data Platform servers such as the [[Data Platform/Systems/Stat hosts|stat hosts]] should [[Data Platform/Internal API requests|use the internal API URLs]] in place of the public URLs.|reminder}} [[meta:Data_dumps|Dumps]]: * [https://dumps.wikimedia.org/backup-index.html Wikitext] (use [https://pypi.org/project/mwparserfromhell/ mwparserfromhell]) * [https://dumps.wikimedia.org/other/enterprise_html/ HTML] (use [https://pypi.org/project/mwparserfromhtml/ mwparserfromhtml]) * [https://dumps.wikimedia.org/other/wikibase/commonswiki/ Structured data (image depicts)] Specialized datasets: * [[meta:Research:Knowledge_Gaps_Index/Datasets|Content Gaps]] MediaWiki [[mw:Special:MyLanguage/Manual:Database_layout|database tables]]: * [[mw:Special:MyLanguage/Manual:Text_table|Text]] }} }} == Contributing and edits data == Data about wiki revisions, pages, and users. Includes data about editors and their characteristics. {{ContentGrid |content= {{Colored box |title = Private edits data |content = Edits datasets are generated as monthly snapshots, not continuously updated. This data includes: * [[Data_Platform/Data_Lake/Edits/MediaWiki_history |MediaWiki_history]]: Fully denormalized dataset with user, page and revision data * [[Data_Platform/Data_Lake/Edits#Raw_Mediawiki_data| Raw, unprocessed copies of MediaWiki database tables]], bundled to facilitate cross-wiki queries. Full dataset list at [[Data_Platform/Data_Lake/Edits | Data Lake/Edits]]. }} {{Colored box |title = Private contributors data |content = Private datasets about contributors or editors includes: * [[Data_Platform/Data_Lake/Edits/Geoeditors|Geoeditors]]: Counts of editors by project by country [[File:Datahublogo.png|30x30px|link=https://datahub.wikimedia.org/search?filter__entityType___false___EQUAL___0=DATASET&filter_tags___false___EQUAL___1=urn%3Ali%3Atag%3Aeditors&page=1&query=&unionType=0|alt="DataHub logo"]] [https://datahub.wikimedia.org/search?filter__entityType___false___EQUAL___0=DATASET&filter_tags___false___EQUAL___1=urn%3Ali%3Atag%3Aeditors&page=1&query=&unionType=0 View datasets tagged with "editors" in DataHub] (requires a [[mw:Developer_account|developer account)]]) }} {{Colored box |title = Public edits data |content = APIs: * [https://www.mediawiki.org/w/api.php?action=help&modules=query%2Brevisions Revisions] * [https://www.mediawiki.org/w/api.php?action=help&modules=query%2Ballrevisions Allrevisions] * [https://stream.wikimedia.org/?doc#/streams MediaWiki Event Streams] {{Remark|Code running on Data Platform servers such as the [[Data Platform/Systems/Stat hosts|stat hosts]] should [[Data Platform/Internal API requests|use the internal API URLs]] in place of the public URLs.|reminder}} [[meta:Data_dumps|Dumps]]: * [https://dumps.wikimedia.org/other/mediawiki_history/readme.html Mediawiki_history] MediaWiki [[mw:Special:MyLanguage/Manual:Database_layout|database tables]]: *[[mw:Special:MyLanguage/Manual:Revision_table | Revision table]] Dashboards: * [https://stats.wikimedia.org/ Wikistats] * [https://xtools.wmcloud.org/ XTools] }} {{Colored box |title = Public contributors data |content = APIs: * [https://wikimedia.org/api/rest_v1/#/Editors%20data Geoeditors] * [https://www.mediawiki.org/w/api.php?action=help&modules=query%2Busers users] * [https://www.mediawiki.org/w/api.php?action=help&modules=query%2Busercontribs usercontribs] * [https://www.mediawiki.org/w/api.php?action=help&modules=query%2Bglobaluserinfo globaluserinfo] [[meta:Data_dumps|Dumps]]: * [https://dumps.wikimedia.org/other/mediawiki_history/readme.html Mediawiki_history] * [https://dumps.wikimedia.org/other/geoeditors/readme.html Geoeditors] Specialized datasets: * [[meta:Differential_privacy/Completed/Geoeditors|Differentially private geoeditors (hourly/monthly)]] MediaWiki [[mw:Special:MyLanguage/Manual:Database_layout|database tables]]: * [[mw:Special:MyLanguage/Manual:Actor_table | actor]] * [[mw:Special:MyLanguage/Manual:User_table |user]] * [[mw:Special:MyLanguage/Manual:User_groups_table |user_groups]] * [[mw:Special:MyLanguage/Manual:User_properties_table|user_properties]] Dashboards: * [https://stats.wikimedia.org/ Wikistats] * [https://xtools.wmcloud.org/ XTools] * [https://development-metrics.wmcloud.org/ Development metrics (was Bitergia)] }} }} == Instrumentation and events data == {{Colored box |title = View and query events data |content = Through the [[Event Platform]] and [[Metrics Platform]], you can create and deploy your own instruments to collect event data. Events are [[Data Engineering/Systems/Hadoop Event Ingestion Lifecycle|ingested]] into <code>event</code> and <code>event_sanitized</code> databases in the [[Data_Platform/Data Lake|Data Lake]]. * The Hive table name is a normalized version of the stream name. * The <code>event</code>database stores original (unsanitized) events within a 90 day retention period. * The <code>event_sanitized</code> database is an archive of sanitized events, beyond the 90 day retention period. ** [[Data_Platform/Systems/Event Sanitization|Sanitized event data]] is processed per WMF’s [[foundation:Privacy_policy|Privacy Policy]] and [[metawiki:Data_retention_guidelines|Data Retention Guidelines]]. After the data becomes available, you can [[Data_Platform/Analyze_data|access it with standard query tools]] and [[Data_Platform/Transform_data#Share_data_and_dashboards|create dashboards based on the data]]. See the [[Event_Platform/Instrumentation_How_To#Viewing_and_querying_events|Instrumentation tutorial]] for how to consume events directly from [[Kafka]] or through the internal [[EventStreams]] instance. }} == How to query private data == Visit [[Data_Platform/Analyze_data | Analyze data]] to learn how to run queries and generate visualizations using WMF private datasets and analysis tools. == Report data issues == [[Data_Platform/Data_Lake/Data_Issues|Data Issue reports]] [[Category:Landing page]] [[Category:Data platform]] j7gk0ju9x8wof7oh4zwk1et4wprwttu 498 458787 2426624 2425667 2026-06-13T13:01:34Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27467463967 (https://github.com/cluebotng/component-configs/commits/3dc535380a54d2290621b9d585a5018fdc4669a2) 2426624 wikitext text/x-wiki === 2026-06-13 === * 13:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27467463967 (https://github.com/cluebotng/component-configs/commits/3dc535380a54d2290621b9d585a5018fdc4669a2) === 2026-06-10 === * 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285127629 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b) * 13:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27280628230 (https://github.com/cluebotng/component-configs/commits/9b8bd8bb539a243486f5bdbbee92001379161762) * 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788856 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884) * 12:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276642792 (https://github.com/cluebotng/component-configs/commits/0fef439bee61144769bfe62dddc830113380624e) * 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276510216 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1) * 12:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275397658 (https://github.com/cluebotng/component-configs/commits/33d203fb0e6b88ac6dc34e82ee630f7d4e6fdb56) * 12:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275339730 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11) === 2026-06-05 === * 14:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27021602036 (https://github.com/cluebotng/component-configs/commits/d4efd5a504c17f41f2d280dabcb635f9c4f07000) === 2026-06-01 === * 15:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26762978281 (https://github.com/cluebotng/component-configs/commits/9a088c9b8375555c696948825fff7700458b4254) * 13:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26757969997 (https://github.com/cluebotng/component-configs/commits/4790ebea51ebfbd67e51894987e6273e5940cbf1) === 2026-05-31 === * 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864799 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3) * 17:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719863204 (https://github.com/cluebotng/component-configs/commits/9a01604060b738d8bb8b39dd300634a9976b8737) * 17:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719593861 (https://github.com/cluebotng/component-configs/commits/b7c56032c7bb94a330138a58ef1bf5bb59f8c94c) * 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719577030 (https://github.com/cluebotng/component-configs/commits/9b802544b4b8a568ce3b129be53084ea1f979385) === 2026-05-27 === * 22:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26543153283 (https://github.com/cluebotng/component-configs/commits/b9f54eeee581a4f5a3788d3e16acbc76c7fa6fbb) === 2026-05-12 === * 09:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842151 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c) * 09:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724562336 (https://github.com/cluebotng/component-configs/commits/bd0e188642746ab949ec3762676ac730afff1c17) * 08:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723280675 (https://github.com/cluebotng/component-configs/commits/51d7c1919958a7672895885cbb3a1061934d2788) === 2026-05-08 === * 05:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25537718177 (https://github.com/cluebotng/component-configs/commits/ef11c0e9dcb6c448a2eaeb147343007003b2874f) === 2026-04-29 === * 23:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25139750319 (https://github.com/cluebotng/component-configs/commits/9a101172ba64f47f96b75a6a6d77f65ee589ab4e) === 2026-04-23 === * 09:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24826851188 (https://github.com/cluebotng/component-configs/commits/22ea2eb955d25f4a15e6b234e72a24bc01127a79) * 05:52 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24819177834 (https://github.com/cluebotng/component-configs/commits/50ebb073215919775512bff653c7d337c343c315) === 2026-04-10 === * 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250823601 (https://github.com/cluebotng/component-configs/commits/78526aec436ada2123387a5fc328bf8e3fe7d4a8) * 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250714955 (https://github.com/cluebotng/component-configs/commits/6c8100fde23d02e6b289d65ddd7fc06332eabee3) * 15:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442472 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f) * 15:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250023341 (https://github.com/cluebotng/component-configs/commits/d9e72fa744a319bc8d37238dc1895ad5d11732ba) * 15:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897967 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3) * 14:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24248770650 (https://github.com/cluebotng/component-configs/commits/30b6b38a296396ea7c907cb624fb55006729e637) * 14:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247614901 (https://github.com/cluebotng/component-configs/commits/30bda68a3ea7a1674d174e43cc8651d301c7485c) * 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247572438 (https://github.com/cluebotng/component-configs/commits/e5facd6bf8968a234c139ac18c8d2e72f8345d9e) === 2026-04-09 === * 18:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093212 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253) === 2026-03-27 === * 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659656791 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4) * 17:42 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656791 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4) * 17:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537335 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b) === 2026-03-25 === * 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23535729399 (https://github.com/cluebotng/component-configs/commits/c1468d960041cd66ab50902f344fec1ac65ddcad) === 2026-03-21 === * 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640351 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa) * 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383554920 (https://github.com/cluebotng/component-configs/commits/87ca816f48bdfbf0cbb10d469dacfe87cefd0184) * 16:14 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383551602 (https://github.com/cluebotng/component-configs/commits/ffff74b90a37a0c6bdd565128d3c11ae195e0763) === 2026-02-15 === * 10:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22034380287 (https://github.com/cluebotng/component-configs/commits/842b50dc5d3160000352a25c5fdf09ea88ebf3eb) === 2025-11-11 === * 15:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642949 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c) * 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270301618 (https://github.com/cluebotng/component-configs/commits/f28dcaec8c5882b4a1b7d861fe7f5e400312a5b4) === 2025-11-09 === * 22:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19215642431 (https://github.com/cluebotng/component-configs/commits/68b314152d3679c2a780d1247682bbceaf08ee20) === 2025-11-05 === * 19:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19113995191 (https://github.com/cluebotng/component-configs/commits/586f2c46dcbbb09a9f7926e991bc5fbe45f4a1e9) * 12:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19102217950 (https://github.com/cluebotng/component-configs/commits/24f3dc9fe5e2211d861c754a4b9342a6127f4a4a) === 2025-09-29 === * 16:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101416 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5) * 15:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101460177 (https://github.com/cluebotng/component-configs/commits/f43490cf3ca4913763b07a84c7ac0aa4281e96b4) * 09:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18092072445 (https://github.com/cluebotng/component-configs/commits/a0d50b624a6cdfa221225a08b11c52ed85e54d0c) === 2025-09-26 === * 18:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18046649035 (https://github.com/cluebotng/component-configs/commits/07b907ff75f0289f350549bae5e75bf4e91c91ca) * 12:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18037926539 (https://github.com/cluebotng/component-configs/commits/4950150f14c22c0a7d3df1739fa5537aeba4157d) === 2025-09-25 === * 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998408 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c) * 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801883 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c) * 16:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221945 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d) * 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782309 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e) * 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18012641380 (https://github.com/cluebotng/component-configs/commits/87c176492b1f1fb18570dbb70687258843c5773c) === 2025-09-24 === * 17:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198519 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4) * 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820843 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960) * 17:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009170 (https://github.com/cluebotng/component-configs/commits/refs/heads/main) * 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983738823 (https://github.com/cluebotng/component-configs/commits/refs/heads/main) === 2025-09-22 === * 19:10 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925753138 * 19:01 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925551560 * 18:50 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925326118 === 2025-08-29 === * 00:10 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main === 2025-08-15 === * 21:12 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main * 20:58 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main * 12:55 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main * 00:28 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main * 00:07 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main === 2025-08-11 === * 12:36 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main === 2025-08-10 === * 17:36 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main === 2025-08-08 === * 15:11 wmbot~damian@tools-bastion-13: report deployed @ refs/heads/main * 14:53 wmbot~damian@tools-bastion-13: report deployed @ refs/heads/main === 2025-08-07 === * 15:35 wmbot~damian@tools-bastion-13: report deployed @ v1.0.27 === 2025-05-22 === * 14:42 taavi: cleanup 200G+ of old log files per [[phab:T395006|T395006]] <noinclude>[[Category:SAL]]</noinclude> 04ai9atg7u5wbxagfkp3tq98bpfijpv 116 460070 2426622 2424110 2026-06-13T12:08:11Z EtherpadBackupBot 54504 EYG01 archived by Chlod 2426622 wikitext text/x-wiki === 2026-06-13 === * 12:08 [[etherpad:p/EYG01]] was archived as [[etherpadbackup:EYG01]] by [[User:Chlod|Chlod]] === 2026-06-08 === * 09:23 [[etherpad:p/umlsisXFA4ql4IF1gi2-]] was archived as [[etherpadbackup:umlsisXFA4ql4IF1gi2-]] by [[User:Aristorkle|Aristorkle]] * 09:23 [[etherpad:p/umlsisXFA4ql4IF1gi2-]] was archived as [[etherpadbackup:umlsisXFA4ql4IF1gi2-]] by [[User:Aristorkle|Aristorkle]] === 2026-05-29 === * 03:47 [[etherpad:p/ESEAPCon26_sessions_csv]] was archived as [[etherpadbackup:ESEAPCon26_sessions_csv]] by [[User:Robertsky|Robertsky]] * 03:41 [[etherpad:p/ESEAPCon26_day2_Room2_DB9UHD]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_DB9UHD]] by [[User:Robertsky|Robertsky]] * 03:36 [[etherpad:p/ESEAPCon26_day3_Outofvenue_JD3H99]] was archived as [[etherpadbackup:ESEAPCon26_day3_Outofvenue_JD3H99]] by [[User:Robertsky|Robertsky]] * 03:36 [[etherpad:p/ESEAPCon26_day3_Outofvenue_DBKL7W]] was archived as [[etherpadbackup:ESEAPCon26_day3_Outofvenue_DBKL7W]] by [[User:Robertsky|Robertsky]] * 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_3KZYPS]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_3KZYPS]] by [[User:Robertsky|Robertsky]] * 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_YF7VEZ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_YF7VEZ]] by [[User:Robertsky|Robertsky]] * 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_8TXHAF]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_8TXHAF]] by [[User:Robertsky|Robertsky]] * 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_QRT8BV]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_QRT8BV]] by [[User:Robertsky|Robertsky]] * 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_RFAU8M]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_RFAU8M]] by [[User:Robertsky|Robertsky]] * 03:35 [[etherpad:p/ESEAPCon26_day3_Room2_VU3QL8]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_VU3QL8]] by [[User:Robertsky|Robertsky]] * 03:35 [[etherpad:p/ESEAPCon26_day3_Room2_M3QUZD]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_M3QUZD]] by [[User:Robertsky|Robertsky]] * 03:34 [[etherpad:p/ESEAPCon26_day3_Room2_TFFBBH]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_TFFBBH]] by [[User:Robertsky|Robertsky]] * 03:34 [[etherpad:p/ESEAPCon26_day3_Room2_PQYYHB]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_PQYYHB]] by [[User:Robertsky|Robertsky]] * 03:34 [[etherpad:p/ESEAPCon26_day3_Room2_CA7G3X]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_CA7G3X]] by [[User:Robertsky|Robertsky]] * 03:34 [[etherpad:p/ESEAPCon26_day3_Room2_8RX8NZ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_8RX8NZ]] by [[User:Robertsky|Robertsky]] * 03:34 [[etherpad:p/ESEAPCon26_day3_Room1_JYNFKE]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_JYNFKE]] by [[User:Robertsky|Robertsky]] * 03:34 [[etherpad:p/ESEAPCon26_day3_Room1_LFV39K]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_LFV39K]] by [[User:Robertsky|Robertsky]] * 03:34 [[etherpad:p/ESEAPCon26_day3_Room1_H9JCWN]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_H9JCWN]] by [[User:Robertsky|Robertsky]] * 03:33 [[etherpad:p/ESEAPCon26_day3_Room1_9EF8DZ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_9EF8DZ]] by [[User:Robertsky|Robertsky]] * 03:33 [[etherpad:p/ESEAPCon26_day3_Room1_8EQWQL]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_8EQWQL]] by [[User:Robertsky|Robertsky]] * 03:33 [[etherpad:p/ESEAPCon26_day3_Room1_A87WBV]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_A87WBV]] by [[User:Robertsky|Robertsky]] * 03:33 [[etherpad:p/ESEAPCon26_day3_Room1_3K3T9M]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_3K3T9M]] by [[User:Robertsky|Robertsky]] * 03:32 [[etherpad:p/ESEAPCon26_day3_Room1_9DJJP7]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_9DJJP7]] by [[User:Robertsky|Robertsky]] * 03:32 [[etherpad:p/ESEAPCon26_day3_Room1_AKYUKC]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_AKYUKC]] by [[User:Robertsky|Robertsky]] * 03:32 [[etherpad:p/ESEAPCon26_day3_Room1_S99FWD]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_S99FWD]] by [[User:Robertsky|Robertsky]] * 03:32 [[etherpad:p/ESEAPCon26_day3_Mainroom_HGHY9E]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_HGHY9E]] by [[User:Robertsky|Robertsky]] * 03:32 [[etherpad:p/ESEAPCon26_day3_Mainroom_YL8ST7]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_YL8ST7]] by [[User:Robertsky|Robertsky]] * 03:32 [[etherpad:p/ESEAPCon26_day3_Mainroom_8ZNVWS]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_8ZNVWS]] by [[User:Robertsky|Robertsky]] * 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_C7ZRET]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_C7ZRET]] by [[User:Robertsky|Robertsky]] * 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_NCHDPG]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_NCHDPG]] by [[User:Robertsky|Robertsky]] * 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_TW9WFJ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_TW9WFJ]] by [[User:Robertsky|Robertsky]] * 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_VNX3UJ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_VNX3UJ]] by [[User:Robertsky|Robertsky]] * 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_BQHSPF]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_BQHSPF]] by [[User:Robertsky|Robertsky]] * 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_ZNUEAR]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_ZNUEAR]] by [[User:Robertsky|Robertsky]] * 03:30 [[etherpad:p/ESEAPCon26_day2_Outofvenue_VVR7KR]] was archived as [[etherpadbackup:ESEAPCon26_day2_Outofvenue_VVR7KR]] by [[User:Robertsky|Robertsky]] * 03:29 [[etherpad:p/ESEAPCon26_day2_Outofvenue_CJTRQM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Outofvenue_CJTRQM]] by [[User:Robertsky|Robertsky]] * 03:29 [[etherpad:p/ESEAPCon26_day2_Room3_DEXKVE]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_DEXKVE]] by [[User:Robertsky|Robertsky]] * 03:29 [[etherpad:p/ESEAPCon26_day2_Room3_7DSNAX]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_7DSNAX]] by [[User:Robertsky|Robertsky]] * 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_DZGGRB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_DZGGRB]] by [[User:Robertsky|Robertsky]] * 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_7MBXLG]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_7MBXLG]] by [[User:Robertsky|Robertsky]] * 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_Q3XQYZ]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_Q3XQYZ]] by [[User:Robertsky|Robertsky]] * 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_G9S7TM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_G9S7TM]] by [[User:Robertsky|Robertsky]] * 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_TDXWE3]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_TDXWE3]] by [[User:Robertsky|Robertsky]] * 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_HZZZKL]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_HZZZKL]] by [[User:Robertsky|Robertsky]] * 03:27 [[etherpad:p/ESEAPCon26_day2_Room2_JTBXUG]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_JTBXUG]] by [[User:Robertsky|Robertsky]] * 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_ALULGS]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_ALULGS]] by [[User:Robertsky|Robertsky]] * 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_MNBFWL]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_MNBFWL]] by [[User:Robertsky|Robertsky]] * 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_RRHPKR]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_RRHPKR]] by [[User:Robertsky|Robertsky]] * 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_QP3MFV]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_QP3MFV]] by [[User:Robertsky|Robertsky]] * 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_AURUVZ]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_AURUVZ]] by [[User:Robertsky|Robertsky]] * 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_3MAKHH]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_3MAKHH]] by [[User:Robertsky|Robertsky]] * 03:25 [[etherpad:p/ESEAPCon26_day2_Room2_TXFRZB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_TXFRZB]] by [[User:Robertsky|Robertsky]] * 03:25 [[etherpad:p/ESEAPCon26_day2_Room2_QUFR97]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_QUFR97]] by [[User:Robertsky|Robertsky]] * 03:25 [[etherpad:p/ESEAPCon26_day2_Room2_JT7TFH]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_JT7TFH]] by [[User:Robertsky|Robertsky]] * 03:24 [[etherpad:p/ESEAPCon26_day2_Room1_LJ3BL8]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_LJ3BL8]] by [[User:Robertsky|Robertsky]] * 03:24 [[etherpad:p/ESEAPCon26_day2_Room1_9YDQCT]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_9YDQCT]] by [[User:Robertsky|Robertsky]] * 03:24 [[etherpad:p/ESEAPCon26_day2_Room1_GKN9F7]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_GKN9F7]] by [[User:Robertsky|Robertsky]] * 03:24 [[etherpad:p/ESEAPCon26_day2_Room1_CFUMZ9]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_CFUMZ9]] by [[User:Robertsky|Robertsky]] * 03:22 [[etherpad:p/ESEAPCon26_day2_Room1_CRTN93]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_CRTN93]] by [[User:Robertsky|Robertsky]] * 03:22 [[etherpad:p/ESEAPCon26_day2_Room1_MGZMP8]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_MGZMP8]] by [[User:Robertsky|Robertsky]] * 03:22 [[etherpad:p/ESEAPCon26_day2_Room1_8WQJKB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_8WQJKB]] by [[User:Robertsky|Robertsky]] * 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_LLPZSB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_LLPZSB]] by [[User:Robertsky|Robertsky]] * 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_HMSJPM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_HMSJPM]] by [[User:Robertsky|Robertsky]] * 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_SWCDHM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_SWCDHM]] by [[User:Robertsky|Robertsky]] * 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_3CDL7X]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_3CDL7X]] by [[User:Robertsky|Robertsky]] * 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_AHCNHB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_AHCNHB]] by [[User:Robertsky|Robertsky]] * 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_LTSVEH]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_LTSVEH]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_ZVYUWS]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_ZVYUWS]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_T8YSUB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_T8YSUB]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_LMXU7L]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_LMXU7L]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_YKMM9C]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_YKMM9C]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_RZKSWJ]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_RZKSWJ]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day1_Outofvenue_3RMDEL]] was archived as [[etherpadbackup:ESEAPCon26_day1_Outofvenue_3RMDEL]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day1_Room3_XAJUWY]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room3_XAJUWY]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day1_Room2_FSZJK7]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_FSZJK7]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day1_Room2_W3ND3N]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_W3ND3N]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day1_Room2_8AR9U9]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_8AR9U9]] by [[User:Robertsky|Robertsky]] * 03:15 [[etherpad:p/ESEAPCon26_day1_Room2_NUWQDM]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_NUWQDM]] by [[User:Robertsky|Robertsky]] * 03:15 [[etherpad:p/ESEAPCon26_day1_Room2_DGZT88]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_DGZT88]] by [[User:Robertsky|Robertsky]] * 03:15 [[etherpad:p/ESEAPCon26_day1_Room2_FPWESV]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_FPWESV]] by [[User:Robertsky|Robertsky]] * 03:15 [[etherpad:p/ESEAPCon26_day1_Room1_7EKDB9]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_7EKDB9]] by [[User:Robertsky|Robertsky]] * 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_XVFLQU]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_XVFLQU]] by [[User:Robertsky|Robertsky]] * 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_KCR737]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_KCR737]] by [[User:Robertsky|Robertsky]] * 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_EBATCZ]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_EBATCZ]] by [[User:Robertsky|Robertsky]] * 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_MGKRNF]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_MGKRNF]] by [[User:Robertsky|Robertsky]] * 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_DHFDW7]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_DHFDW7]] by [[User:Robertsky|Robertsky]] * 03:14 [[etherpad:p/ESEAPCon26_day1_Mainroom_CLMDUY]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_CLMDUY]] by [[User:Robertsky|Robertsky]] * 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_B7Y3PM]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_B7Y3PM]] by [[User:Robertsky|Robertsky]] * 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_WGT9AQ]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_WGT9AQ]] by [[User:Robertsky|Robertsky]] * 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_AXG3KB]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_AXG3KB]] by [[User:Robertsky|Robertsky]] * 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_L8GRH9]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_L8GRH9]] by [[User:Robertsky|Robertsky]] * 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_UAF8RA]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_UAF8RA]] by [[User:Robertsky|Robertsky]] * 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_NTXNZV]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_NTXNZV]] by [[User:Robertsky|Robertsky]] === 2026-05-26 === * 21:47 [[etherpad:p/Wikimedia_Hackathon_2026_Closing_Showcase]] was archived as [[etherpadbackup:Wikimedia_Hackathon_2026_Closing_Showcase]] by [[User:Gnoeee|Gnoeee]] * 21:47 [[etherpad:p/South_Asia_Community_Call]] was archived as [[etherpadbackup:South_Asia_Community_Call]] by [[User:Gnoeee|Gnoeee]] === 2026-05-23 === * 09:18 [[etherpad:p/ESEAPCon26_day2_Room3_G9S7TM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_G9S7TM]] by [[User:Ericliu1912|Ericliu1912]] * 09:18 [[etherpad:p/ESEAPCon26_day2_Room2_QP3MFV]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_QP3MFV]] by [[User:Ericliu1912|Ericliu1912]] * 02:46 [[etherpad:p/org_structure_reading_group]] was archived as [[etherpadbackup:org_structure_reading_group]] by [[User:BryanDavis|BryanDavis]] === 2026-05-18 === * 19:20 [[etherpad:p/Chartres]] was archived as [[etherpadbackup:Chartres]] by [[User:Sj|Sj]] * 19:08 [[etherpad:p/city-homelessness-okn]] was archived as [[etherpadbackup:city-homelessness-okn]] by [[User:Sj|Sj]] * 19:07 [[etherpad:p/WM-structural-problems-overview]] was archived as [[etherpadbackup:WM-structural-problems-overview]] by [[User:Sj|Sj]] * 19:03 [[etherpad:p/WLM-2021-May]] was archived as [[etherpadbackup:WLM-2021-May]] by [[User:Sj|Sj]] * 19:02 [[etherpad:p/WLM-2021-April]] was archived as [[etherpadbackup:WLM-2021-April]] by [[User:Sj|Sj]] * 19:02 [[etherpad:p/shexcg_minutes]] was archived as [[etherpadbackup:shexcg_minutes]] by [[User:Sj|Sj]] * 19:01 [[etherpad:p/ctp-letter]] was archived as [[etherpadbackup:ctp-letter]] by [[User:Sj|Sj]] * 18:59 [[etherpad:p/Wikimania_oral_history]] was archived as [[etherpadbackup:Wikimania_oral_history]] by [[User:Sj|Sj]] * 18:55 [[etherpad:p/covid19]] was archived as [[etherpadbackup:covid19]] by [[User:Sj|Sj]] * 18:54 [[etherpad:p/Covering_George_Floyd]] was archived as [[etherpadbackup:Covering_George_Floyd]] by [[User:Sj|Sj]] * 17:37 [[etherpad:p/JROSTS18-HACKDAY]] was archived as [[etherpadbackup:JROSTS18-HACKDAY]] by [[User:Sj|Sj]] * 17:36 [[etherpad:p/JROSTS18-UNCFF]] was archived as [[etherpadbackup:JROSTS18-UNCFF]] by [[User:Sj|Sj]] * 17:36 [[etherpad:p/JROSTS18-UNCFE]] was archived as [[etherpadbackup:JROSTS18-UNCFE]] by [[User:Sj|Sj]] * 17:36 [[etherpad:p/JROSTS18-UNCFD]] was archived as [[etherpadbackup:JROSTS18-UNCFD]] by [[User:Sj|Sj]] * 17:36 [[etherpad:p/JROSTS18-UNCFC]] was archived as [[etherpadbackup:JROSTS18-UNCFC]] by [[User:Sj|Sj]] * 17:36 [[etherpad:p/JROSTS18-UNCFB]] was archived as [[etherpadbackup:JROSTS18-UNCFB]] by [[User:Sj|Sj]] * 17:36 [[etherpad:p/JROSTS18-OA.2]] was archived as [[etherpadbackup:JROSTS18-OA.2]] by [[User:Sj|Sj]] * 17:36 [[etherpad:p/JROSTS18-UNCONFERENCING]] was archived as [[etherpadbackup:JROSTS18-UNCONFERENCING]] by [[User:Sj|Sj]] * 17:36 [[etherpad:p/JROSTS18-JROST]] was archived as [[etherpadbackup:JROSTS18-JROST]] by [[User:Sj|Sj]] * 17:35 [[etherpad:p/JROSTS18-OB.1]] was archived as [[etherpadbackup:JROSTS18-OB.1]] by [[User:Sj|Sj]] * 17:35 [[etherpad:p/JROSTS18-EOA]] was archived as [[etherpadbackup:JROSTS18-EOA]] by [[User:Sj|Sj]] * 17:35 [[etherpad:p/JROSTS18-WFC.3]] was archived as [[etherpadbackup:JROSTS18-WFC.3]] by [[User:Sj|Sj]] * 17:35 [[etherpad:p/JROSTS18-WFC.2]] was archived as [[etherpadbackup:JROSTS18-WFC.2]] by [[User:Sj|Sj]] * 17:34 [[etherpad:p/JROSTS18-WFA.2]] was archived as [[etherpadbackup:JROSTS18-WFA.2]] by [[User:Sj|Sj]] * 17:34 [[etherpad:p/JROSTS18-WFA.1]] was archived as [[etherpadbackup:JROSTS18-WFA.1]] by [[User:Sj|Sj]] * 17:33 [[etherpad:p/JROSTS18-CROSSCUTTING]] was archived as [[etherpadbackup:JROSTS18-CROSSCUTTING]] by [[User:Sj|Sj]] * 17:33 [[etherpad:p/JROST-Master]] was archived as [[etherpadbackup:JROST-Master]] by [[User:Sj|Sj]] * 17:33 [[etherpad:p/JROSTS18-WFC.1]] was archived as [[etherpadbackup:JROSTS18-WFC.1]] by [[User:Sj|Sj]] * 17:19 [[etherpad:p/15.320]] was archived as [[etherpadbackup:15.320]] by [[User:Sj|Sj]] * 17:17 [[etherpad:p/medical_hotspots]] was archived as [[etherpadbackup:medical_hotspots]] by [[User:Sj|Sj]] * 17:16 [[etherpad:p/community_health]] was archived as [[etherpadbackup:community_health]] by [[User:Sj|Sj]] * 17:14 [[etherpad:p/whatsnext]] was archived as [[etherpadbackup:whatsnext]] by [[User:Sj|Sj]] * 17:12 [[etherpad:p/MWF20160318]] was archived as [[etherpadbackup:MWF20160318]] by [[User:Sj|Sj]] * 17:12 [[etherpad:p/MWF20160226]] was archived as [[etherpadbackup:MWF20160226]] by [[User:Sj|Sj]] * 17:11 [[etherpad:p/ceem14-pr]] was archived as [[etherpadbackup:ceem14-pr]] by [[User:Sj|Sj]] * 16:54 [[etherpad:p/Templates,_Page_Components_and_editing]] was archived as [[etherpadbackup:Templates,_Page_Components_and_editing]] by [[User:Sj|Sj]] * 16:54 [[etherpad:p/GLAM-Wikipedia-fr]] was archived as [[etherpadbackup:GLAM-Wikipedia-fr]] by [[User:Sj|Sj]] * 16:49 [[etherpad:p/editcontentmodel]] was archived as [[etherpadbackup:editcontentmodel]] by [[User:Sj|Sj]] * 16:43 [[etherpad:p/multimedia-weekly-meeting-2014-05-14]] was archived as [[etherpadbackup:multimedia-weekly-meeting-2014-05-14]] by [[User:Sj|Sj]] * 16:06 [[etherpad:p/cuny-wikiwed-2026-04]] was archived as [[etherpadbackup:cuny-wikiwed-2026-04]] by [[User:Pharos|Pharos]] * 16:06 [[etherpad:p/cuny-wikiwed-2026-03]] was archived as [[etherpadbackup:cuny-wikiwed-2026-03]] by [[User:Pharos|Pharos]] * 16:06 [[etherpad:p/cuny-wikiwed-2026-01]] was archived as [[etherpadbackup:cuny-wikiwed-2026-01]] by [[User:Pharos|Pharos]] * 16:06 [[etherpad:p/cuny-wikiwed-2025-11]] was archived as [[etherpadbackup:cuny-wikiwed-2025-11]] by [[User:Pharos|Pharos]] * 16:05 [[etherpad:p/cuny-wikiwed-2025-10]] was archived as [[etherpadbackup:cuny-wikiwed-2025-10]] by [[User:Pharos|Pharos]] * 16:05 [[etherpad:p/cuny-wikiwed-2025-09]] was archived as [[etherpadbackup:cuny-wikiwed-2025-09]] by [[User:Pharos|Pharos]] * 16:05 [[etherpad:p/cuny-wikiwed-2025-08]] was archived as [[etherpadbackup:cuny-wikiwed-2025-08]] by [[User:Pharos|Pharos]] * 16:05 [[etherpad:p/cuny-wikiwed-2025-07]] was archived as [[etherpadbackup:cuny-wikiwed-2025-07]] by [[User:Pharos|Pharos]] * 16:04 [[etherpad:p/cuny-wikiwed-2025-06]] was archived as [[etherpadbackup:cuny-wikiwed-2025-06]] by [[User:Pharos|Pharos]] * 16:04 [[etherpad:p/cuny-wikiwed-2025-05]] was archived as [[etherpadbackup:cuny-wikiwed-2025-05]] by [[User:Pharos|Pharos]] * 16:04 [[etherpad:p/cuny-wikiwed-2025-04]] was archived as [[etherpadbackup:cuny-wikiwed-2025-04]] by [[User:Pharos|Pharos]] === 2026-05-10 === * 18:10 [[etherpad:p/wmhackshowcase2020]] was archived as [[etherpadbackup:wmhackshowcase2020]] by [[User:BryanDavis|BryanDavis]] * 18:03 [[etherpad:p/wmh2023-Past,_Present_and_Future_of_Wikimedia]] was archived as [[etherpadbackup:wmh2023-Past,_Present_and_Future_of_Wikimedia]] by [[User:BryanDavis|BryanDavis]] * 18:01 [[etherpad:p/wdqs-500s]] was archived as [[etherpadbackup:wdqs-500s]] by [[User:BryanDavis|BryanDavis]] * 18:00 [[etherpad:p/wep_data_task_force_meeting_#1]] was archived as [[etherpadbackup:wep_data_task_force_meeting_#1]] by [[User:BryanDavis|BryanDavis]] === 2026-05-07 === * 00:35 [[etherpad:p/zh-ve-screenshots]] was archived as [[etherpadbackup:zh-ve-screenshots]] by [[User:BryanDavis|BryanDavis]] * 00:34 [[etherpad:p/zerocrash]] was archived as [[etherpadbackup:zerocrash]] by [[User:BryanDavis|BryanDavis]] * 00:34 [[etherpad:p/zero-mf-sync-up]] was archived as [[etherpadbackup:zero-mf-sync-up]] by [[User:BryanDavis|BryanDavis]] * 00:33 [[etherpad:p/yuvidump]] was archived as [[etherpadbackup:yuvidump]] by [[User:BryanDavis|BryanDavis]] * 00:33 [[etherpad:p/yuvi-knowledge-transfer]] was archived as [[etherpadbackup:yuvi-knowledge-transfer]] by [[User:BryanDavis|BryanDavis]] * 00:32 [[etherpad:p/yeZfmYkMu2]] was archived as [[etherpadbackup:yeZfmYkMu2]] by [[User:BryanDavis|BryanDavis]] * 00:32 [[etherpad:p/yAqOuwkDKuhwhwjvd44j]] was archived as [[etherpadbackup:yAqOuwkDKuhwhwjvd44j]] by [[User:BryanDavis|BryanDavis]] * 00:31 [[etherpad:p/y4gqz6<fe46Q]] was archived as [[etherpadbackup:y4gqz6<fe46Q]] by [[User:BryanDavis|BryanDavis]] * 00:31 [[etherpad:p/y4gqz6<fe46Q]] was archived as [[etherpadbackup:y4gqz6<fe46Q]] by [[User:BryanDavis|BryanDavis]] * 00:30 [[etherpad:p/xZLrPEASWV]] was archived as [[etherpadbackup:xZLrPEASWV]] by [[User:BryanDavis|BryanDavis]] * 00:30 [[etherpad:p/x-analytics]] was archived as [[etherpadbackup:x-analytics]] by [[User:BryanDavis|BryanDavis]] * 00:29 [[etherpad:p/wtf-is-production]] was archived as [[etherpadbackup:wtf-is-production]] by [[User:BryanDavis|BryanDavis]] * 00:29 [[etherpad:p/wtf]] was archived as [[etherpadbackup:wtf]] by [[User:BryanDavis|BryanDavis]] * 00:28 [[etherpad:p/wp-app-1-1-release-post]] was archived as [[etherpadbackup:wp-app-1-1-release-post]] by [[User:BryanDavis|BryanDavis]] * 00:28 [[etherpad:p/wmhack2021-phab]] was archived as [[etherpadbackup:wmhack2021-phab]] by [[User:BryanDavis|BryanDavis]] * 00:27 [[etherpad:p/wmhack2017-ciw]] was archived as [[etherpadbackup:wmhack2017-ciw]] by [[User:BryanDavis|BryanDavis]] * 00:26 [[etherpad:p/wmhack15-wikidata-architecture-session]] was archived as [[etherpadbackup:wmhack15-wikidata-architecture-session]] by [[User:BryanDavis|BryanDavis]] * 00:26 [[etherpad:p/wmh2024-leaving-coordination]] was archived as [[etherpadbackup:wmh2024-leaving-coordination]] by [[User:BryanDavis|BryanDavis]] * 00:25 [[etherpad:p/wmh2023-Toolhub,_Toolhunt,_and_the_quest]] was archived as [[etherpadbackup:wmh2023-Toolhub,_Toolhunt,_and_the_quest]] by [[User:BryanDavis|BryanDavis]] * 00:25 [[etherpad:p/wmh2023-Past,_Present_and_Future_of_Wikimedia]] was archived as [[etherpadbackup:wmh2023-Past,_Present_and_Future_of_Wikimedia]] by [[User:BryanDavis|BryanDavis]] * 00:24 [[etherpad:p/wmh2023-Overview_of_technical_areas_%26_projects]] was archived as [[etherpadbackup:wmh2023-Overview_of_technical_areas_%26_projects]] by [[User:BryanDavis|BryanDavis]] * 00:24 [[etherpad:p/wmh2023-LLMs,_ChatGPT,_machine_learning_tools,_etc]] was archived as [[etherpadbackup:wmh2023-LLMs,_ChatGPT,_machine_learning_tools,_etc]] by [[User:BryanDavis|BryanDavis]] * 00:23 [[etherpad:p/wmh2023-Best_practices_&_ideas_for_implementing]] was archived as [[etherpadbackup:wmh2023-Best_practices_&_ideas_for_implementing]] by [[User:BryanDavis|BryanDavis]] * 00:23 [[etherpad:p/wmf11and12]] was archived as [[etherpadbackup:wmf11and12]] by [[User:BryanDavis|BryanDavis]] * 00:22 [[etherpad:p/wmcslightning]] was archived as [[etherpadbackup:wmcslightning]] by [[User:BryanDavis|BryanDavis]] * 00:22 [[etherpad:p/wmcs_nfs_future]] was archived as [[etherpadbackup:wmcs_nfs_future]] by [[User:BryanDavis|BryanDavis]] * 00:21 [[etherpad:p/wmcs-vs-rowb-upgrade]] was archived as [[etherpadbackup:wmcs-vs-rowb-upgrade]] by [[User:BryanDavis|BryanDavis]] * 00:21 [[etherpad:p/wmcs-goals-q4-16-17]] was archived as [[etherpadbackup:wmcs-goals-q4-16-17]] by [[User:BryanDavis|BryanDavis]] * 00:20 [[etherpad:p/wmcs-goals-q2-17-18]] was archived as [[etherpadbackup:wmcs-goals-q2-17-18]] by [[User:BryanDavis|BryanDavis]] * 00:20 [[etherpad:p/wmcs-fallout]] was archived as [[etherpadbackup:wmcs-fallout]] by [[User:BryanDavis|BryanDavis]] * 00:19 [[etherpad:p/wmcs-cumin-fail]] was archived as [[etherpadbackup:wmcs-cumin-fail]] by [[User:BryanDavis|BryanDavis]] * 00:19 [[etherpad:p/wmcs-contributors]] was archived as [[etherpadbackup:wmcs-contributors]] by [[User:BryanDavis|BryanDavis]] * 00:18 [[etherpad:p/wmcs-2023-06-07]] was archived as [[etherpadbackup:wmcs-2023-06-07]] by [[User:BryanDavis|BryanDavis]] * 00:18 [[etherpad:p/wmconf2013-evaluating_programs]] was archived as [[etherpadbackup:wmconf2013-evaluating_programs]] by [[User:BryanDavis|BryanDavis]] * 00:17 [[etherpad:p/wmconf2013-evaluating-programs]] was archived as [[etherpadbackup:wmconf2013-evaluating-programs]] by [[User:BryanDavis|BryanDavis]] * 00:17 [[etherpad:p/wlmapp-iteration1-story-estimation-and-review]] was archived as [[etherpadbackup:wlmapp-iteration1-story-estimation-and-review]] by [[User:BryanDavis|BryanDavis]] * 00:16 [[etherpad:p/wlm-sprint5-retro]] was archived as [[etherpadbackup:wlm-sprint5-retro]] by [[User:BryanDavis|BryanDavis]] * 00:16 [[etherpad:p/wlm-sprint4-retro]] was archived as [[etherpadbackup:wlm-sprint4-retro]] by [[User:BryanDavis|BryanDavis]] * 00:15 [[etherpad:p/wlm-sprint3-retro]] was archived as [[etherpadbackup:wlm-sprint3-retro]] by [[User:BryanDavis|BryanDavis]] * 00:15 [[etherpad:p/wlm-sprint2-retro]] was archived as [[etherpadbackup:wlm-sprint2-retro]] by [[User:BryanDavis|BryanDavis]] * 00:14 [[etherpad:p/wlm-sprint1-retro]] was archived as [[etherpadbackup:wlm-sprint1-retro]] by [[User:BryanDavis|BryanDavis]] * 00:13 [[etherpad:p/wlm-project-retro]] was archived as [[etherpadbackup:wlm-project-retro]] by [[User:BryanDavis|BryanDavis]] * 00:13 [[etherpad:p/wlm-on-labs-steps]] was archived as [[etherpadbackup:wlm-on-labs-steps]] by [[User:BryanDavis|BryanDavis]] * 00:12 [[etherpad:p/wlm-api-admin-tree]] was archived as [[etherpadbackup:wlm-api-admin-tree]] by [[User:BryanDavis|BryanDavis]] * 00:12 [[etherpad:p/wishathon-march-2024-wishlist-overview-session]] was archived as [[etherpadbackup:wishathon-march-2024-wishlist-overview-session]] by [[User:BryanDavis|BryanDavis]] * 00:11 [[etherpad:p/wishathon-march-2024-showcase]] was archived as [[etherpadbackup:wishathon-march-2024-showcase]] by [[User:BryanDavis|BryanDavis]] * 00:11 [[etherpad:p/wishathon-march-2024-office-hour]] was archived as [[etherpadbackup:wishathon-march-2024-office-hour]] by [[User:BryanDavis|BryanDavis]] * 00:10 [[etherpad:p/wishathon-march-2024-fotw-session]] was archived as [[etherpadbackup:wishathon-march-2024-fotw-session]] by [[User:BryanDavis|BryanDavis]] * 00:10 [[etherpad:p/wikitechrebuild]] was archived as [[etherpadbackup:wikitechrebuild]] by [[User:BryanDavis|BryanDavis]] * 00:09 [[etherpad:p/wikitech-extensions]] was archived as [[etherpadbackup:wikitech-extensions]] by [[User:BryanDavis|BryanDavis]] * 00:09 [[etherpad:p/wikireplica-dns-switch]] was archived as [[etherpadbackup:wikireplica-dns-switch]] by [[User:BryanDavis|BryanDavis]] * 00:08 [[etherpad:p/wikireplica-beta]] was archived as [[etherpadbackup:wikireplica-beta]] by [[User:BryanDavis|BryanDavis]] * 00:08 [[etherpad:p/wikipedia-mobile-1-2-beta1-notes]] was archived as [[etherpadbackup:wikipedia-mobile-1-2-beta1-notes]] by [[User:BryanDavis|BryanDavis]] * 00:07 [[etherpad:p/wikipedia-ios-4.0.5-release-notes]] was archived as [[etherpadbackup:wikipedia-ios-4.0.5-release-notes]] by [[User:BryanDavis|BryanDavis]] * 00:07 [[etherpad:p/wikipedia-app-saved-pages-architecture]] was archived as [[etherpadbackup:wikipedia-app-saved-pages-architecture]] by [[User:BryanDavis|BryanDavis]] * 00:06 [[etherpad:p/wikipedia-app-ios-issues]] was archived as [[etherpadbackup:wikipedia-app-ios-issues]] by [[User:BryanDavis|BryanDavis]] * 00:06 [[etherpad:p/wikimania_knowledge]] was archived as [[etherpadbackup:wikimania_knowledge]] by [[User:BryanDavis|BryanDavis]] * 00:05 [[etherpad:p/wikigroknextsteps]] was archived as [[etherpadbackup:wikigroknextsteps]] by [[User:BryanDavis|BryanDavis]] * 00:05 [[etherpad:p/wikigrokcollections]] was archived as [[etherpadbackup:wikigrokcollections]] by [[User:BryanDavis|BryanDavis]] * 00:04 [[etherpad:p/wikidev17-devwishlist]] was archived as [[etherpadbackup:wikidev17-devwishlist]] by [[User:BryanDavis|BryanDavis]] * 00:04 [[etherpad:p/wikidata_query_service_checkin]] was archived as [[etherpadbackup:wikidata_query_service_checkin]] by [[User:BryanDavis|BryanDavis]] * 00:03 [[etherpad:p/wikibits-undefault]] was archived as [[etherpadbackup:wikibits-undefault]] by [[User:BryanDavis|BryanDavis]] * 00:03 [[etherpad:p/wiki_technology_notes]] was archived as [[etherpadbackup:wiki_technology_notes]] by [[User:BryanDavis|BryanDavis]] * 00:02 [[etherpad:p/widgets-for-uploadwizard]] was archived as [[etherpadbackup:widgets-for-uploadwizard]] by [[User:BryanDavis|BryanDavis]] * 00:01 [[etherpad:p/whm2023-Overview_of_technical_areas_&_projects]] was archived as [[etherpadbackup:whm2023-Overview_of_technical_areas_&_projects]] by [[User:BryanDavis|BryanDavis]] * 00:01 [[etherpad:p/whats-left-in-toolforge-eqiad]] was archived as [[etherpadbackup:whats-left-in-toolforge-eqiad]] by [[User:BryanDavis|BryanDavis]] * 00:00 [[etherpad:p/what_is_left_in_tampa]] was archived as [[etherpadbackup:what_is_left_in_tampa]] by [[User:BryanDavis|BryanDavis]] * 00:00 [[etherpad:p/webservice-k8s-announce]] was archived as [[etherpadbackup:webservice-k8s-announce]] by [[User:BryanDavis|BryanDavis]] === 2026-05-06 === * 23:59 [[etherpad:p/webrequest-2021-Q4-loss]] was archived as [[etherpadbackup:webrequest-2021-Q4-loss]] by [[User:BryanDavis|BryanDavis]] * 23:59 [[etherpad:p/webreq_update]] was archived as [[etherpadbackup:webreq_update]] by [[User:BryanDavis|BryanDavis]] * 23:58 [[etherpad:p/we-really-love-nfs]] was archived as [[etherpadbackup:we-really-love-nfs]] by [[User:BryanDavis|BryanDavis]] * 23:58 [[etherpad:p/wdqs_graph_split_production_deploy]] was archived as [[etherpadbackup:wdqs_graph_split_production_deploy]] by [[User:BryanDavis|BryanDavis]] * 23:57 [[etherpad:p/wdqs-split-graph-rollout-plan]] was archived as [[etherpadbackup:wdqs-split-graph-rollout-plan]] by [[User:BryanDavis|BryanDavis]] * 23:57 [[etherpad:p/wdqs-2024-08-08]] was archived as [[etherpadbackup:wdqs-2024-08-08]] by [[User:BryanDavis|BryanDavis]] * 23:56 [[etherpad:p/wdio]] was archived as [[etherpadbackup:wdio]] by [[User:BryanDavis|BryanDavis]] * 23:56 [[etherpad:p/wcna_tools]] was archived as [[etherpadbackup:wcna_tools]] by [[User:BryanDavis|BryanDavis]] * 23:55 [[etherpad:p/wbq_test]] was archived as [[etherpadbackup:wbq_test]] by [[User:BryanDavis|BryanDavis]] * 23:55 [[etherpad:p/watchlist_id]] was archived as [[etherpadbackup:watchlist_id]] by [[User:BryanDavis|BryanDavis]] * 23:54 [[etherpad:p/wallaby-dev-magnum-install]] was archived as [[etherpadbackup:wallaby-dev-magnum-install]] by [[User:BryanDavis|BryanDavis]] * 23:54 [[etherpad:p/volans-tmp]] was archived as [[etherpadbackup:volans-tmp]] by [[User:BryanDavis|BryanDavis]] * 23:53 [[etherpad:p/vk-jumbo-cleanup]] was archived as [[etherpadbackup:vk-jumbo-cleanup]] by [[User:BryanDavis|BryanDavis]] * 23:53 [[etherpad:p/virt1006migrate]] was archived as [[etherpadbackup:virt1006migrate]] by [[User:BryanDavis|BryanDavis]] * 23:52 [[etherpad:p/vikassy]] was archived as [[etherpadbackup:vikassy]] by [[User:BryanDavis|BryanDavis]] * 23:52 [[etherpad:p/vikas_ve]] was archived as [[etherpadbackup:vikas_ve]] by [[User:BryanDavis|BryanDavis]] * 23:51 [[etherpad:p/vikas_vagrant]] was archived as [[etherpadbackup:vikas_vagrant]] by [[User:BryanDavis|BryanDavis]] * 23:51 [[etherpad:p/vikas-upload-error]] was archived as [[etherpadbackup:vikas-upload-error]] by [[User:BryanDavis|BryanDavis]] * 23:50 [[etherpad:p/vikas-ruby-style]] was archived as [[etherpadbackup:vikas-ruby-style]] by [[User:BryanDavis|BryanDavis]] * 23:49 [[etherpad:p/vikas-gerrit]] was archived as [[etherpadbackup:vikas-gerrit]] by [[User:BryanDavis|BryanDavis]] * 23:49 [[etherpad:p/vikas]] was archived as [[etherpadbackup:vikas]] by [[User:BryanDavis|BryanDavis]] * 23:48 [[etherpad:p/ve-future-thoughts-2015]] was archived as [[etherpadbackup:ve-future-thoughts-2015]] by [[User:BryanDavis|BryanDavis]] * 23:48 [[etherpad:p/ve-2016-05-05]] was archived as [[etherpadbackup:ve-2016-05-05]] by [[User:BryanDavis|BryanDavis]] * 23:47 [[etherpad:p/ve]] was archived as [[etherpadbackup:ve]] by [[User:BryanDavis|BryanDavis]] * 23:47 [[etherpad:p/varnishkafka_troubleshooting]] was archived as [[etherpadbackup:varnishkafka_troubleshooting]] by [[User:BryanDavis|BryanDavis]] * 23:46 [[etherpad:p/vandrew]] was archived as [[etherpadbackup:vandrew]] by [[User:BryanDavis|BryanDavis]] * 23:46 [[etherpad:p/uw-cucumber-sketch]] was archived as [[etherpadbackup:uw-cucumber-sketch]] by [[User:BryanDavis|BryanDavis]] * 23:45 [[etherpad:p/uw-bugday-2014-09-09]] was archived as [[etherpadbackup:uw-bugday-2014-09-09]] by [[User:BryanDavis|BryanDavis]] * 23:45 [[etherpad:p/uploadwizard-refactor-planning]] was archived as [[etherpadbackup:uploadwizard-refactor-planning]] by [[User:BryanDavis|BryanDavis]] * 23:44 [[etherpad:p/uploadwizard-browser-tests]] was archived as [[etherpadbackup:uploadwizard-browser-tests]] by [[User:BryanDavis|BryanDavis]] * 23:44 [[etherpad:p/uploadcampaigns-squids]] was archived as [[etherpadbackup:uploadcampaigns-squids]] by [[User:BryanDavis|BryanDavis]] * 23:43 [[etherpad:p/upload-tool-class-structure]] was archived as [[etherpadbackup:upload-tool-class-structure]] by [[User:BryanDavis|BryanDavis]] * 23:43 [[etherpad:p/unsophabricated]] was archived as [[etherpadbackup:unsophabricated]] by [[User:BryanDavis|BryanDavis]] * 23:42 [[etherpad:p/unittestingbrownbag]] was archived as [[etherpadbackup:unittestingbrownbag]] by [[User:BryanDavis|BryanDavis]] * 23:42 [[etherpad:p/unified]] was archived as [[etherpadbackup:unified]] by [[User:BryanDavis|BryanDavis]] * 23:41 [[etherpad:p/unaliased]] was archived as [[etherpadbackup:unaliased]] by [[User:BryanDavis|BryanDavis]] * 23:41 [[etherpad:p/unable_to_move_translated_page]] was archived as [[etherpadbackup:unable_to_move_translated_page]] by [[User:BryanDavis|BryanDavis]] * 23:40 [[etherpad:p/twocolconflictdeploy]] was archived as [[etherpadbackup:twocolconflictdeploy]] by [[User:BryanDavis|BryanDavis]] * 23:40 [[etherpad:p/twiterbot-debuging]] was archived as [[etherpadbackup:twiterbot-debuging]] by [[User:BryanDavis|BryanDavis]] * 23:39 [[etherpad:p/trusty]] was archived as [[etherpadbackup:trusty]] by [[User:BryanDavis|BryanDavis]] * 23:39 [[etherpad:p/trixie-announcement]] was archived as [[etherpadbackup:trixie-announcement]] by [[User:BryanDavis|BryanDavis]] * 23:38 [[etherpad:p/trend-prod]] was archived as [[etherpadbackup:trend-prod]] by [[User:BryanDavis|BryanDavis]] * 23:38 [[etherpad:p/trebuchet-propose]] was archived as [[etherpadbackup:trebuchet-propose]] by [[User:BryanDavis|BryanDavis]] * 23:37 [[etherpad:p/toomanythings]] was archived as [[etherpadbackup:toomanythings]] by [[User:BryanDavis|BryanDavis]] * 23:36 [[etherpad:p/toolssupport]] was archived as [[etherpadbackup:toolssupport]] by [[User:BryanDavis|BryanDavis]] * 23:36 [[etherpad:p/toolsdb-upgrade]] was archived as [[etherpadbackup:toolsdb-upgrade]] by [[User:BryanDavis|BryanDavis]] * 23:35 [[etherpad:p/toolsdb-10.6]] was archived as [[etherpadbackup:toolsdb-10.6]] by [[User:BryanDavis|BryanDavis]] * 23:35 [[etherpad:p/tools-trusty-move]] was archived as [[etherpadbackup:tools-trusty-move]] by [[User:BryanDavis|BryanDavis]] * 23:34 [[etherpad:p/tools-the-great-recompress]] was archived as [[etherpadbackup:tools-the-great-recompress]] by [[User:BryanDavis|BryanDavis]] * 23:34 [[etherpad:p/tools-reboots-cve-0728]] was archived as [[etherpadbackup:tools-reboots-cve-0728]] by [[User:BryanDavis|BryanDavis]] * 23:33 [[etherpad:p/tools-reboots]] was archived as [[etherpadbackup:tools-reboots]] by [[User:BryanDavis|BryanDavis]] * 23:33 [[etherpad:p/tools-puppet-prefixes]] was archived as [[etherpadbackup:tools-puppet-prefixes]] by [[User:BryanDavis|BryanDavis]] * 23:32 [[etherpad:p/tools-migration-nodes]] was archived as [[etherpadbackup:tools-migration-nodes]] by [[User:BryanDavis|BryanDavis]] * 23:32 [[etherpad:p/tools-migration]] was archived as [[etherpadbackup:tools-migration]] by [[User:BryanDavis|BryanDavis]] * 23:31 [[etherpad:p/tools-kubernetes-scenarios]] was archived as [[etherpadbackup:tools-kubernetes-scenarios]] by [[User:BryanDavis|BryanDavis]] * 23:31 [[etherpad:p/tools-k8s-chat-01]] was archived as [[etherpadbackup:tools-k8s-chat-01]] by [[User:BryanDavis|BryanDavis]] * 23:30 [[etherpad:p/toollabs20160127]] was archived as [[etherpadbackup:toollabs20160127]] by [[User:BryanDavis|BryanDavis]] * 23:30 [[etherpad:p/toollabs-webservice-trusty-switch]] was archived as [[etherpadbackup:toollabs-webservice-trusty-switch]] by [[User:BryanDavis|BryanDavis]] * 23:29 [[etherpad:p/toollabs-uwsgi]] was archived as [[etherpadbackup:toollabs-uwsgi]] by [[User:BryanDavis|BryanDavis]] * 23:29 [[etherpad:p/toollabs-quarter-goal-q4-2014-15]] was archived as [[etherpadbackup:toollabs-quarter-goal-q4-2014-15]] by [[User:BryanDavis|BryanDavis]] * 23:28 [[etherpad:p/toollabs-impact]] was archived as [[etherpadbackup:toollabs-impact]] by [[User:BryanDavis|BryanDavis]] * 23:28 [[etherpad:p/toollabs-ideal-getting-started]] was archived as [[etherpadbackup:toollabs-ideal-getting-started]] by [[User:BryanDavis|BryanDavis]] * 23:27 [[etherpad:p/toollabs-cdnjs]] was archived as [[etherpadbackup:toollabs-cdnjs]] by [[User:BryanDavis|BryanDavis]] * 23:27 [[etherpad:p/toolforge-reboots]] was archived as [[etherpadbackup:toolforge-reboots]] by [[User:BryanDavis|BryanDavis]] * 23:26 [[etherpad:p/toolforge-php72-webservice]] was archived as [[etherpadbackup:toolforge-php72-webservice]] by [[User:BryanDavis|BryanDavis]] * 23:25 [[etherpad:p/toolforge-k8s-upgrade-1.23]] was archived as [[etherpadbackup:toolforge-k8s-upgrade-1.23]] by [[User:BryanDavis|BryanDavis]] * 23:25 [[etherpad:p/toolforge-k8s-1.31]] was archived as [[etherpadbackup:toolforge-k8s-1.31]] by [[User:BryanDavis|BryanDavis]] * 23:24 [[etherpad:p/toolforge-k8s-1.28-to-1.29-upgrade]] was archived as [[etherpadbackup:toolforge-k8s-1.28-to-1.29-upgrade]] by [[User:BryanDavis|BryanDavis]] * 23:24 [[etherpad:p/toolforge-2021-12-14]] was archived as [[etherpadbackup:toolforge-2021-12-14]] by [[User:BryanDavis|BryanDavis]] * 23:23 [[etherpad:p/tool_labs-doc]] was archived as [[etherpadbackup:tool_labs-doc]] by [[User:BryanDavis|BryanDavis]] * 23:23 [[etherpad:p/tonythomas01]] was archived as [[etherpadbackup:tonythomas01]] by [[User:BryanDavis|BryanDavis]] * 23:22 [[etherpad:p/tofu]] was archived as [[etherpadbackup:tofu]] by [[User:BryanDavis|BryanDavis]] * 23:22 [[etherpad:p/todays-regressions]] was archived as [[etherpadbackup:todays-regressions]] by [[User:BryanDavis|BryanDavis]] * 23:21 [[etherpad:p/title_value]] was archived as [[etherpadbackup:title_value]] by [[User:BryanDavis|BryanDavis]] * 23:21 [[etherpad:p/things_to_restart_after_rabbit_explosion]] was archived as [[etherpadbackup:things_to_restart_after_rabbit_explosion]] by [[User:BryanDavis|BryanDavis]] * 06:03 [[etherpad:p/test]] was archived as [[etherpadbackup:test]] by [[User:BryanDavis|BryanDavis]] * 06:03 [[etherpad:p/termdata]] was archived as [[etherpadbackup:termdata]] by [[User:BryanDavis|BryanDavis]] * 06:02 [[etherpad:p/temp]] was archived as [[etherpadbackup:temp]] by [[User:BryanDavis|BryanDavis]] * 06:02 [[etherpad:p/teach-yuvi-to-walk]] was archived as [[etherpadbackup:teach-yuvi-to-walk]] by [[User:BryanDavis|BryanDavis]] * 06:01 [[etherpad:p/tcinterviewquestions]] was archived as [[etherpadbackup:tcinterviewquestions]] by [[User:BryanDavis|BryanDavis]] * 06:00 [[etherpad:p/tallinntransportation]] was archived as [[etherpadbackup:tallinntransportation]] by [[User:BryanDavis|BryanDavis]] * 06:00 [[etherpad:p/tabsinpuppet]] was archived as [[etherpadbackup:tabsinpuppet]] by [[User:BryanDavis|BryanDavis]] * 05:59 [[etherpad:p/tFyLAurf0Z]] was archived as [[etherpadbackup:tFyLAurf0Z]] by [[User:BryanDavis|BryanDavis]] * 05:59 [[etherpad:p/switchover]] was archived as [[etherpadbackup:switchover]] by [[User:BryanDavis|BryanDavis]] * 05:58 [[etherpad:p/swiftfail]] was archived as [[etherpadbackup:swiftfail]] by [[User:BryanDavis|BryanDavis]] * 05:58 [[etherpad:p/swhYPuS04T]] was archived as [[etherpadbackup:swhYPuS04T]] by [[User:BryanDavis|BryanDavis]] * 05:57 [[etherpad:p/supersetdeprecation]] was archived as [[etherpadbackup:supersetdeprecation]] by [[User:BryanDavis|BryanDavis]] * 05:57 [[etherpad:p/sulf]] was archived as [[etherpadbackup:sulf]] by [[User:BryanDavis|BryanDavis]] * 05:56 [[etherpad:p/styleguide]] was archived as [[etherpadbackup:styleguide]] by [[User:BryanDavis|BryanDavis]] * 05:56 [[etherpad:p/stuffs]] was archived as [[etherpadbackup:stuffs]] by [[User:BryanDavis|BryanDavis]] * 05:55 [[etherpad:p/stubsExpansion]] was archived as [[etherpadbackup:stubsExpansion]] by [[User:BryanDavis|BryanDavis]] * 05:55 [[etherpad:p/striker-prod-deploy]] was archived as [[etherpadbackup:striker-prod-deploy]] by [[User:BryanDavis|BryanDavis]] * 05:54 [[etherpad:p/striker-gitlab-20220906]] was archived as [[etherpadbackup:striker-gitlab-20220906]] by [[User:BryanDavis|BryanDavis]] * 05:54 [[etherpad:p/striker]] was archived as [[etherpadbackup:striker]] by [[User:BryanDavis|BryanDavis]] * 05:53 [[etherpad:p/streaming_updater_cutover]] was archived as [[etherpadbackup:streaming_updater_cutover]] by [[User:BryanDavis|BryanDavis]] * 05:53 [[etherpad:p/streaming-wdqs]] was archived as [[etherpadbackup:streaming-wdqs]] by [[User:BryanDavis|BryanDavis]] * 05:52 [[etherpad:p/stillnopms]] was archived as [[etherpadbackup:stillnopms]] by [[User:BryanDavis|BryanDavis]] * 05:52 [[etherpad:p/state_of_union_-_if_-network_bullet_points]] was archived as [[etherpadbackup:state_of_union_-_if_-network_bullet_points]] by [[User:BryanDavis|BryanDavis]] * 05:51 [[etherpad:p/stat1_accounts]] was archived as [[etherpadbackup:stat1_accounts]] by [[User:BryanDavis|BryanDavis]] * 05:51 [[etherpad:p/stat-analytics-vlan]] was archived as [[etherpadbackup:stat-analytics-vlan]] by [[User:BryanDavis|BryanDavis]] * 05:50 [[etherpad:p/standalone_masters]] was archived as [[etherpadbackup:standalone_masters]] by [[User:BryanDavis|BryanDavis]] * 05:50 [[etherpad:p/ssl]] was archived as [[etherpadbackup:ssl]] by [[User:BryanDavis|BryanDavis]] * 05:49 [[etherpad:p/ssh-key-change-trixie]] was archived as [[etherpadbackup:ssh-key-change-trixie]] by [[User:BryanDavis|BryanDavis]] * 05:48 [[etherpad:p/spark_pair_coding]] was archived as [[etherpadbackup:spark_pair_coding]] by [[User:BryanDavis|BryanDavis]] * 05:48 [[etherpad:p/spark2-planning]] was archived as [[etherpadbackup:spark2-planning]] by [[User:BryanDavis|BryanDavis]] * 05:47 [[etherpad:p/so1920.cs.unibo.it]] was archived as [[etherpadbackup:so1920.cs.unibo.it]] by [[User:BryanDavis|BryanDavis]] * 05:47 [[etherpad:p/sistersearch-AB-test]] was archived as [[etherpadbackup:sistersearch-AB-test]] by [[User:BryanDavis|BryanDavis]] * 05:46 [[etherpad:p/single-edit-tab]] was archived as [[etherpadbackup:single-edit-tab]] by [[User:BryanDavis|BryanDavis]] * 05:46 [[etherpad:p/shit_tim_says]] was archived as [[etherpadbackup:shit_tim_says]] by [[User:BryanDavis|BryanDavis]] * 05:45 [[etherpad:p/setting_up_ORES_lab_cluster]] was archived as [[etherpadbackup:setting_up_ORES_lab_cluster]] by [[User:BryanDavis|BryanDavis]] * 05:45 [[etherpad:p/sendBeacon-2014-10-03]] was archived as [[etherpadbackup:sendBeacon-2014-10-03]] by [[User:BryanDavis|BryanDavis]] * 05:44 [[etherpad:p/selenium]] was archived as [[etherpadbackup:selenium]] by [[User:BryanDavis|BryanDavis]] * 05:44 [[etherpad:p/security_group_tests]] was archived as [[etherpadbackup:security_group_tests]] by [[User:BryanDavis|BryanDavis]] * 05:43 [[etherpad:p/search_enhancements]] was archived as [[etherpadbackup:search_enhancements]] by [[User:BryanDavis|BryanDavis]] * 05:43 [[etherpad:p/search-logging]] was archived as [[etherpadbackup:search-logging]] by [[User:BryanDavis|BryanDavis]] * 05:42 [[etherpad:p/search-hypotesis-24-25]] was archived as [[etherpadbackup:search-hypotesis-24-25]] by [[User:BryanDavis|BryanDavis]] * 05:42 [[etherpad:p/search-503s]] was archived as [[etherpadbackup:search-503s]] by [[User:BryanDavis|BryanDavis]] * 05:41 [[etherpad:p/scrum-of-scrums]] was archived as [[etherpadbackup:scrum-of-scrums]] by [[User:BryanDavis|BryanDavis]] * 05:41 [[etherpad:p/scoping-troubles]] was archived as [[etherpadbackup:scoping-troubles]] by [[User:BryanDavis|BryanDavis]] * 05:40 [[etherpad:p/scap-commands]] was archived as [[etherpadbackup:scap-commands]] by [[User:BryanDavis|BryanDavis]] * 05:40 [[etherpad:p/scap-201617-q2]] was archived as [[etherpadbackup:scap-201617-q2]] by [[User:BryanDavis|BryanDavis]] * 05:39 [[etherpad:p/rubocop-2015-02-10]] was archived as [[etherpadbackup:rubocop-2015-02-10]] by [[User:BryanDavis|BryanDavis]] * 05:39 [[etherpad:p/rsyncnova]] was archived as [[etherpadbackup:rsyncnova]] by [[User:BryanDavis|BryanDavis]] * 05:38 [[etherpad:p/rollout-ldap-block-change]] was archived as [[etherpadbackup:rollout-ldap-block-change]] by [[User:BryanDavis|BryanDavis]] * 05:38 [[etherpad:p/role-mediainfo]] was archived as [[etherpadbackup:role-mediainfo]] by [[User:BryanDavis|BryanDavis]] * 05:37 [[etherpad:p/rockyupgrade]] was archived as [[etherpadbackup:rockyupgrade]] by [[User:BryanDavis|BryanDavis]] * 05:37 [[etherpad:p/robots.txt]] was archived as [[etherpadbackup:robots.txt]] by [[User:BryanDavis|BryanDavis]] * 05:36 [[etherpad:p/rmrebootsforcephmons]] was archived as [[etherpadbackup:rmrebootsforcephmons]] by [[User:BryanDavis|BryanDavis]] * 05:35 [[etherpad:p/revscoring_user_oriented]] was archived as [[etherpadbackup:revscoring_user_oriented]] by [[User:BryanDavis|BryanDavis]] * 05:35 [[etherpad:p/revscoring_process]] was archived as [[etherpadbackup:revscoring_process]] by [[User:BryanDavis|BryanDavis]] * 05:34 [[etherpad:p/revscoring_languages]] was archived as [[etherpadbackup:revscoring_languages]] by [[User:BryanDavis|BryanDavis]] * 05:34 [[etherpad:p/revscoring_hackathon]] was archived as [[etherpadbackup:revscoring_hackathon]] by [[User:BryanDavis|BryanDavis]] * 05:33 [[etherpad:p/review-workflow-yuvi]] was archived as [[etherpadbackup:review-workflow-yuvi]] by [[User:BryanDavis|BryanDavis]] * 05:33 [[etherpad:p/resurrect_omega]] was archived as [[etherpadbackup:resurrect_omega]] by [[User:BryanDavis|BryanDavis]] * 05:32 [[etherpad:p/resthack]] was archived as [[etherpadbackup:resthack]] by [[User:BryanDavis|BryanDavis]] * 05:32 [[etherpad:p/restbase-parsoid-storage-rollout]] was archived as [[etherpadbackup:restbase-parsoid-storage-rollout]] by [[User:BryanDavis|BryanDavis]] * 05:31 [[etherpad:p/researchers_in_our_midst]] was archived as [[etherpadbackup:researchers_in_our_midst]] by [[User:BryanDavis|BryanDavis]] * 05:31 [[etherpad:p/research_showcase_dec14]] was archived as [[etherpadbackup:research_showcase_dec14]] by [[User:BryanDavis|BryanDavis]] * 05:30 [[etherpad:p/replicas-temp-accounts]] was archived as [[etherpadbackup:replicas-temp-accounts]] by [[User:BryanDavis|BryanDavis]] * 05:30 [[etherpad:p/rep-lag-s8]] was archived as [[etherpadbackup:rep-lag-s8]] by [[User:BryanDavis|BryanDavis]] * 05:29 [[etherpad:p/remaining-ldap]] was archived as [[etherpadbackup:remaining-ldap]] by [[User:BryanDavis|BryanDavis]] * 05:29 [[etherpad:p/refinery_deploy]] was archived as [[etherpadbackup:refinery_deploy]] by [[User:BryanDavis|BryanDavis]] * 05:28 [[etherpad:p/red-links]] was archived as [[etherpadbackup:red-links]] by [[User:BryanDavis|BryanDavis]] * 05:28 [[etherpad:p/realm_networks]] was archived as [[etherpadbackup:realm_networks]] by [[User:BryanDavis|BryanDavis]] * 05:27 [[etherpad:p/reading-infra-2015-16-q4]] was archived as [[etherpadbackup:reading-infra-2015-16-q4]] by [[User:BryanDavis|BryanDavis]] * 05:27 [[etherpad:p/rdf-flink-k8s]] was archived as [[etherpadbackup:rdf-flink-k8s]] by [[User:BryanDavis|BryanDavis]] * 05:26 [[etherpad:p/rdbmess-2021]] was archived as [[etherpadbackup:rdbmess-2021]] by [[User:BryanDavis|BryanDavis]] * 05:26 [[etherpad:p/rcstream-misc]] was archived as [[etherpadbackup:rcstream-misc]] by [[User:BryanDavis|BryanDavis]] * 05:25 [[etherpad:p/rc1-release-email]] was archived as [[etherpadbackup:rc1-release-email]] by [[User:BryanDavis|BryanDavis]] * 05:25 [[etherpad:p/rc.2-announce]] was archived as [[etherpadbackup:rc.2-announce]] by [[User:BryanDavis|BryanDavis]] * 05:24 [[etherpad:p/rar]] was archived as [[etherpadbackup:rar]] by [[User:BryanDavis|BryanDavis]] * 05:24 [[etherpad:p/rada12dubna]] was archived as [[etherpadbackup:rada12dubna]] by [[User:BryanDavis|BryanDavis]] * 05:23 [[etherpad:p/r.d2e7710af5a20eefd93babda3f38929a]] was archived as [[etherpadbackup:r.d2e7710af5a20eefd93babda3f38929a]] by [[User:BryanDavis|BryanDavis]] * 05:22 [[etherpad:p/r.B0XlRhOdKRWT6xuH]] was archived as [[etherpadbackup:r.B0XlRhOdKRWT6xuH]] by [[User:BryanDavis|BryanDavis]] * 05:22 [[etherpad:p/quicksurveys]] was archived as [[etherpadbackup:quicksurveys]] by [[User:BryanDavis|BryanDavis]] * 05:21 [[etherpad:p/quibble-0.0.36]] was archived as [[etherpadbackup:quibble-0.0.36]] by [[User:BryanDavis|BryanDavis]] * 05:21 [[etherpad:p/quarry-wdqs-integration]] was archived as [[etherpadbackup:quarry-wdqs-integration]] by [[User:BryanDavis|BryanDavis]] * 05:20 [[etherpad:p/quarry-for-pwb-scripts]] was archived as [[etherpadbackup:quarry-for-pwb-scripts]] by [[User:BryanDavis|BryanDavis]] * 05:20 [[etherpad:p/quarry-announce-email]] was archived as [[etherpadbackup:quarry-announce-email]] by [[User:BryanDavis|BryanDavis]] * 05:19 [[etherpad:p/q_planning_retro]] was archived as [[etherpadbackup:q_planning_retro]] by [[User:BryanDavis|BryanDavis]] * 05:19 [[etherpad:p/q4planningretro]] was archived as [[etherpadbackup:q4planningretro]] by [[User:BryanDavis|BryanDavis]] * 05:18 [[etherpad:p/q4planningnotes]] was archived as [[etherpadbackup:q4planningnotes]] by [[User:BryanDavis|BryanDavis]] * 05:18 [[etherpad:p/q4_checkin]] was archived as [[etherpadbackup:q4_checkin]] by [[User:BryanDavis|BryanDavis]] * 05:17 [[etherpad:p/q2_eng_goal]] was archived as [[etherpadbackup:q2_eng_goal]] by [[User:BryanDavis|BryanDavis]] * 05:17 [[etherpad:p/python_mediawiki_devs]] was archived as [[etherpadbackup:python_mediawiki_devs]] by [[User:BryanDavis|BryanDavis]] * 05:16 [[etherpad:p/python]] was archived as [[etherpadbackup:python]] by [[User:BryanDavis|BryanDavis]] * 05:16 [[etherpad:p/push-to-deploy-beta-announce]] was archived as [[etherpadbackup:push-to-deploy-beta-announce]] by [[User:BryanDavis|BryanDavis]] * 05:15 [[etherpad:p/puppetswat]] was archived as [[etherpadbackup:puppetswat]] by [[User:BryanDavis|BryanDavis]] * 05:15 [[etherpad:p/puppetmaster-enc]] was archived as [[etherpadbackup:puppetmaster-enc]] by [[User:BryanDavis|BryanDavis]] * 05:14 [[etherpad:p/puppet_upgrade_vms]] was archived as [[etherpadbackup:puppet_upgrade_vms]] by [[User:BryanDavis|BryanDavis]] * 05:14 [[etherpad:p/puppet3]] was archived as [[etherpadbackup:puppet3]] by [[User:BryanDavis|BryanDavis]] * 05:13 [[etherpad:p/publish-new-mobile-report]] was archived as [[etherpadbackup:publish-new-mobile-report]] by [[User:BryanDavis|BryanDavis]] * 05:13 [[etherpad:p/proxy-buster]] was archived as [[etherpadbackup:proxy-buster]] by [[User:BryanDavis|BryanDavis]] * 05:12 [[etherpad:p/preciseonlabs]] was archived as [[etherpadbackup:preciseonlabs]] by [[User:BryanDavis|BryanDavis]] * 05:12 [[etherpad:p/precise-tools]] was archived as [[etherpadbackup:precise-tools]] by [[User:BryanDavis|BryanDavis]] * 05:11 [[etherpad:p/poem]] was archived as [[etherpadbackup:poem]] by [[User:BryanDavis|BryanDavis]] * 05:11 [[etherpad:p/play_store_what's_new]] was archived as [[etherpadbackup:play_store_what's_new]] by [[User:BryanDavis|BryanDavis]] * 05:10 [[etherpad:p/pii_pageviews]] was archived as [[etherpadbackup:pii_pageviews]] by [[User:BryanDavis|BryanDavis]] * 05:09 [[etherpad:p/phab_roles]] was archived as [[etherpadbackup:phab_roles]] by [[User:BryanDavis|BryanDavis]] * 05:09 [[etherpad:p/phab-2016-02-03]] was archived as [[etherpadbackup:phab-2016-02-03]] by [[User:BryanDavis|BryanDavis]] * 05:08 [[etherpad:p/phab]] was archived as [[etherpadbackup:phab]] by [[User:BryanDavis|BryanDavis]] * 05:08 [[etherpad:p/performance-team-weekly-update]] was archived as [[etherpadbackup:performance-team-weekly-update]] by [[User:BryanDavis|BryanDavis]] * 05:07 [[etherpad:p/performance-k8s]] was archived as [[etherpadbackup:performance-k8s]] by [[User:BryanDavis|BryanDavis]] * 05:07 [[etherpad:p/perfnotice]] was archived as [[etherpadbackup:perfnotice]] by [[User:BryanDavis|BryanDavis]] * 05:06 [[etherpad:p/perf-announcements]] was archived as [[etherpadbackup:perf-announcements]] by [[User:BryanDavis|BryanDavis]] * 05:06 [[etherpad:p/perf-20160223]] was archived as [[etherpadbackup:perf-20160223]] by [[User:BryanDavis|BryanDavis]] * 05:05 [[etherpad:p/paws-tools]] was archived as [[etherpadbackup:paws-tools]] by [[User:BryanDavis|BryanDavis]] * 05:05 [[etherpad:p/paws-public-url-structure]] was archived as [[etherpadbackup:paws-public-url-structure]] by [[User:BryanDavis|BryanDavis]] * 05:04 [[etherpad:p/paste]] was archived as [[etherpadbackup:paste]] by [[User:BryanDavis|BryanDavis]] * 05:04 [[etherpad:p/parsoid_docs]] was archived as [[etherpadbackup:parsoid_docs]] by [[User:BryanDavis|BryanDavis]] * 05:03 [[etherpad:p/parsoid-2019-01-16]] was archived as [[etherpadbackup:parsoid-2019-01-16]] by [[User:BryanDavis|BryanDavis]] * 05:03 [[etherpad:p/parsing_team_meeting_archive]] was archived as [[etherpadbackup:parsing_team_meeting_archive]] by [[User:BryanDavis|BryanDavis]] * 05:02 [[etherpad:p/parser-api-2018]] was archived as [[etherpadbackup:parser-api-2018]] by [[User:BryanDavis|BryanDavis]] * 05:02 [[etherpad:p/pageview_api_nodes]] was archived as [[etherpadbackup:pageview_api_nodes]] by [[User:BryanDavis|BryanDavis]] * 05:01 [[etherpad:p/pages-needing-investigation]] was archived as [[etherpadbackup:pages-needing-investigation]] by [[User:BryanDavis|BryanDavis]] * 05:01 [[etherpad:p/page-styling-zurich]] was archived as [[etherpadbackup:page-styling-zurich]] by [[User:BryanDavis|BryanDavis]] * 05:00 [[etherpad:p/otrs-migration]] was archived as [[etherpadbackup:otrs-migration]] by [[User:BryanDavis|BryanDavis]] * 05:00 [[etherpad:p/ores_task_tracking]] was archived as [[etherpadbackup:ores_task_tracking]] by [[User:BryanDavis|BryanDavis]] * 04:59 [[etherpad:p/ores]] was archived as [[etherpadbackup:ores]] by [[User:BryanDavis|BryanDavis]] * 04:58 [[etherpad:p/ops-offsite-discussions]] was archived as [[etherpadbackup:ops-offsite-discussions]] by [[User:BryanDavis|BryanDavis]] * 04:58 [[etherpad:p/ops-offsite-2016-lightning-talks]] was archived as [[etherpadbackup:ops-offsite-2016-lightning-talks]] by [[User:BryanDavis|BryanDavis]] * 04:57 [[etherpad:p/ops-offsite-2016-labstore-future]] was archived as [[etherpadbackup:ops-offsite-2016-labstore-future]] by [[User:BryanDavis|BryanDavis]] * 04:57 [[etherpad:p/opensearch-opportunities-for-improvement]] was archived as [[etherpadbackup:opensearch-opportunities-for-improvement]] by [[User:BryanDavis|BryanDavis]] * 04:56 [[etherpad:p/opendj-migration]] was archived as [[etherpadbackup:opendj-migration]] by [[User:BryanDavis|BryanDavis]] * 04:56 [[etherpad:p/open_infra_workshop]] was archived as [[etherpadbackup:open_infra_workshop]] by [[User:BryanDavis|BryanDavis]] * 04:55 [[etherpad:p/opcache-💩-2020-10-01]] was archived as [[etherpadbackup:opcache-💩-2020-10-01]] by [[User:BryanDavis|BryanDavis]] * 04:55 [[etherpad:p/oozie-restart]] was archived as [[etherpadbackup:oozie-restart]] by [[User:BryanDavis|BryanDavis]] * 04:54 [[etherpad:p/oozie]] was archived as [[etherpadbackup:oozie]] by [[User:BryanDavis|BryanDavis]] * 04:54 [[etherpad:p/omega-restarts]] was archived as [[etherpadbackup:omega-restarts]] by [[User:BryanDavis|BryanDavis]] * 04:53 [[etherpad:p/oldbastion]] was archived as [[etherpadbackup:oldbastion]] by [[User:BryanDavis|BryanDavis]] * 04:53 [[etherpad:p/okapi-dumps-2021-02-01]] was archived as [[etherpadbackup:okapi-dumps-2021-02-01]] by [[User:BryanDavis|BryanDavis]] * 04:52 [[etherpad:p/ocata-upgrade]] was archived as [[etherpadbackup:ocata-upgrade]] by [[User:BryanDavis|BryanDavis]] * 04:52 [[etherpad:p/o1Ub87jizX]] was archived as [[etherpadbackup:o1Ub87jizX]] by [[User:BryanDavis|BryanDavis]] * 04:51 [[etherpad:p/o]] was archived as [[etherpadbackup:o]] by [[User:BryanDavis|BryanDavis]] * 04:51 [[etherpad:p/nuria]] was archived as [[etherpadbackup:nuria]] by [[User:BryanDavis|BryanDavis]] * 04:50 [[etherpad:p/npp_discussion_wmhack17]] was archived as [[etherpadbackup:npp_discussion_wmhack17]] by [[User:BryanDavis|BryanDavis]] * 04:50 [[etherpad:p/nodes_with_a_public_IP]] was archived as [[etherpadbackup:nodes_with_a_public_IP]] by [[User:BryanDavis|BryanDavis]] * 04:49 [[etherpad:p/nodepool-migration]] was archived as [[etherpadbackup:nodepool-migration]] by [[User:BryanDavis|BryanDavis]] * 04:49 [[etherpad:p/night-mode-colors]] was archived as [[etherpadbackup:night-mode-colors]] by [[User:BryanDavis|BryanDavis]] * 04:48 [[etherpad:p/ng8EYRhQ7SIXy7cLBexZ]] was archived as [[etherpadbackup:ng8EYRhQ7SIXy7cLBexZ]] by [[User:BryanDavis|BryanDavis]] * 04:48 [[etherpad:p/nfsprojectchange]] was archived as [[etherpadbackup:nfsprojectchange]] by [[User:BryanDavis|BryanDavis]] * 04:47 [[etherpad:p/nfsmaint]] was archived as [[etherpadbackup:nfsmaint]] by [[User:BryanDavis|BryanDavis]] * 04:47 [[etherpad:p/nfs]] was archived as [[etherpadbackup:nfs]] by [[User:BryanDavis|BryanDavis]] * 04:46 [[etherpad:p/newton_upgrade_plan]] was archived as [[etherpadbackup:newton_upgrade_plan]] by [[User:BryanDavis|BryanDavis]] * 04:45 [[etherpad:p/newsletters-contenthandler]] was archived as [[etherpadbackup:newsletters-contenthandler]] by [[User:BryanDavis|BryanDavis]] * 04:45 [[etherpad:p/newpyter]] was archived as [[etherpadbackup:newpyter]] by [[User:BryanDavis|BryanDavis]] * 04:44 [[etherpad:p/new-webnode-toollabs]] was archived as [[etherpadbackup:new-webnode-toollabs]] by [[User:BryanDavis|BryanDavis]] * 04:44 [[etherpad:p/networkoutage]] was archived as [[etherpadbackup:networkoutage]] by [[User:BryanDavis|BryanDavis]] * 04:43 [[etherpad:p/netbox4-upgrade]] was archived as [[etherpadbackup:netbox4-upgrade]] by [[User:BryanDavis|BryanDavis]] * 04:43 [[etherpad:p/netbox-fail]] was archived as [[etherpadbackup:netbox-fail]] by [[User:BryanDavis|BryanDavis]] * 04:42 [[etherpad:p/naming-visualization]] was archived as [[etherpadbackup:naming-visualization]] by [[User:BryanDavis|BryanDavis]] * 04:42 [[etherpad:p/nTbAb5yPvBFATUylV7Vj]] was archived as [[etherpadbackup:nTbAb5yPvBFATUylV7Vj]] by [[User:BryanDavis|BryanDavis]] * 04:41 [[etherpad:p/mwtidy-checkerrors]] was archived as [[etherpadbackup:mwtidy-checkerrors]] by [[User:BryanDavis|BryanDavis]] * 04:41 [[etherpad:p/mwds15-spec-oriented-architecture]] was archived as [[etherpadbackup:mwds15-spec-oriented-architecture]] by [[User:BryanDavis|BryanDavis]] * 04:40 [[etherpad:p/mwds-profiling]] was archived as [[etherpadbackup:mwds-profiling]] by [[User:BryanDavis|BryanDavis]] * 04:40 [[etherpad:p/mw1-25-jsdeprecate]] was archived as [[etherpadbackup:mw1-25-jsdeprecate]] by [[User:BryanDavis|BryanDavis]] * 04:39 [[etherpad:p/mw-php70-tests]] was archived as [[etherpadbackup:mw-php70-tests]] by [[User:BryanDavis|BryanDavis]] * 04:39 [[etherpad:p/mvrelease]] was archived as [[etherpadbackup:mvrelease]] by [[User:BryanDavis|BryanDavis]] * 04:38 [[etherpad:p/multimedia-weekly-meeting-2015-12-24]] was archived as [[etherpadbackup:multimedia-weekly-meeting-2015-12-24]] by [[User:BryanDavis|BryanDavis]] * 04:38 [[etherpad:p/multimedia-weekly-meeting-2015-01-14]] was archived as [[etherpadbackup:multimedia-weekly-meeting-2015-01-14]] by [[User:BryanDavis|BryanDavis]] * 04:37 [[etherpad:p/multimedia-weekly-meeting-2014-01-06]] was archived as [[etherpadbackup:multimedia-weekly-meeting-2014-01-06]] by [[User:BryanDavis|BryanDavis]] * 04:37 [[etherpad:p/multimedia-mmv-technical-retrospective]] was archived as [[etherpadbackup:multimedia-mmv-technical-retrospective]] by [[User:BryanDavis|BryanDavis]] * 04:36 [[etherpad:p/multimedia-design-05-15-2014]] was archived as [[etherpadbackup:multimedia-design-05-15-2014]] by [[User:BryanDavis|BryanDavis]] * 04:36 [[etherpad:p/multimedia-analytics-queries]] was archived as [[etherpadbackup:multimedia-analytics-queries]] by [[User:BryanDavis|BryanDavis]] * 04:35 [[etherpad:p/morethan10g]] was archived as [[etherpadbackup:morethan10g]] by [[User:BryanDavis|BryanDavis]] * 04:35 [[etherpad:p/monitoring-lvs-l2]] was archived as [[etherpadbackup:monitoring-lvs-l2]] by [[User:BryanDavis|BryanDavis]] * 04:34 [[etherpad:p/mod-conf]] was archived as [[etherpadbackup:mod-conf]] by [[User:BryanDavis|BryanDavis]] * 04:34 [[etherpad:p/mobileweb-wsj]] was archived as [[etherpadbackup:mobileweb-wsj]] by [[User:BryanDavis|BryanDavis]] * 04:33 [[etherpad:p/mobilesprintname]] was archived as [[etherpadbackup:mobilesprintname]] by [[User:BryanDavis|BryanDavis]] * 04:33 [[etherpad:p/mobilenav-sprint2-retro]] was archived as [[etherpadbackup:mobilenav-sprint2-retro]] by [[User:BryanDavis|BryanDavis]] * 04:32 [[etherpad:p/mobilenav-sprint1-retro]] was archived as [[etherpadbackup:mobilenav-sprint1-retro]] by [[User:BryanDavis|BryanDavis]] * 04:32 [[etherpad:p/mobileappplanningQ32014]] was archived as [[etherpadbackup:mobileappplanningQ32014]] by [[User:BryanDavis|BryanDavis]] * 04:31 [[etherpad:p/mobile_web_q3]] was archived as [[etherpadbackup:mobile_web_q3]] by [[User:BryanDavis|BryanDavis]] * 04:30 [[etherpad:p/mobile_web_health_check]] was archived as [[etherpadbackup:mobile_web_health_check]] by [[User:BryanDavis|BryanDavis]] * 04:30 [[etherpad:p/mobile_web_design_review]] was archived as [[etherpadbackup:mobile_web_design_review]] by [[User:BryanDavis|BryanDavis]] * 04:29 [[etherpad:p/mobile_web_breakout]] was archived as [[etherpadbackup:mobile_web_breakout]] by [[User:BryanDavis|BryanDavis]] * 04:29 [[etherpad:p/mobile_apps_onboarding_brian]] was archived as [[etherpadbackup:mobile_apps_onboarding_brian]] by [[User:BryanDavis|BryanDavis]] * 04:28 [[etherpad:p/mobile-users-faq]] was archived as [[etherpadbackup:mobile-users-faq]] by [[User:BryanDavis|BryanDavis]] * 04:28 [[etherpad:p/mobile-tutorial-practice]] was archived as [[etherpadbackup:mobile-tutorial-practice]] by [[User:BryanDavis|BryanDavis]] * 04:27 [[etherpad:p/mobile-retrospective-q3-2013]] was archived as [[etherpadbackup:mobile-retrospective-q3-2013]] by [[User:BryanDavis|BryanDavis]] * 04:27 [[etherpad:p/mobile-retrospective-20130128]] was archived as [[etherpadbackup:mobile-retrospective-20130128]] by [[User:BryanDavis|BryanDavis]] * 04:26 [[etherpad:p/mobile-retrospective-20130114]] was archived as [[etherpadbackup:mobile-retrospective-20130114]] by [[User:BryanDavis|BryanDavis]] * 04:26 [[etherpad:p/mobile-retrospective-20121217]] was archived as [[etherpadbackup:mobile-retrospective-20121217]] by [[User:BryanDavis|BryanDavis]] * 04:25 [[etherpad:p/mobile-retrospective-20120102]] was archived as [[etherpadbackup:mobile-retrospective-20120102]] by [[User:BryanDavis|BryanDavis]] * 04:25 [[etherpad:p/mobile-q4-planning-2013]] was archived as [[etherpadbackup:mobile-q4-planning-2013]] by [[User:BryanDavis|BryanDavis]] * 04:24 [[etherpad:p/mobile-q3-planning-2013]] was archived as [[etherpadbackup:mobile-q3-planning-2013]] by [[User:BryanDavis|BryanDavis]] * 04:24 [[etherpad:p/mobile-ops-syncup-28feb2013]] was archived as [[etherpadbackup:mobile-ops-syncup-28feb2013]] by [[User:BryanDavis|BryanDavis]] * 04:23 [[etherpad:p/mobile-ime-support]] was archived as [[etherpadbackup:mobile-ime-support]] by [[User:BryanDavis|BryanDavis]] * 04:23 [[etherpad:p/mobile-campaigns]] was archived as [[etherpadbackup:mobile-campaigns]] by [[User:BryanDavis|BryanDavis]] * 04:22 [[etherpad:p/mobile-app-workflow-el-1]] was archived as [[etherpadbackup:mobile-app-workflow-el-1]] by [[User:BryanDavis|BryanDavis]] * 04:22 [[etherpad:p/mobile-app-versions]] was archived as [[etherpadbackup:mobile-app-versions]] by [[User:BryanDavis|BryanDavis]] * 04:21 [[etherpad:p/mobile-app-otrs-feedback]] was archived as [[etherpadbackup:mobile-app-otrs-feedback]] by [[User:BryanDavis|BryanDavis]] * 04:21 [[etherpad:p/mobile-app-dashboard]] was archived as [[etherpadbackup:mobile-app-dashboard]] by [[User:BryanDavis|BryanDavis]] * 04:20 [[etherpad:p/mmv-text-size-changes]] was archived as [[etherpadbackup:mmv-text-size-changes]] by [[User:BryanDavis|BryanDavis]] * 04:20 [[etherpad:p/mmodell]] was archived as [[etherpadbackup:mmodell]] by [[User:BryanDavis|BryanDavis]] * 04:19 [[etherpad:p/mm-standup-notes]] was archived as [[etherpadbackup:mm-standup-notes]] by [[User:BryanDavis|BryanDavis]] * 04:19 [[etherpad:p/misc-migration]] was archived as [[etherpadbackup:misc-migration]] by [[User:BryanDavis|BryanDavis]] * 04:18 [[etherpad:p/migrate-k8s-etcd]] was archived as [[etherpadbackup:migrate-k8s-etcd]] by [[User:BryanDavis|BryanDavis]] * 04:18 [[etherpad:p/mf]] was archived as [[etherpadbackup:mf]] by [[User:BryanDavis|BryanDavis]] * 04:17 [[etherpad:p/mediawiki-vagrant]] was archived as [[etherpadbackup:mediawiki-vagrant]] by [[User:BryanDavis|BryanDavis]] * 04:16 [[etherpad:p/mazza]] was archived as [[etherpadbackup:mazza]] by [[User:BryanDavis|BryanDavis]] * 04:16 [[etherpad:p/mayankmadan]] was archived as [[etherpadbackup:mayankmadan]] by [[User:BryanDavis|BryanDavis]] * 04:15 [[etherpad:p/mailman-Aug-2015]] was archived as [[etherpadbackup:mailman-Aug-2015]] by [[User:BryanDavis|BryanDavis]] * 04:15 [[etherpad:p/magru_server_swaps]] was archived as [[etherpadbackup:magru_server_swaps]] by [[User:BryanDavis|BryanDavis]] * 04:14 [[etherpad:p/magnumusersfeb2026]] was archived as [[etherpadbackup:magnumusersfeb2026]] by [[User:BryanDavis|BryanDavis]] * 04:14 [[etherpad:p/mBBgYyJeBG]] was archived as [[etherpadbackup:mBBgYyJeBG]] by [[User:BryanDavis|BryanDavis]] * 04:13 [[etherpad:p/m1-service-owners]] was archived as [[etherpadbackup:m1-service-owners]] by [[User:BryanDavis|BryanDavis]] * 04:13 [[etherpad:p/lyon-service-template]] was archived as [[etherpadbackup:lyon-service-template]] by [[User:BryanDavis|BryanDavis]] * 04:12 [[etherpad:p/lucid_mini_hack]] was archived as [[etherpadbackup:lucid_mini_hack]] by [[User:BryanDavis|BryanDavis]] * 04:12 [[etherpad:p/lubaochuan]] was archived as [[etherpadbackup:lubaochuan]] by [[User:BryanDavis|BryanDavis]] * 04:11 [[etherpad:p/logchange]] was archived as [[etherpadbackup:logchange]] by [[User:BryanDavis|BryanDavis]] * 04:11 [[etherpad:p/linkpass]] was archived as [[etherpadbackup:linkpass]] by [[User:BryanDavis|BryanDavis]] * 04:10 [[etherpad:p/legal-text-upload-dialog]] was archived as [[etherpadbackup:legal-text-upload-dialog]] by [[User:BryanDavis|BryanDavis]] * 04:10 [[etherpad:p/language-and-localization]] was archived as [[etherpadbackup:language-and-localization]] by [[User:BryanDavis|BryanDavis]] * 04:09 [[etherpad:p/langaugescreenshot-upload-error]] was archived as [[etherpadbackup:langaugescreenshot-upload-error]] by [[User:BryanDavis|BryanDavis]] * 04:09 [[etherpad:p/lang-bar]] was archived as [[etherpadbackup:lang-bar]] by [[User:BryanDavis|BryanDavis]] * 04:08 [[etherpad:p/labwebneedsports]] was archived as [[etherpadbackup:labwebneedsports]] by [[User:BryanDavis|BryanDavis]] * 04:08 [[etherpad:p/labweb]] was archived as [[etherpadbackup:labweb]] by [[User:BryanDavis|BryanDavis]] * 04:07 [[etherpad:p/labvirt_reboots]] was archived as [[etherpadbackup:labvirt_reboots]] by [[User:BryanDavis|BryanDavis]] * 04:07 [[etherpad:p/labvirt1015]] was archived as [[etherpadbackup:labvirt1015]] by [[User:BryanDavis|BryanDavis]] * 04:06 [[etherpad:p/labvirt1003instances]] was archived as [[etherpadbackup:labvirt1003instances]] by [[User:BryanDavis|BryanDavis]] * 04:06 [[etherpad:p/labtes_mitaka_upgrade_notes]] was archived as [[etherpadbackup:labtes_mitaka_upgrade_notes]] by [[User:BryanDavis|BryanDavis]] * 04:05 [[etherpad:p/labstore10045-move]] was archived as [[etherpadbackup:labstore10045-move]] by [[User:BryanDavis|BryanDavis]] * 04:04 [[etherpad:p/labservices_failover]] was archived as [[etherpadbackup:labservices_failover]] by [[User:BryanDavis|BryanDavis]] * 04:04 [[etherpad:p/labsdb10067]] was archived as [[etherpadbackup:labsdb10067]] by [[User:BryanDavis|BryanDavis]] * 04:03 [[etherpad:p/labsdb-replication]] was archived as [[etherpadbackup:labsdb-replication]] by [[User:BryanDavis|BryanDavis]] * 04:03 [[etherpad:p/labsdb-goal]] was archived as [[etherpadbackup:labsdb-goal]] by [[User:BryanDavis|BryanDavis]] * 04:02 [[etherpad:p/labsdb-chat-july-18]] was archived as [[etherpadbackup:labsdb-chat-july-18]] by [[User:BryanDavis|BryanDavis]] * 04:02 [[etherpad:p/labs_migration]] was archived as [[etherpadbackup:labs_migration]] by [[User:BryanDavis|BryanDavis]] * 04:01 [[etherpad:p/labs-survey-interview-questions]] was archived as [[etherpadbackup:labs-survey-interview-questions]] by [[User:BryanDavis|BryanDavis]] * 04:01 [[etherpad:p/labs-report-q4-1]] was archived as [[etherpadbackup:labs-report-q4-1]] by [[User:BryanDavis|BryanDavis]] * 04:00 [[etherpad:p/labs-releng-k8s-confluence]] was archived as [[etherpadbackup:labs-releng-k8s-confluence]] by [[User:BryanDavis|BryanDavis]] * 04:00 [[etherpad:p/labs-nfs-20140413-nfsoutage]] was archived as [[etherpadbackup:labs-nfs-20140413-nfsoutage]] by [[User:BryanDavis|BryanDavis]] * 03:59 [[etherpad:p/labs-migration]] was archived as [[etherpadbackup:labs-migration]] by [[User:BryanDavis|BryanDavis]] * 03:59 [[etherpad:p/labs-meeting-apr-25-2016]] was archived as [[etherpadbackup:labs-meeting-apr-25-2016]] by [[User:BryanDavis|BryanDavis]] * 03:58 [[etherpad:p/labs-incident-timeline]] was archived as [[etherpadbackup:labs-incident-timeline]] by [[User:BryanDavis|BryanDavis]] * 03:58 [[etherpad:p/labs-cleanup-2015]] was archived as [[etherpadbackup:labs-cleanup-2015]] by [[User:BryanDavis|BryanDavis]] * 03:57 [[etherpad:p/labpuppetmaster1001-stragglers]] was archived as [[etherpadbackup:labpuppetmaster1001-stragglers]] by [[User:BryanDavis|BryanDavis]] * 03:57 [[etherpad:p/labnetfailover]] was archived as [[etherpadbackup:labnetfailover]] by [[User:BryanDavis|BryanDavis]] * 03:56 [[etherpad:p/l10nupdate]] was archived as [[etherpadbackup:l10nupdate]] by [[User:BryanDavis|BryanDavis]] * 03:56 [[etherpad:p/l10n-translate-centralnotice]] was archived as [[etherpadbackup:l10n-translate-centralnotice]] by [[User:BryanDavis|BryanDavis]] * 03:55 [[etherpad:p/l10n-team-2013-10]] was archived as [[etherpadbackup:l10n-team-2013-10]] by [[User:BryanDavis|BryanDavis]] * 03:55 [[etherpad:p/kubecon-2017-offsite-agenda]] was archived as [[etherpadbackup:kubecon-2017-offsite-agenda]] by [[User:BryanDavis|BryanDavis]] * 03:54 [[etherpad:p/kubecon]] was archived as [[etherpadbackup:kubecon]] by [[User:BryanDavis|BryanDavis]] * 03:54 [[etherpad:p/kubeadm]] was archived as [[etherpadbackup:kubeadm]] by [[User:BryanDavis|BryanDavis]] * 03:53 [[etherpad:p/kill-gridengine-slowly]] was archived as [[etherpadbackup:kill-gridengine-slowly]] by [[User:BryanDavis|BryanDavis]] * 03:53 [[etherpad:p/keystoneldap]] was archived as [[etherpadbackup:keystoneldap]] by [[User:BryanDavis|BryanDavis]] * 03:52 [[etherpad:p/keystonebootstrap]] was archived as [[etherpadbackup:keystonebootstrap]] by [[User:BryanDavis|BryanDavis]] * 03:52 [[etherpad:p/kafkaswift]] was archived as [[etherpadbackup:kafkaswift]] by [[User:BryanDavis|BryanDavis]] * 03:51 [[etherpad:p/kZaM0swM4c]] was archived as [[etherpadbackup:kZaM0swM4c]] by [[User:BryanDavis|BryanDavis]] * 03:51 [[etherpad:p/kAh7UZSoyI]] was archived as [[etherpadbackup:kAh7UZSoyI]] by [[User:BryanDavis|BryanDavis]] * 03:50 [[etherpad:p/kAKhhwCsJ7]] was archived as [[etherpadbackup:kAKhhwCsJ7]] by [[User:BryanDavis|BryanDavis]] * 03:49 [[etherpad:p/k8s-status-wishlist]] was archived as [[etherpadbackup:k8s-status-wishlist]] by [[User:BryanDavis|BryanDavis]] * 03:49 [[etherpad:p/jupyter-at-wikimedia]] was archived as [[etherpadbackup:jupyter-at-wikimedia]] by [[User:BryanDavis|BryanDavis]] * 03:48 [[etherpad:p/junk]] was archived as [[etherpadbackup:junk]] by [[User:BryanDavis|BryanDavis]] * 03:48 [[etherpad:p/joes-random-notes]] was archived as [[etherpadbackup:joes-random-notes]] by [[User:BryanDavis|BryanDavis]] * 03:47 [[etherpad:p/jmxtrans-packaging-improvements]] was archived as [[etherpadbackup:jmxtrans-packaging-improvements]] by [[User:BryanDavis|BryanDavis]] * 03:47 [[etherpad:p/jjb]] was archived as [[etherpadbackup:jjb]] by [[User:BryanDavis|BryanDavis]] * 03:46 [[etherpad:p/jenkins-upgrading]] was archived as [[etherpadbackup:jenkins-upgrading]] by [[User:BryanDavis|BryanDavis]] * 03:46 [[etherpad:p/jenkins-job-builder]] was archived as [[etherpadbackup:jenkins-job-builder]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/javascript-scoping]] was archived as [[etherpadbackup:javascript-scoping]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/jagori]] was archived as [[etherpadbackup:jagori]] by [[User:BryanDavis|BryanDavis]] * 03:44 [[etherpad:p/jQUprade]] was archived as [[etherpadbackup:jQUprade]] by [[User:BryanDavis|BryanDavis]] * 03:44 [[etherpad:p/j9qaVR_wF9bIWn6BWuIr]] was archived as [[etherpadbackup:j9qaVR_wF9bIWn6BWuIr]] by [[User:BryanDavis|BryanDavis]] * 03:43 [[etherpad:p/j4Obl63I8y]] was archived as [[etherpadbackup:j4Obl63I8y]] by [[User:BryanDavis|BryanDavis]] * 03:43 [[etherpad:p/iterationname]] was archived as [[etherpadbackup:iterationname]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/is]] was archived as [[etherpadbackup:is]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/ipv6]] was archived as [[etherpadbackup:ipv6]] by [[User:BryanDavis|BryanDavis]] * 03:41 [[etherpad:p/iosreleasing]] was archived as [[etherpadbackup:iosreleasing]] by [[User:BryanDavis|BryanDavis]] * 03:41 [[etherpad:p/iosrelease]] was archived as [[etherpadbackup:iosrelease]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/ios_app_update_release_notes]] was archived as [[etherpadbackup:ios_app_update_release_notes]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/ios-5.0]] was archived as [[etherpadbackup:ios-5.0]] by [[User:BryanDavis|BryanDavis]] * 03:39 [[etherpad:p/ios-3-1beta1-release-notes]] was archived as [[etherpadbackup:ios-3-1beta1-release-notes]] by [[User:BryanDavis|BryanDavis]] * 03:39 [[etherpad:p/intro_to_revscores]] was archived as [[etherpadbackup:intro_to_revscores]] by [[User:BryanDavis|BryanDavis]] * 03:38 [[etherpad:p/internal-graph-split-lvs]] was archived as [[etherpadbackup:internal-graph-split-lvs]] by [[User:BryanDavis|BryanDavis]] * 03:38 [[etherpad:p/intercontinental-debugging-may-07-2012]] was archived as [[etherpadbackup:intercontinental-debugging-may-07-2012]] by [[User:BryanDavis|BryanDavis]] * 03:37 [[etherpad:p/incident-20190509-codfwpuppetmasterdown]] was archived as [[etherpadbackup:incident-20190509-codfwpuppetmasterdown]] by [[User:BryanDavis|BryanDavis]] * 03:37 [[etherpad:p/in_article_search_query]] was archived as [[etherpadbackup:in_article_search_query]] by [[User:BryanDavis|BryanDavis]] * 03:36 [[etherpad:p/import_wcqs]] was archived as [[etherpadbackup:import_wcqs]] by [[User:BryanDavis|BryanDavis]] * 03:36 [[etherpad:p/ihatesasl]] was archived as [[etherpadbackup:ihatesasl]] by [[User:BryanDavis|BryanDavis]] * 03:35 [[etherpad:p/iOS_available_locale_ids]] was archived as [[etherpadbackup:iOS_available_locale_ids]] by [[User:BryanDavis|BryanDavis]] * 03:34 [[etherpad:p/iOSDataMerge]] was archived as [[etherpadbackup:iOSDataMerge]] by [[User:BryanDavis|BryanDavis]] * 03:34 [[etherpad:p/iOSAppi18nIssues]] was archived as [[etherpadbackup:iOSAppi18nIssues]] by [[User:BryanDavis|BryanDavis]] * 03:33 [[etherpad:p/iOSAppDescription]] was archived as [[etherpadbackup:iOSAppDescription]] by [[User:BryanDavis|BryanDavis]] * 03:33 [[etherpad:p/iOS-ChangeLog]] was archived as [[etherpadbackup:iOS-ChangeLog]] by [[User:BryanDavis|BryanDavis]] * 03:32 [[etherpad:p/iOS-7-issues]] was archived as [[etherpadbackup:iOS-7-issues]] by [[User:BryanDavis|BryanDavis]] * 03:32 [[etherpad:p/i8XjsnBxxh]] was archived as [[etherpadbackup:i8XjsnBxxh]] by [[User:BryanDavis|BryanDavis]] * 03:31 [[etherpad:p/i18n-rfc-2013-11]] was archived as [[etherpadbackup:i18n-rfc-2013-11]] by [[User:BryanDavis|BryanDavis]] * 03:31 [[etherpad:p/huggle_section]] was archived as [[etherpadbackup:huggle_section]] by [[User:BryanDavis|BryanDavis]] * 03:30 [[etherpad:p/horizonsso]] was archived as [[etherpadbackup:horizonsso]] by [[User:BryanDavis|BryanDavis]] * 03:30 [[etherpad:p/hi6ZshFUi2]] was archived as [[etherpadbackup:hi6ZshFUi2]] by [[User:BryanDavis|BryanDavis]] * 03:29 [[etherpad:p/hhvm]] was archived as [[etherpadbackup:hhvm]] by [[User:BryanDavis|BryanDavis]] * 03:29 [[etherpad:p/helm-toollabs]] was archived as [[etherpadbackup:helm-toollabs]] by [[User:BryanDavis|BryanDavis]] * 03:28 [[etherpad:p/helm-chart-development]] was archived as [[etherpadbackup:helm-chart-development]] by [[User:BryanDavis|BryanDavis]] * 03:28 [[etherpad:p/handoff-wdqs-T388134]] was archived as [[etherpadbackup:handoff-wdqs-T388134]] by [[User:BryanDavis|BryanDavis]] * 03:27 [[etherpad:p/hackathon-showcase-before-the-catastrophe]] was archived as [[etherpadbackup:hackathon-showcase-before-the-catastrophe]] by [[User:BryanDavis|BryanDavis]] * 03:27 [[etherpad:p/hackathon-rhinosf1-barriers]] was archived as [[etherpadbackup:hackathon-rhinosf1-barriers]] by [[User:BryanDavis|BryanDavis]] * 03:26 [[etherpad:p/gridengine-uses]] was archived as [[etherpadbackup:gridengine-uses]] by [[User:BryanDavis|BryanDavis]] * 03:26 [[etherpad:p/grid-backup]] was archived as [[etherpadbackup:grid-backup]] by [[User:BryanDavis|BryanDavis]] * 03:25 [[etherpad:p/graphitetodo]] was archived as [[etherpadbackup:graphitetodo]] by [[User:BryanDavis|BryanDavis]] * 03:25 [[etherpad:p/grantreview]] was archived as [[etherpadbackup:grantreview]] by [[User:BryanDavis|BryanDavis]] * 03:24 [[etherpad:p/gitlab-sync]] was archived as [[etherpadbackup:gitlab-sync]] by [[User:BryanDavis|BryanDavis]] * 03:24 [[etherpad:p/git]] was archived as [[etherpadbackup:git]] by [[User:BryanDavis|BryanDavis]] * 03:23 [[etherpad:p/gerrit-4841]] was archived as [[etherpadbackup:gerrit-4841]] by [[User:BryanDavis|BryanDavis]] * 03:23 [[etherpad:p/gdash-labels-clarification]] was archived as [[etherpadbackup:gdash-labels-clarification]] by [[User:BryanDavis|BryanDavis]] * 03:22 [[etherpad:p/gcimail]] was archived as [[etherpadbackup:gcimail]] by [[User:BryanDavis|BryanDavis]] * 03:22 [[etherpad:p/gannum]] was archived as [[etherpadbackup:gannum]] by [[User:BryanDavis|BryanDavis]] * 03:21 [[etherpad:p/g5an_WR4fa02lVjmHQse]] was archived as [[etherpadbackup:g5an_WR4fa02lVjmHQse]] by [[User:BryanDavis|BryanDavis]] * 03:21 [[etherpad:p/g505410announce]] was archived as [[etherpadbackup:g505410announce]] by [[User:BryanDavis|BryanDavis]] * 03:20 [[etherpad:p/g4-flavors]] was archived as [[etherpadbackup:g4-flavors]] by [[User:BryanDavis|BryanDavis]] * 03:19 [[etherpad:p/future-of-language-converter]] was archived as [[etherpadbackup:future-of-language-converter]] by [[User:BryanDavis|BryanDavis]] * 03:19 [[etherpad:p/fullstack_leaks]] was archived as [[etherpadbackup:fullstack_leaks]] by [[User:BryanDavis|BryanDavis]] * 03:18 [[etherpad:p/fullstack-flapping]] was archived as [[etherpadbackup:fullstack-flapping]] by [[User:BryanDavis|BryanDavis]] * 03:18 [[etherpad:p/fr-tech_chores_2023_fish_HEAD^]] was archived as [[etherpadbackup:fr-tech_chores_2023_fish_HEAD^]] by [[User:BryanDavis|BryanDavis]] * 03:17 [[etherpad:p/fp-Hrk0fve_GXMeMZFJk]] was archived as [[etherpadbackup:fp-Hrk0fve_GXMeMZFJk]] by [[User:BryanDavis|BryanDavis]] * 03:17 [[etherpad:p/floating-ip-aliaser]] was archived as [[etherpadbackup:floating-ip-aliaser]] by [[User:BryanDavis|BryanDavis]] * 03:16 [[etherpad:p/flagellating-funnel]] was archived as [[etherpadbackup:flagellating-funnel]] by [[User:BryanDavis|BryanDavis]] * 03:16 [[etherpad:p/extdist]] was archived as [[etherpadbackup:extdist]] by [[User:BryanDavis|BryanDavis]] * 03:15 [[etherpad:p/existensions]] was archived as [[etherpadbackup:existensions]] by [[User:BryanDavis|BryanDavis]] * 03:15 [[etherpad:p/eventlogging_stag]] was archived as [[etherpadbackup:eventlogging_stag]] by [[User:BryanDavis|BryanDavis]] * 03:14 [[etherpad:p/eventbus-statsd]] was archived as [[etherpadbackup:eventbus-statsd]] by [[User:BryanDavis|BryanDavis]] * 03:14 [[etherpad:p/essex-upgrade]] was archived as [[etherpadbackup:essex-upgrade]] by [[User:BryanDavis|BryanDavis]] * 03:13 [[etherpad:p/esams-followup]] was archived as [[etherpadbackup:esams-followup]] by [[User:BryanDavis|BryanDavis]] * 03:13 [[etherpad:p/eqiad1-upgrade-pike]] was archived as [[etherpadbackup:eqiad1-upgrade-pike]] by [[User:BryanDavis|BryanDavis]] * 03:12 [[etherpad:p/eqiad1-neutron-bootstrap]] was archived as [[etherpadbackup:eqiad1-neutron-bootstrap]] by [[User:BryanDavis|BryanDavis]] * 03:12 [[etherpad:p/eqiad1-keystone-bootstrap]] was archived as [[etherpadbackup:eqiad1-keystone-bootstrap]] by [[User:BryanDavis|BryanDavis]] * 03:11 [[etherpad:p/eqiad1]] was archived as [[etherpadbackup:eqiad1]] by [[User:BryanDavis|BryanDavis]] * 03:11 [[etherpad:p/enwikiSOPADBUpgrades]] was archived as [[etherpadbackup:enwikiSOPADBUpgrades]] by [[User:BryanDavis|BryanDavis]] * 03:10 [[etherpad:p/engprod-irc]] was archived as [[etherpadbackup:engprod-irc]] by [[User:BryanDavis|BryanDavis]] * 03:10 [[etherpad:p/empty_vps_projects]] was archived as [[etherpadbackup:empty_vps_projects]] by [[User:BryanDavis|BryanDavis]] * 03:09 [[etherpad:p/embeddings_and_topic_models]] was archived as [[etherpadbackup:embeddings_and_topic_models]] by [[User:BryanDavis|BryanDavis]] * 03:09 [[etherpad:p/email-tools]] was archived as [[etherpadbackup:email-tools]] by [[User:BryanDavis|BryanDavis]] * 03:08 [[etherpad:p/elukey-netflow]] was archived as [[etherpadbackup:elukey-netflow]] by [[User:BryanDavis|BryanDavis]] * 03:07 [[etherpad:p/elastic-single-node-testing]] was archived as [[etherpadbackup:elastic-single-node-testing]] by [[User:BryanDavis|BryanDavis]] * 03:07 [[etherpad:p/elastic-2-opensearch-T388610]] was archived as [[etherpadbackup:elastic-2-opensearch-T388610]] by [[User:BryanDavis|BryanDavis]] * 03:06 [[etherpad:p/el_utils]] was archived as [[etherpadbackup:el_utils]] by [[User:BryanDavis|BryanDavis]] * 03:06 [[etherpad:p/eiwGl_RxtBIFSyuxv5Yr]] was archived as [[etherpadbackup:eiwGl_RxtBIFSyuxv5Yr]] by [[User:BryanDavis|BryanDavis]] * 03:05 [[etherpad:p/effiee]] was archived as [[etherpadbackup:effiee]] by [[User:BryanDavis|BryanDavis]] * 03:05 [[etherpad:p/editing_logging]] was archived as [[etherpadbackup:editing_logging]] by [[User:BryanDavis|BryanDavis]] * 03:04 [[etherpad:p/edit_types_event_schema]] was archived as [[etherpadbackup:edit_types_event_schema]] by [[User:BryanDavis|BryanDavis]] * 03:04 [[etherpad:p/edit_history_vetting]] was archived as [[etherpadbackup:edit_history_vetting]] by [[User:BryanDavis|BryanDavis]] * 03:03 [[etherpad:p/e1]] was archived as [[etherpadbackup:e1]] by [[User:BryanDavis|BryanDavis]] * 03:03 [[etherpad:p/e]] was archived as [[etherpadbackup:e]] by [[User:BryanDavis|BryanDavis]] * 03:02 [[etherpad:p/doc-from-git]] was archived as [[etherpadbackup:doc-from-git]] by [[User:BryanDavis|BryanDavis]] * 03:02 [[etherpad:p/dnsthings]] was archived as [[etherpadbackup:dnsthings]] by [[User:BryanDavis|BryanDavis]] * 03:01 [[etherpad:p/dns_cleanup]] was archived as [[etherpadbackup:dns_cleanup]] by [[User:BryanDavis|BryanDavis]] * 03:01 [[etherpad:p/dmitry-onboarding]] was archived as [[etherpadbackup:dmitry-onboarding]] by [[User:BryanDavis|BryanDavis]] * 03:00 [[etherpad:p/disk_aio_setting]] was archived as [[etherpadbackup:disk_aio_setting]] by [[User:BryanDavis|BryanDavis]] * 03:00 [[etherpad:p/diff_test]] was archived as [[etherpadbackup:diff_test]] by [[User:BryanDavis|BryanDavis]] * 02:59 [[etherpad:p/diamond-deployment]] was archived as [[etherpadbackup:diamond-deployment]] by [[User:BryanDavis|BryanDavis]] * 02:59 [[etherpad:p/dhcp-option82-sunset]] was archived as [[etherpadbackup:dhcp-option82-sunset]] by [[User:BryanDavis|BryanDavis]] * 02:58 [[etherpad:p/devsummit18-thirdpartymediawiki]] was archived as [[etherpadbackup:devsummit18-thirdpartymediawiki]] by [[User:BryanDavis|BryanDavis]] * 02:58 [[etherpad:p/devsummit17-support-global-preferences]] was archived as [[etherpadbackup:devsummit17-support-global-preferences]] by [[User:BryanDavis|BryanDavis]] * 02:57 [[etherpad:p/devsummit17-scaling-database-schema]] was archived as [[etherpadbackup:devsummit17-scaling-database-schema]] by [[User:BryanDavis|BryanDavis]] * 02:57 [[etherpad:p/devsummit17-integrating-mediawiki]] was archived as [[etherpadbackup:devsummit17-integrating-mediawiki]] by [[User:BryanDavis|BryanDavis]] * 02:56 [[etherpad:p/devsummit17-algorithmic-dangers]] was archived as [[etherpadbackup:devsummit17-algorithmic-dangers]] by [[User:BryanDavis|BryanDavis]] * 02:56 [[etherpad:p/devsummit17-ai-wishlist]] was archived as [[etherpadbackup:devsummit17-ai-wishlist]] by [[User:BryanDavis|BryanDavis]] * 02:55 [[etherpad:p/devsummit17-T149624]] was archived as [[etherpadbackup:devsummit17-T149624]] by [[User:BryanDavis|BryanDavis]] * 02:55 [[etherpad:p/devsummit17-RedesignSpecialSearchPage]] was archived as [[etherpadbackup:devsummit17-RedesignSpecialSearchPage]] by [[User:BryanDavis|BryanDavis]] * 02:54 [[etherpad:p/devsummit17-ProjectOREO]] was archived as [[etherpadbackup:devsummit17-ProjectOREO]] by [[User:BryanDavis|BryanDavis]] * 02:53 [[etherpad:p/devsummit17-CodeReview]] was archived as [[etherpadbackup:devsummit17-CodeReview]] by [[User:BryanDavis|BryanDavis]] * 02:53 [[etherpad:p/devpowpow-2015-02-25]] was archived as [[etherpadbackup:devpowpow-2015-02-25]] by [[User:BryanDavis|BryanDavis]] * 02:52 [[etherpad:p/design_review_10_22]] was archived as [[etherpadbackup:design_review_10_22]] by [[User:BryanDavis|BryanDavis]] * 02:52 [[etherpad:p/design-thingies-jan-android]] was archived as [[etherpadbackup:design-thingies-jan-android]] by [[User:BryanDavis|BryanDavis]] * 02:51 [[etherpad:p/design]] was archived as [[etherpadbackup:design]] by [[User:BryanDavis|BryanDavis]] * 02:51 [[etherpad:p/deprecate-precise]] was archived as [[etherpadbackup:deprecate-precise]] by [[User:BryanDavis|BryanDavis]] * 02:50 [[etherpad:p/depool]] was archived as [[etherpadbackup:depool]] by [[User:BryanDavis|BryanDavis]] * 02:50 [[etherpad:p/deployworkinggroup]] was archived as [[etherpadbackup:deployworkinggroup]] by [[User:BryanDavis|BryanDavis]] * 02:49 [[etherpad:p/deploytrainingprep]] was archived as [[etherpadbackup:deploytrainingprep]] by [[User:BryanDavis|BryanDavis]] * 02:49 [[etherpad:p/deploy-20150427-SWAT-evening]] was archived as [[etherpadbackup:deploy-20150427-SWAT-evening]] by [[User:BryanDavis|BryanDavis]] * 02:48 [[etherpad:p/deploy-20150416-SWAT-evening]] was archived as [[etherpadbackup:deploy-20150416-SWAT-evening]] by [[User:BryanDavis|BryanDavis]] * 02:48 [[etherpad:p/deletedesedomains]] was archived as [[etherpadbackup:deletedesedomains]] by [[User:BryanDavis|BryanDavis]] * 02:47 [[etherpad:p/dear-tool-user]] was archived as [[etherpadbackup:dear-tool-user]] by [[User:BryanDavis|BryanDavis]] * 02:47 [[etherpad:p/dchanPlate2015]] was archived as [[etherpadbackup:dchanPlate2015]] by [[User:BryanDavis|BryanDavis]] * 02:46 [[etherpad:p/db-renames]] was archived as [[etherpadbackup:db-renames]] by [[User:BryanDavis|BryanDavis]] * 02:46 [[etherpad:p/datalossjuly2]] was archived as [[etherpadbackup:datalossjuly2]] by [[User:BryanDavis|BryanDavis]] * 02:45 [[etherpad:p/cx-presentation]] was archived as [[etherpadbackup:cx-presentation]] by [[User:BryanDavis|BryanDavis]] * 02:45 [[etherpad:p/cx-markup-alignment-uppercase]] was archived as [[etherpadbackup:cx-markup-alignment-uppercase]] by [[User:BryanDavis|BryanDavis]] * 02:44 [[etherpad:p/cx-markup-alignment]] was archived as [[etherpadbackup:cx-markup-alignment]] by [[User:BryanDavis|BryanDavis]] * 02:44 [[etherpad:p/cx-invite]] was archived as [[etherpadbackup:cx-invite]] by [[User:BryanDavis|BryanDavis]] * 02:43 [[etherpad:p/cx-hack]] was archived as [[etherpadbackup:cx-hack]] by [[User:BryanDavis|BryanDavis]] * 02:43 [[etherpad:p/cx-composer]] was archived as [[etherpadbackup:cx-composer]] by [[User:BryanDavis|BryanDavis]] * 02:42 [[etherpad:p/csKD3No9oL]] was archived as [[etherpadbackup:csKD3No9oL]] by [[User:BryanDavis|BryanDavis]] * 02:42 [[etherpad:p/cronerror]] was archived as [[etherpadbackup:cronerror]] by [[User:BryanDavis|BryanDavis]] * 02:41 [[etherpad:p/cr2-codfw_fpc0]] was archived as [[etherpadbackup:cr2-codfw_fpc0]] by [[User:BryanDavis|BryanDavis]] * 02:41 [[etherpad:p/conftool-T379329]] was archived as [[etherpadbackup:conftool-T379329]] by [[User:BryanDavis|BryanDavis]] * 02:40 [[etherpad:p/completion-ab-sanity-check]] was archived as [[etherpadbackup:completion-ab-sanity-check]] by [[User:BryanDavis|BryanDavis]] * 02:40 [[etherpad:p/compat-network]] was archived as [[etherpadbackup:compat-network]] by [[User:BryanDavis|BryanDavis]] * 02:39 [[etherpad:p/commons-ios-fixes]] was archived as [[etherpadbackup:commons-ios-fixes]] by [[User:BryanDavis|BryanDavis]] * 02:38 [[etherpad:p/commons-cross-wiki-uploads]] was archived as [[etherpadbackup:commons-cross-wiki-uploads]] by [[User:BryanDavis|BryanDavis]] * 02:38 [[etherpad:p/commons-app-post-draft]] was archived as [[etherpadbackup:commons-app-post-draft]] by [[User:BryanDavis|BryanDavis]] * 02:37 [[etherpad:p/collation]] was archived as [[etherpadbackup:collation]] by [[User:BryanDavis|BryanDavis]] * 02:37 [[etherpad:p/codeacademy-diwanship]] was archived as [[etherpadbackup:codeacademy-diwanship]] by [[User:BryanDavis|BryanDavis]] * 02:36 [[etherpad:p/cna4GMpRlJCm65THgk-l]] was archived as [[etherpadbackup:cna4GMpRlJCm65THgk-l]] by [[User:BryanDavis|BryanDavis]] * 02:36 [[etherpad:p/cloudvpskernels]] was archived as [[etherpadbackup:cloudvpskernels]] by [[User:BryanDavis|BryanDavis]] * 02:35 [[etherpad:p/cloudvps-no-agent-forwarding]] was archived as [[etherpadbackup:cloudvps-no-agent-forwarding]] by [[User:BryanDavis|BryanDavis]] * 02:35 [[etherpad:p/cloudvirt1018-20190213]] was archived as [[etherpadbackup:cloudvirt1018-20190213]] by [[User:BryanDavis|BryanDavis]] * 02:34 [[etherpad:p/cloudservices1003-upgrade]] was archived as [[etherpadbackup:cloudservices1003-upgrade]] by [[User:BryanDavis|BryanDavis]] * 02:34 [[etherpad:p/cloudreboots]] was archived as [[etherpadbackup:cloudreboots]] by [[User:BryanDavis|BryanDavis]] * 02:33 [[etherpad:p/cloudprojectrequeststweak]] was archived as [[etherpadbackup:cloudprojectrequeststweak]] by [[User:BryanDavis|BryanDavis]] * 02:33 [[etherpad:p/cloudnative]] was archived as [[etherpadbackup:cloudnative]] by [[User:BryanDavis|BryanDavis]] * 02:32 [[etherpad:p/cloudgw-migration]] was archived as [[etherpadbackup:cloudgw-migration]] by [[User:BryanDavis|BryanDavis]] * 02:32 [[etherpad:p/cloudfontcdn]] was archived as [[etherpadbackup:cloudfontcdn]] by [[User:BryanDavis|BryanDavis]] * 02:31 [[etherpad:p/cloudelastic-mystery-red]] was archived as [[etherpadbackup:cloudelastic-mystery-red]] by [[User:BryanDavis|BryanDavis]] * 02:31 [[etherpad:p/cloudelastic-ipip]] was archived as [[etherpadbackup:cloudelastic-ipip]] by [[User:BryanDavis|BryanDavis]] * 02:30 [[etherpad:p/cloudelastic-2-opensearch]] was archived as [[etherpadbackup:cloudelastic-2-opensearch]] by [[User:BryanDavis|BryanDavis]] * 02:30 [[etherpad:p/cloud_purge_2018_ping]] was archived as [[etherpadbackup:cloud_purge_2018_ping]] by [[User:BryanDavis|BryanDavis]] * 02:29 [[etherpad:p/cloud_announcement_20180910]] was archived as [[etherpadbackup:cloud_announcement_20180910]] by [[User:BryanDavis|BryanDavis]] * 02:29 [[etherpad:p/cloud-vps-purge-deletions]] was archived as [[etherpadbackup:cloud-vps-purge-deletions]] by [[User:BryanDavis|BryanDavis]] * 02:28 [[etherpad:p/citoid-TD-fun]] was archived as [[etherpadbackup:citoid-TD-fun]] by [[User:BryanDavis|BryanDavis]] * 02:28 [[etherpad:p/cirrus_cross_project_search_debug_settings]] was archived as [[etherpadbackup:cirrus_cross_project_search_debug_settings]] by [[User:BryanDavis|BryanDavis]] * 02:27 [[etherpad:p/ci-apt-docker-why]] was archived as [[etherpadbackup:ci-apt-docker-why]] by [[User:BryanDavis|BryanDavis]] * 02:27 [[etherpad:p/chrismcmahon]] was archived as [[etherpadbackup:chrismcmahon]] by [[User:BryanDavis|BryanDavis]] * 02:26 [[etherpad:p/changeuid]] was archived as [[etherpadbackup:changeuid]] by [[User:BryanDavis|BryanDavis]] * 02:26 [[etherpad:p/changes]] was archived as [[etherpadbackup:changes]] by [[User:BryanDavis|BryanDavis]] * 02:25 [[etherpad:p/ceph-poc-to-prod]] was archived as [[etherpadbackup:ceph-poc-to-prod]] by [[User:BryanDavis|BryanDavis]] * 02:25 [[etherpad:p/categories-lag]] was archived as [[etherpadbackup:categories-lag]] by [[User:BryanDavis|BryanDavis]] * 02:24 [[etherpad:p/calendar-minimums]] was archived as [[etherpadbackup:calendar-minimums]] by [[User:BryanDavis|BryanDavis]] * 02:24 [[etherpad:p/busterreminder]] was archived as [[etherpadbackup:busterreminder]] by [[User:BryanDavis|BryanDavis]] * 02:23 [[etherpad:p/bug35939]] was archived as [[etherpadbackup:bug35939]] by [[User:BryanDavis|BryanDavis]] * 02:22 [[etherpad:p/brokenpms]] was archived as [[etherpadbackup:brokenpms]] by [[User:BryanDavis|BryanDavis]] * 02:22 [[etherpad:p/bloated-website-code-drains-yo]] was archived as [[etherpadbackup:bloated-website-code-drains-yo]] by [[User:BryanDavis|BryanDavis]] * 02:21 [[etherpad:p/bios-be-gone]] was archived as [[etherpadbackup:bios-be-gone]] by [[User:BryanDavis|BryanDavis]] * 02:21 [[etherpad:p/bigbrother-almost-no-more]] was archived as [[etherpadbackup:bigbrother-almost-no-more]] by [[User:BryanDavis|BryanDavis]] * 02:20 [[etherpad:p/beta-new-ext]] was archived as [[etherpadbackup:beta-new-ext]] by [[User:BryanDavis|BryanDavis]] * 02:20 [[etherpad:p/bernd-onboarding]] was archived as [[etherpadbackup:bernd-onboarding]] by [[User:BryanDavis|BryanDavis]] * 02:19 [[etherpad:p/beginner-hackathons-thoughts]] was archived as [[etherpadbackup:beginner-hackathons-thoughts]] by [[User:BryanDavis|BryanDavis]] * 02:19 [[etherpad:p/bash]] was archived as [[etherpadbackup:bash]] by [[User:BryanDavis|BryanDavis]] * 02:18 [[etherpad:p/bangarang]] was archived as [[etherpadbackup:bangarang]] by [[User:BryanDavis|BryanDavis]] * 02:18 [[etherpad:p/band_names_web]] was archived as [[etherpadbackup:band_names_web]] by [[User:BryanDavis|BryanDavis]] * 02:17 [[etherpad:p/avro]] was archived as [[etherpadbackup:avro]] by [[User:BryanDavis|BryanDavis]] * 02:17 [[etherpad:p/aqs-standup]] was archived as [[etherpadbackup:aqs-standup]] by [[User:BryanDavis|BryanDavis]] * 02:16 [[etherpad:p/app_q3_planning]] was archived as [[etherpadbackup:app_q3_planning]] by [[User:BryanDavis|BryanDavis]] * 02:16 [[etherpad:p/app_q3_health_check]] was archived as [[etherpadbackup:app_q3_health_check]] by [[User:BryanDavis|BryanDavis]] * 02:15 [[etherpad:p/app_process_improvements]] was archived as [[etherpadbackup:app_process_improvements]] by [[User:BryanDavis|BryanDavis]] * 02:15 [[etherpad:p/appQ4planning]] was archived as [[etherpadbackup:appQ4planning]] by [[User:BryanDavis|BryanDavis]] * 02:14 [[etherpad:p/app-strategy-discussion]] was archived as [[etherpadbackup:app-strategy-discussion]] by [[User:BryanDavis|BryanDavis]] * 02:14 [[etherpad:p/app-server-upgrade]] was archived as [[etherpadbackup:app-server-upgrade]] by [[User:BryanDavis|BryanDavis]] * 02:13 [[etherpad:p/app-1-1beta4-3-1beta3-release-notes]] was archived as [[etherpadbackup:app-1-1beta4-3-1beta3-release-notes]] by [[User:BryanDavis|BryanDavis]] * 02:13 [[etherpad:p/apiv2]] was archived as [[etherpadbackup:apiv2]] by [[User:BryanDavis|BryanDavis]] * 02:12 [[etherpad:p/api.php]] was archived as [[etherpadbackup:api.php]] by [[User:BryanDavis|BryanDavis]] * 02:12 [[etherpad:p/androidappfeedback]] was archived as [[etherpadbackup:androidappfeedback]] by [[User:BryanDavis|BryanDavis]] * 02:11 [[etherpad:p/android-ime]] was archived as [[etherpadbackup:android-ime]] by [[User:BryanDavis|BryanDavis]] * 02:11 [[etherpad:p/android-commons-app-1-0]] was archived as [[etherpadbackup:android-commons-app-1-0]] by [[User:BryanDavis|BryanDavis]] * 02:10 [[etherpad:p/android-1-1rc1-release-notes]] was archived as [[etherpadbackup:android-1-1rc1-release-notes]] by [[User:BryanDavis|BryanDavis]] * 02:10 [[etherpad:p/android-1-1beta1-release-notes]] was archived as [[etherpadbackup:android-1-1beta1-release-notes]] by [[User:BryanDavis|BryanDavis]] * 02:09 [[etherpad:p/android-1-1-iOS-3-1-third-beta-notes]] was archived as [[etherpadbackup:android-1-1-iOS-3-1-third-beta-notes]] by [[User:BryanDavis|BryanDavis]] * 02:08 [[etherpad:p/android-1-1-beta2-release-notes]] was archived as [[etherpadbackup:android-1-1-beta2-release-notes]] by [[User:BryanDavis|BryanDavis]] * 02:08 [[etherpad:p/analytics163]] was archived as [[etherpadbackup:analytics163]] by [[User:BryanDavis|BryanDavis]] * 02:07 [[etherpad:p/analytics1003-reinstall]] was archived as [[etherpadbackup:analytics1003-reinstall]] by [[User:BryanDavis|BryanDavis]] * 02:07 [[etherpad:p/analytics-todo]] was archived as [[etherpadbackup:analytics-todo]] by [[User:BryanDavis|BryanDavis]] * 02:06 [[etherpad:p/analytics-tech-debt]] was archived as [[etherpadbackup:analytics-tech-debt]] by [[User:BryanDavis|BryanDavis]] * 02:06 [[etherpad:p/analytics-staff-meeting]] was archived as [[etherpadbackup:analytics-staff-meeting]] by [[User:BryanDavis|BryanDavis]] * 02:05 [[etherpad:p/analytics-row-d-maintenance]] was archived as [[etherpadbackup:analytics-row-d-maintenance]] by [[User:BryanDavis|BryanDavis]] * 02:05 [[etherpad:p/analytics-retrospective]] was archived as [[etherpadbackup:analytics-retrospective]] by [[User:BryanDavis|BryanDavis]] * 02:04 [[etherpad:p/analytics-ops-druid]] was archived as [[etherpadbackup:analytics-ops-druid]] by [[User:BryanDavis|BryanDavis]] * 02:04 [[etherpad:p/analytics-ops]] was archived as [[etherpadbackup:analytics-ops]] by [[User:BryanDavis|BryanDavis]] * 02:03 [[etherpad:p/analytics-oozie-13102016]] was archived as [[etherpadbackup:analytics-oozie-13102016]] by [[User:BryanDavis|BryanDavis]] * 02:03 [[etherpad:p/analytics-naming]] was archived as [[etherpadbackup:analytics-naming]] by [[User:BryanDavis|BryanDavis]] * 02:02 [[etherpad:p/analytics-mwds-2015]] was archived as [[etherpadbackup:analytics-mwds-2015]] by [[User:BryanDavis|BryanDavis]] * 02:02 [[etherpad:p/analytics-ingestion]] was archived as [[etherpadbackup:analytics-ingestion]] by [[User:BryanDavis|BryanDavis]] * 02:01 [[etherpad:p/analytics-hive-udf]] was archived as [[etherpadbackup:analytics-hive-udf]] by [[User:BryanDavis|BryanDavis]] * 02:01 [[etherpad:p/analytics-gobblin-mess]] was archived as [[etherpadbackup:analytics-gobblin-mess]] by [[User:BryanDavis|BryanDavis]] * 02:00 [[etherpad:p/analytics-goals]] was archived as [[etherpadbackup:analytics-goals]] by [[User:BryanDavis|BryanDavis]] * 02:00 [[etherpad:p/analytics-eventlogging-postmortem-2014-11]] was archived as [[etherpadbackup:analytics-eventlogging-postmortem-2014-11]] by [[User:BryanDavis|BryanDavis]] * 01:59 [[etherpad:p/analytics-email-drafts]] was archived as [[etherpadbackup:analytics-email-drafts]] by [[User:BryanDavis|BryanDavis]] * 01:59 [[etherpad:p/analytics-druid-upgrade]] was archived as [[etherpadbackup:analytics-druid-upgrade]] by [[User:BryanDavis|BryanDavis]] * 01:58 [[etherpad:p/analytics-druid-migration]] was archived as [[etherpadbackup:analytics-druid-migration]] by [[User:BryanDavis|BryanDavis]] * 01:58 [[etherpad:p/analytics-deploy-aqs]] was archived as [[etherpadbackup:analytics-deploy-aqs]] by [[User:BryanDavis|BryanDavis]] * 01:57 [[etherpad:p/analytics-bikeshedding]] was archived as [[etherpadbackup:analytics-bikeshedding]] by [[User:BryanDavis|BryanDavis]] * 01:57 [[etherpad:p/analytics-bikeshed]] was archived as [[etherpadbackup:analytics-bikeshed]] by [[User:BryanDavis|BryanDavis]] * 01:56 [[etherpad:p/analytics-analytics1003-jessie-upgrade]] was archived as [[etherpadbackup:analytics-analytics1003-jessie-upgrade]] by [[User:BryanDavis|BryanDavis]] * 01:56 [[etherpad:p/analytics-VE]] was archived as [[etherpadbackup:analytics-VE]] by [[User:BryanDavis|BryanDavis]] * 01:55 [[etherpad:p/analytics-73331]] was archived as [[etherpadbackup:analytics-73331]] by [[User:BryanDavis|BryanDavis]] * 01:55 [[etherpad:p/analytics-72745]] was archived as [[etherpadbackup:analytics-72745]] by [[User:BryanDavis|BryanDavis]] * 01:54 [[etherpad:p/analytics-72740]] was archived as [[etherpadbackup:analytics-72740]] by [[User:BryanDavis|BryanDavis]] * 01:54 [[etherpad:p/analytics-72739]] was archived as [[etherpadbackup:analytics-72739]] by [[User:BryanDavis|BryanDavis]] * 01:53 [[etherpad:p/analytics-72738]] was archived as [[etherpadbackup:analytics-72738]] by [[User:BryanDavis|BryanDavis]] * 01:52 [[etherpad:p/analytics-72737]] was archived as [[etherpadbackup:analytics-72737]] by [[User:BryanDavis|BryanDavis]] * 01:52 [[etherpad:p/analytics-72736]] was archived as [[etherpadbackup:analytics-72736]] by [[User:BryanDavis|BryanDavis]] * 01:51 [[etherpad:p/analytics-72735]] was archived as [[etherpadbackup:analytics-72735]] by [[User:BryanDavis|BryanDavis]] * 01:51 [[etherpad:p/analytics-69254]] was archived as [[etherpadbackup:analytics-69254]] by [[User:BryanDavis|BryanDavis]] * 01:50 [[etherpad:p/alpha-signature-change]] was archived as [[etherpadbackup:alpha-signature-change]] by [[User:BryanDavis|BryanDavis]] * 01:50 [[etherpad:p/ai_vision_wikimedia]] was archived as [[etherpadbackup:ai_vision_wikimedia]] by [[User:BryanDavis|BryanDavis]] * 01:49 [[etherpad:p/aharoni-20160420]] was archived as [[etherpadbackup:aharoni-20160420]] by [[User:BryanDavis|BryanDavis]] * 01:49 [[etherpad:p/aftereffects]] was archived as [[etherpadbackup:aftereffects]] by [[User:BryanDavis|BryanDavis]] * 01:48 [[etherpad:p/admin_accounts_cleanup]] was archived as [[etherpadbackup:admin_accounts_cleanup]] by [[User:BryanDavis|BryanDavis]] * 01:48 [[etherpad:p/admin]] was archived as [[etherpadbackup:admin]] by [[User:BryanDavis|BryanDavis]] * 01:47 [[etherpad:p/addwiki]] was archived as [[etherpadbackup:addwiki]] by [[User:BryanDavis|BryanDavis]] * 01:47 [[etherpad:p/addshore]] was archived as [[etherpadbackup:addshore]] by [[User:BryanDavis|BryanDavis]] * 01:46 [[etherpad:p/actionbar_checklist]] was archived as [[etherpadbackup:actionbar_checklist]] by [[User:BryanDavis|BryanDavis]] * 01:46 [[etherpad:p/access_request_wikis]] was archived as [[etherpadbackup:access_request_wikis]] by [[User:BryanDavis|BryanDavis]] * 01:45 [[etherpad:p/absents]] was archived as [[etherpadbackup:absents]] by [[User:BryanDavis|BryanDavis]] * 01:45 [[etherpad:p/a4NCijsjQp]] was archived as [[etherpadbackup:a4NCijsjQp]] by [[User:BryanDavis|BryanDavis]] * 01:44 [[etherpad:p/ZVFiVE3E09]] was archived as [[etherpadbackup:ZVFiVE3E09]] by [[User:BryanDavis|BryanDavis]] * 01:44 [[etherpad:p/XcC5u59nDdRc_itxINnb]] was archived as [[etherpadbackup:XcC5u59nDdRc_itxINnb]] by [[User:BryanDavis|BryanDavis]] * 01:43 [[etherpad:p/XH6WkSmARmw8dy7UMiIR]] was archived as [[etherpadbackup:XH6WkSmARmw8dy7UMiIR]] by [[User:BryanDavis|BryanDavis]] * 01:43 [[etherpad:p/WsWWUiRllf]] was archived as [[etherpadbackup:WsWWUiRllf]] by [[User:BryanDavis|BryanDavis]] * 01:42 [[etherpad:p/WorT2b8qpR]] was archived as [[etherpadbackup:WorT2b8qpR]] by [[User:BryanDavis|BryanDavis]] * 01:42 [[etherpad:p/Wj4HcOWoHk]] was archived as [[etherpadbackup:Wj4HcOWoHk]] by [[User:BryanDavis|BryanDavis]] * 01:41 [[etherpad:p/WikipediaiOSTesting]] was archived as [[etherpadbackup:WikipediaiOSTesting]] by [[User:BryanDavis|BryanDavis]] * 01:41 [[etherpad:p/WikipediaZeroRetrospective]] was archived as [[etherpadbackup:WikipediaZeroRetrospective]] by [[User:BryanDavis|BryanDavis]] * 01:40 [[etherpad:p/WikipediaPhoneGapAndroidMeetup]] was archived as [[etherpadbackup:WikipediaPhoneGapAndroidMeetup]] by [[User:BryanDavis|BryanDavis]] * 01:40 [[etherpad:p/WikipediaMobileiOS3-1Changelog]] was archived as [[etherpadbackup:WikipediaMobileiOS3-1Changelog]] by [[User:BryanDavis|BryanDavis]] * 01:39 [[etherpad:p/WikipediaMobileAndroidV1-0-1]] was archived as [[etherpadbackup:WikipediaMobileAndroidV1-0-1]] by [[User:BryanDavis|BryanDavis]] * 01:39 [[etherpad:p/WikipediaMobileAndroidRelease]] was archived as [[etherpadbackup:WikipediaMobileAndroidRelease]] by [[User:BryanDavis|BryanDavis]] * 01:38 [[etherpad:p/WikipediaMobileAndroid-V1-0-2]] was archived as [[etherpadbackup:WikipediaMobileAndroid-V1-0-2]] by [[User:BryanDavis|BryanDavis]] * 01:37 [[etherpad:p/WikipediaMobile-3-1-1-scratchpad]] was archived as [[etherpadbackup:WikipediaMobile-3-1-1-scratchpad]] by [[User:BryanDavis|BryanDavis]] * 01:37 [[etherpad:p/WikipediaLite]] was archived as [[etherpadbackup:WikipediaLite]] by [[User:BryanDavis|BryanDavis]] * 01:36 [[etherpad:p/Wikipedia-iOS-Rejection-Message]] was archived as [[etherpadbackup:Wikipedia-iOS-Rejection-Message]] by [[User:BryanDavis|BryanDavis]] * 01:36 [[etherpad:p/Wikipedia-iOS-3-1-1-issues]] was archived as [[etherpadbackup:Wikipedia-iOS-3-1-1-issues]] by [[User:BryanDavis|BryanDavis]] * 01:35 [[etherpad:p/Wikipedia-iOS-3-1-1]] was archived as [[etherpadbackup:Wikipedia-iOS-3-1-1]] by [[User:BryanDavis|BryanDavis]] * 01:35 [[etherpad:p/Wikipedia-iOS]] was archived as [[etherpadbackup:Wikipedia-iOS]] by [[User:BryanDavis|BryanDavis]] * 01:34 [[etherpad:p/WikipdiaMobileRC4]] was archived as [[etherpadbackup:WikipdiaMobileRC4]] by [[User:BryanDavis|BryanDavis]] * 01:34 [[etherpad:p/WikimediaTelepresence]] was archived as [[etherpadbackup:WikimediaTelepresence]] by [[User:BryanDavis|BryanDavis]] * 01:33 [[etherpad:p/Wikimania_Hackathon_2025_-_Opening_Ceremony]] was archived as [[etherpadbackup:Wikimania_Hackathon_2025_-_Opening_Ceremony]] by [[User:BryanDavis|BryanDavis]] * 01:33 [[etherpad:p/Wikimania_2023_Hackathon]] was archived as [[etherpadbackup:Wikimania_2023_Hackathon]] by [[User:BryanDavis|BryanDavis]] * 01:32 [[etherpad:p/Wikigrok_Q4]] was archived as [[etherpadbackup:Wikigrok_Q4]] by [[User:BryanDavis|BryanDavis]] * 01:32 [[etherpad:p/Wikidata_technical_needs]] was archived as [[etherpadbackup:Wikidata_technical_needs]] by [[User:BryanDavis|BryanDavis]] * 01:31 [[etherpad:p/WikiGrokTest1]] was archived as [[etherpadbackup:WikiGrokTest1]] by [[User:BryanDavis|BryanDavis]] * 01:31 [[etherpad:p/WikiDev16-T114045]] was archived as [[etherpadbackup:WikiDev16-T114045]] by [[User:BryanDavis|BryanDavis]] * 01:30 [[etherpad:p/Watchlist-20beta-20status]] was archived as [[etherpadbackup:Watchlist-20beta-20status]] by [[User:BryanDavis|BryanDavis]] * 01:30 [[etherpad:p/WRN202003]] was archived as [[etherpadbackup:WRN202003]] by [[User:BryanDavis|BryanDavis]] * 01:29 [[etherpad:p/WRN201409]] was archived as [[etherpadbackup:WRN201409]] by [[User:BryanDavis|BryanDavis]] * 01:29 [[etherpad:p/WRN201408]] was archived as [[etherpadbackup:WRN201408]] by [[User:BryanDavis|BryanDavis]] * 01:28 [[etherpad:p/WP_BRD_generally]] was archived as [[etherpadbackup:WP_BRD_generally]] by [[User:BryanDavis|BryanDavis]] * 01:28 [[etherpad:p/WMTC19-T238265]] was archived as [[etherpadbackup:WMTC19-T238265]] by [[User:BryanDavis|BryanDavis]] * 01:27 [[etherpad:p/WMTC19-T238227]] was archived as [[etherpadbackup:WMTC19-T238227]] by [[User:BryanDavis|BryanDavis]] * 01:27 [[etherpad:p/WMTC19-T234662]] was archived as [[etherpadbackup:WMTC19-T234662]] by [[User:BryanDavis|BryanDavis]] * 01:26 [[etherpad:p/WMTC19-T234655]] was archived as [[etherpadbackup:WMTC19-T234655]] by [[User:BryanDavis|BryanDavis]] * 01:26 [[etherpad:p/WMTC19-T234654]] was archived as [[etherpadbackup:WMTC19-T234654]] by [[User:BryanDavis|BryanDavis]] * 01:25 [[etherpad:p/WMTC19-T234649]] was archived as [[etherpadbackup:WMTC19-T234649]] by [[User:BryanDavis|BryanDavis]] * 01:25 [[etherpad:p/WMTC19-T234641]] was archived as [[etherpadbackup:WMTC19-T234641]] by [[User:BryanDavis|BryanDavis]] * 01:24 [[etherpad:p/WMTC19-T234636]] was archived as [[etherpadbackup:WMTC19-T234636]] by [[User:BryanDavis|BryanDavis]] * 01:24 [[etherpad:p/WMTC19-T234632]] was archived as [[etherpadbackup:WMTC19-T234632]] by [[User:BryanDavis|BryanDavis]] * 01:23 [[etherpad:p/WMHack_Feature_Ideas_for_WorkAdventure]] was archived as [[etherpadbackup:WMHack_Feature_Ideas_for_WorkAdventure]] by [[User:BryanDavis|BryanDavis]] * 01:23 [[etherpad:p/WMHack25__Wikimedia_Hackathon_y_Latinoamérica]] was archived as [[etherpadbackup:WMHack25__Wikimedia_Hackathon_y_Latinoamérica]] by [[User:BryanDavis|BryanDavis]] * 01:22 [[etherpad:p/WMF_Research_Office_Hours|Notes]] was archived as [[etherpadbackup:WMF_Research_Office_Hours|Notes]] by [[User:BryanDavis|BryanDavis]] * 01:21 [[etherpad:p/WMCS-techsupport-task]] was archived as [[etherpadbackup:WMCS-techsupport-task]] by [[User:BryanDavis|BryanDavis]] * 01:21 [[etherpad:p/WMCS-infrafoundations-2021-11-24]] was archived as [[etherpadbackup:WMCS-infrafoundations-2021-11-24]] by [[User:BryanDavis|BryanDavis]] * 01:20 [[etherpad:p/WMCS-Renaming-Announce]] was archived as [[etherpadbackup:WMCS-Renaming-Announce]] by [[User:BryanDavis|BryanDavis]] * 01:20 [[etherpad:p/WMCS-PDU-ops-2019-10-24]] was archived as [[etherpadbackup:WMCS-PDU-ops-2019-10-24]] by [[User:BryanDavis|BryanDavis]] * 01:19 [[etherpad:p/WMCS-2026-02-12]] was archived as [[etherpadbackup:WMCS-2026-02-12]] by [[User:BryanDavis|BryanDavis]] * 01:19 [[etherpad:p/WMCS-2025-12-18]] was archived as [[etherpadbackup:WMCS-2025-12-18]] by [[User:BryanDavis|BryanDavis]] * 01:18 [[etherpad:p/WMCS-2025-09-04]] was archived as [[etherpadbackup:WMCS-2025-09-04]] by [[User:BryanDavis|BryanDavis]] * 01:18 [[etherpad:p/WMCS-2025-08-14]] was archived as [[etherpadbackup:WMCS-2025-08-14]] by [[User:BryanDavis|BryanDavis]] * 01:17 [[etherpad:p/WMCS-2024-02-07]] was archived as [[etherpadbackup:WMCS-2024-02-07]] by [[User:BryanDavis|BryanDavis]] * 01:17 [[etherpad:p/WMCS-2023-10-04]] was archived as [[etherpadbackup:WMCS-2023-10-04]] by [[User:BryanDavis|BryanDavis]] * 01:16 [[etherpad:p/WMCS-2023-09-20]] was archived as [[etherpadbackup:WMCS-2023-09-20]] by [[User:BryanDavis|BryanDavis]] * 01:16 [[etherpad:p/WMCS-2023-08-10]] was archived as [[etherpadbackup:WMCS-2023-08-10]] by [[User:BryanDavis|BryanDavis]] * 01:15 [[etherpad:p/WMCS-2023-06-28]] was archived as [[etherpadbackup:WMCS-2023-06-28]] by [[User:BryanDavis|BryanDavis]] * 01:15 [[etherpad:p/WMCS-2023-04-09-tools-k8s-upgrade]] was archived as [[etherpadbackup:WMCS-2023-04-09-tools-k8s-upgrade]] by [[User:BryanDavis|BryanDavis]] * 01:14 [[etherpad:p/WMCS-2022-11-09]] was archived as [[etherpadbackup:WMCS-2022-11-09]] by [[User:BryanDavis|BryanDavis]] * 01:14 [[etherpad:p/WMCS-2022-09-07]] was archived as [[etherpadbackup:WMCS-2022-09-07]] by [[User:BryanDavis|BryanDavis]] * 01:13 [[etherpad:p/WMCS-2022-07-20]] was archived as [[etherpadbackup:WMCS-2022-07-20]] by [[User:BryanDavis|BryanDavis]] * 01:13 [[etherpad:p/WMCS-2022-07-13]] was archived as [[etherpadbackup:WMCS-2022-07-13]] by [[User:BryanDavis|BryanDavis]] * 01:12 [[etherpad:p/WMCS-2022-05-11-tools-k8s-upgrade]] was archived as [[etherpadbackup:WMCS-2022-05-11-tools-k8s-upgrade]] by [[User:BryanDavis|BryanDavis]] * 01:12 [[etherpad:p/WMCS-2021-12-15]] was archived as [[etherpadbackup:WMCS-2021-12-15]] by [[User:BryanDavis|BryanDavis]] * 01:11 [[etherpad:p/WMCS-2021-11-03]] was archived as [[etherpadbackup:WMCS-2021-11-03]] by [[User:BryanDavis|BryanDavis]] * 01:10 [[etherpad:p/WMCS-2020-12-02]] was archived as [[etherpadbackup:WMCS-2020-12-02]] by [[User:BryanDavis|BryanDavis]] * 01:10 [[etherpad:p/WMCS-2020-11-25]] was archived as [[etherpadbackup:WMCS-2020-11-25]] by [[User:BryanDavis|BryanDavis]] * 01:09 [[etherpad:p/WMCS-2020-11-04]] was archived as [[etherpadbackup:WMCS-2020-11-04]] by [[User:BryanDavis|BryanDavis]] * 01:09 [[etherpad:p/WMCS-2020-04-22]] was archived as [[etherpadbackup:WMCS-2020-04-22]] by [[User:BryanDavis|BryanDavis]] * 01:08 [[etherpad:p/WMCS-2020-01-07]] was archived as [[etherpadbackup:WMCS-2020-01-07]] by [[User:BryanDavis|BryanDavis]] * 01:08 [[etherpad:p/WMCS-2019-10-24]] was archived as [[etherpadbackup:WMCS-2019-10-24]] by [[User:BryanDavis|BryanDavis]] * 01:07 [[etherpad:p/WMCS-2019-09-03]] was archived as [[etherpadbackup:WMCS-2019-09-03]] by [[User:BryanDavis|BryanDavis]] * 01:07 [[etherpad:p/WMCS-2019-07-30]] was archived as [[etherpadbackup:WMCS-2019-07-30]] by [[User:BryanDavis|BryanDavis]] * 01:06 [[etherpad:p/WMCS-2019-01-29]] was archived as [[etherpadbackup:WMCS-2019-01-29]] by [[User:BryanDavis|BryanDavis]] * 01:06 [[etherpad:p/WMCS-2018-10-02]] was archived as [[etherpadbackup:WMCS-2018-10-02]] by [[User:BryanDavis|BryanDavis]] * 01:05 [[etherpad:p/WMCS-2018-08-14]] was archived as [[etherpadbackup:WMCS-2018-08-14]] by [[User:BryanDavis|BryanDavis]] * 01:05 [[etherpad:p/WMCS-2018-07-31]] was archived as [[etherpadbackup:WMCS-2018-07-31]] by [[User:BryanDavis|BryanDavis]] * 01:04 [[etherpad:p/WMCS-2018-07-10]] was archived as [[etherpadbackup:WMCS-2018-07-10]] by [[User:BryanDavis|BryanDavis]] * 01:04 [[etherpad:p/WMCS-2018-06-19]] was archived as [[etherpadbackup:WMCS-2018-06-19]] by [[User:BryanDavis|BryanDavis]] * 01:03 [[etherpad:p/WMCS-2018-06-05]] was archived as [[etherpadbackup:WMCS-2018-06-05]] by [[User:BryanDavis|BryanDavis]] * 01:03 [[etherpad:p/WMCS-2018-05-29]] was archived as [[etherpadbackup:WMCS-2018-05-29]] by [[User:BryanDavis|BryanDavis]] * 01:02 [[etherpad:p/WMCS-2018-05-01]] was archived as [[etherpadbackup:WMCS-2018-05-01]] by [[User:BryanDavis|BryanDavis]] * 01:02 [[etherpad:p/WMCS-2018-04-10]] was archived as [[etherpadbackup:WMCS-2018-04-10]] by [[User:BryanDavis|BryanDavis]] * 01:01 [[etherpad:p/WMCS-2018-04-03]] was archived as [[etherpadbackup:WMCS-2018-04-03]] by [[User:BryanDavis|BryanDavis]] * 01:01 [[etherpad:p/WMCS-2018-02-13]] was archived as [[etherpadbackup:WMCS-2018-02-13]] by [[User:BryanDavis|BryanDavis]] * 01:00 [[etherpad:p/WMCS-2018-02-05]] was archived as [[etherpadbackup:WMCS-2018-02-05]] by [[User:BryanDavis|BryanDavis]] * 01:00 [[etherpad:p/WMCS-2018-01-23]] was archived as [[etherpadbackup:WMCS-2018-01-23]] by [[User:BryanDavis|BryanDavis]] * 00:59 [[etherpad:p/WMCS-2018-01-09]] was archived as [[etherpadbackup:WMCS-2018-01-09]] by [[User:BryanDavis|BryanDavis]] * 00:59 [[etherpad:p/WMCS-2017-12-19]] was archived as [[etherpadbackup:WMCS-2017-12-19]] by [[User:BryanDavis|BryanDavis]] * 00:58 [[etherpad:p/WMCS-2017-11-28]] was archived as [[etherpadbackup:WMCS-2017-11-28]] by [[User:BryanDavis|BryanDavis]] * 00:58 [[etherpad:p/WMCS-2017-11-21]] was archived as [[etherpadbackup:WMCS-2017-11-21]] by [[User:BryanDavis|BryanDavis]] * 00:57 [[etherpad:p/WMCS-2017-11-14]] was archived as [[etherpadbackup:WMCS-2017-11-14]] by [[User:BryanDavis|BryanDavis]] * 00:57 [[etherpad:p/WMCS-2017-11-07]] was archived as [[etherpadbackup:WMCS-2017-11-07]] by [[User:BryanDavis|BryanDavis]] * 00:56 [[etherpad:p/WMCS-2017-10-30]] was archived as [[etherpadbackup:WMCS-2017-10-30]] by [[User:BryanDavis|BryanDavis]] * 00:56 [[etherpad:p/WMCS-2017-10-24]] was archived as [[etherpadbackup:WMCS-2017-10-24]] by [[User:BryanDavis|BryanDavis]] * 00:55 [[etherpad:p/WMCS-2017-10-03]] was archived as [[etherpadbackup:WMCS-2017-10-03]] by [[User:BryanDavis|BryanDavis]] * 00:54 [[etherpad:p/WMCS-2017-09-19]] was archived as [[etherpadbackup:WMCS-2017-09-19]] by [[User:BryanDavis|BryanDavis]] * 00:54 [[etherpad:p/WMCS-2017-05-15]] was archived as [[etherpadbackup:WMCS-2017-05-15]] by [[User:BryanDavis|BryanDavis]] * 00:53 [[etherpad:p/WMCS-2017-05-08]] was archived as [[etherpadbackup:WMCS-2017-05-08]] by [[User:BryanDavis|BryanDavis]] * 00:53 [[etherpad:p/WMCS-2017-04-17]] was archived as [[etherpadbackup:WMCS-2017-04-17]] by [[User:BryanDavis|BryanDavis]] * 00:52 [[etherpad:p/WMCS-2017-04-10]] was archived as [[etherpadbackup:WMCS-2017-04-10]] by [[User:BryanDavis|BryanDavis]] * 00:52 [[etherpad:p/WM2024_Day4_Lviv-_Rooms_21+22+23]] was archived as [[etherpadbackup:WM2024_Day4_Lviv-_Rooms_21+22+23]] by [[User:BryanDavis|BryanDavis]] * 00:51 [[etherpad:p/WM2024_Day1_Warsaw-_Rooms_20+24]] was archived as [[etherpadbackup:WM2024_Day1_Warsaw-_Rooms_20+24]] by [[User:BryanDavis|BryanDavis]] * 00:51 [[etherpad:p/WLMshowcase]] was archived as [[etherpadbackup:WLMshowcase]] by [[User:BryanDavis|BryanDavis]] * 00:50 [[etherpad:p/WLMShowcase2]] was archived as [[etherpadbackup:WLMShowcase2]] by [[User:BryanDavis|BryanDavis]] * 00:50 [[etherpad:p/WLMMobile]] was archived as [[etherpadbackup:WLMMobile]] by [[User:BryanDavis|BryanDavis]] * 00:49 [[etherpad:p/WLMMarketToDo]] was archived as [[etherpadbackup:WLMMarketToDo]] by [[User:BryanDavis|BryanDavis]] * 00:49 [[etherpad:p/WLMAppEmail]] was archived as [[etherpadbackup:WLMAppEmail]] by [[User:BryanDavis|BryanDavis]] * 00:48 [[etherpad:p/WLM-on-cluster]] was archived as [[etherpadbackup:WLM-on-cluster]] by [[User:BryanDavis|BryanDavis]] * 00:48 [[etherpad:p/WLM-mobile]] was archived as [[etherpadbackup:WLM-mobile]] by [[User:BryanDavis|BryanDavis]] * 00:47 [[etherpad:p/WLM-API-to-WMF]] was archived as [[etherpadbackup:WLM-API-to-WMF]] by [[User:BryanDavis|BryanDavis]] * 00:47 [[etherpad:p/WLM-2013-07-05]] was archived as [[etherpadbackup:WLM-2013-07-05]] by [[User:BryanDavis|BryanDavis]] * 00:46 [[etherpad:p/VzkZSy0PKK]] was archived as [[etherpadbackup:VzkZSy0PKK]] by [[User:BryanDavis|BryanDavis]] * 00:46 [[etherpad:p/VisualEditorTeamLunchOrder20140327]] was archived as [[etherpadbackup:VisualEditorTeamLunchOrder20140327]] by [[User:BryanDavis|BryanDavis]] * 00:45 [[etherpad:p/VelocityNYC2015-Day1]] was archived as [[etherpadbackup:VelocityNYC2015-Day1]] by [[User:BryanDavis|BryanDavis]] * 00:45 [[etherpad:p/Vagrant-wannabees]] was archived as [[etherpadbackup:Vagrant-wannabees]] by [[User:BryanDavis|BryanDavis]] * 00:44 [[etherpad:p/VPT]] was archived as [[etherpadbackup:VPT]] by [[User:BryanDavis|BryanDavis]] * 00:44 [[etherpad:p/VM_puppet_failures]] was archived as [[etherpadbackup:VM_puppet_failures]] by [[User:BryanDavis|BryanDavis]] * 00:43 [[etherpad:p/VM_puppet_disabled]] was archived as [[etherpadbackup:VM_puppet_disabled]] by [[User:BryanDavis|BryanDavis]] * 00:43 [[etherpad:p/VE_bugs]] was archived as [[etherpadbackup:VE_bugs]] by [[User:BryanDavis|BryanDavis]] * 00:42 [[etherpad:p/VETOC]] was archived as [[etherpadbackup:VETOC]] by [[User:BryanDavis|BryanDavis]] * 00:42 [[etherpad:p/VEQ3Perf]] was archived as [[etherpadbackup:VEQ3Perf]] by [[User:BryanDavis|BryanDavis]] * 00:41 [[etherpad:p/VE-Chromium-Upstream]] was archived as [[etherpadbackup:VE-Chromium-Upstream]] by [[User:BryanDavis|BryanDavis]] * 00:41 [[etherpad:p/UwSprintAugust2012]] was archived as [[etherpadbackup:UwSprintAugust2012]] by [[User:BryanDavis|BryanDavis]] * 00:40 [[etherpad:p/Upload_debugging]] was archived as [[etherpadbackup:Upload_debugging]] by [[User:BryanDavis|BryanDavis]] * 00:39 [[etherpad:p/UploadWizard-and-EventLogging]] was archived as [[etherpadbackup:UploadWizard-and-EventLogging]] by [[User:BryanDavis|BryanDavis]] * 00:39 [[etherpad:p/Upgrade-now-bitches]] was archived as [[etherpadbackup:Upgrade-now-bitches]] by [[User:BryanDavis|BryanDavis]] * 00:38 [[etherpad:p/UniqueServiceGroups]] was archived as [[etherpadbackup:UniqueServiceGroups]] by [[User:BryanDavis|BryanDavis]] * 00:38 [[etherpad:p/Udeme_Emmanson]] was archived as [[etherpadbackup:Udeme_Emmanson]] by [[User:BryanDavis|BryanDavis]] * 00:37 [[etherpad:p/UShQLRyUMv]] was archived as [[etherpadbackup:UShQLRyUMv]] by [[User:BryanDavis|BryanDavis]] * 00:37 [[etherpad:p/UA-SchemaUpdate]] was archived as [[etherpadbackup:UA-SchemaUpdate]] by [[User:BryanDavis|BryanDavis]] * 00:36 [[etherpad:p/U84-IdW1WfD2vZ4ogSCx]] was archived as [[etherpadbackup:U84-IdW1WfD2vZ4ogSCx]] by [[User:BryanDavis|BryanDavis]] * 00:36 [[etherpad:p/U53g0zdCtb]] was archived as [[etherpadbackup:U53g0zdCtb]] by [[User:BryanDavis|BryanDavis]] * 00:35 [[etherpad:p/U]] was archived as [[etherpadbackup:U]] by [[User:BryanDavis|BryanDavis]] * 00:35 [[etherpad:p/Trusty_grid_shutdown]] was archived as [[etherpadbackup:Trusty_grid_shutdown]] by [[User:BryanDavis|BryanDavis]] * 00:34 [[etherpad:p/TransclusionModelIntroduction]] was archived as [[etherpadbackup:TransclusionModelIntroduction]] by [[User:BryanDavis|BryanDavis]] * 00:34 [[etherpad:p/Traffic-2021-08-24]] was archived as [[etherpadbackup:Traffic-2021-08-24]] by [[User:BryanDavis|BryanDavis]] * 00:33 [[etherpad:p/Traffic-2021-08-17]] was archived as [[etherpadbackup:Traffic-2021-08-17]] by [[User:BryanDavis|BryanDavis]] * 00:33 [[etherpad:p/Traffic-2021-02-17]] was archived as [[etherpadbackup:Traffic-2021-02-17]] by [[User:BryanDavis|BryanDavis]] * 00:32 [[etherpad:p/Traffic-2020-12-16]] was archived as [[etherpadbackup:Traffic-2020-12-16]] by [[User:BryanDavis|BryanDavis]] * 00:32 [[etherpad:p/Traffic-2020-11-02]] was archived as [[etherpadbackup:Traffic-2020-11-02]] by [[User:BryanDavis|BryanDavis]] * 00:31 [[etherpad:p/Traffic-2020-10-13]] was archived as [[etherpadbackup:Traffic-2020-10-13]] by [[User:BryanDavis|BryanDavis]] * 00:31 [[etherpad:p/Traffic-2020-09-28]] was archived as [[etherpadbackup:Traffic-2020-09-28]] by [[User:BryanDavis|BryanDavis]] * 00:30 [[etherpad:p/Traffic-2018-06-28]] was archived as [[etherpadbackup:Traffic-2018-06-28]] by [[User:BryanDavis|BryanDavis]] * 00:30 [[etherpad:p/Traffic-2018-06-14]] was archived as [[etherpadbackup:Traffic-2018-06-14]] by [[User:BryanDavis|BryanDavis]] * 00:29 [[etherpad:p/ToolLabs_Survey_feedback]] was archived as [[etherpadbackup:ToolLabs_Survey_feedback]] by [[User:BryanDavis|BryanDavis]] * 00:29 [[etherpad:p/TextInputWidgetValidation]] was archived as [[etherpadbackup:TextInputWidgetValidation]] by [[User:BryanDavis|BryanDavis]] * 00:28 [[etherpad:p/Templates]] was archived as [[etherpadbackup:Templates]] by [[User:BryanDavis|BryanDavis]] * 00:28 [[etherpad:p/Technical_Engagment_All_Hands_2019doing_and_should]] was archived as [[etherpadbackup:Technical_Engagment_All_Hands_2019doing_and_should]] by [[User:BryanDavis|BryanDavis]] * 00:27 [[etherpad:p/Technical_Engagment_All_Hands_2019]] was archived as [[etherpadbackup:Technical_Engagment_All_Hands_2019]] by [[User:BryanDavis|BryanDavis]] * 00:27 [[etherpad:p/TechOps-goals-Q4-FY15-16]] was archived as [[etherpadbackup:TechOps-goals-Q4-FY15-16]] by [[User:BryanDavis|BryanDavis]] * 00:26 [[etherpad:p/TechOps-goals-FQ3-FY1617]] was archived as [[etherpadbackup:TechOps-goals-FQ3-FY1617]] by [[User:BryanDavis|BryanDavis]] * 00:26 [[etherpad:p/TechOps-goals-FQ2-FY1617]] was archived as [[etherpadbackup:TechOps-goals-FQ2-FY1617]] by [[User:BryanDavis|BryanDavis]] * 00:25 [[etherpad:p/TechOps-goals-FQ1-FY1617]] was archived as [[etherpadbackup:TechOps-goals-FQ1-FY1617]] by [[User:BryanDavis|BryanDavis]] * 00:25 [[etherpad:p/TechOps-2017-09-18]] was archived as [[etherpadbackup:TechOps-2017-09-18]] by [[User:BryanDavis|BryanDavis]] * 00:24 [[etherpad:p/TechOps-2015-10-14]] was archived as [[etherpadbackup:TechOps-2015-10-14]] by [[User:BryanDavis|BryanDavis]] * 00:24 [[etherpad:p/TechOps-2015-07-27]] was archived as [[etherpadbackup:TechOps-2015-07-27]] by [[User:BryanDavis|BryanDavis]] * 00:23 [[etherpad:p/TechDaysCareers]] was archived as [[etherpadbackup:TechDaysCareers]] by [[User:BryanDavis|BryanDavis]] * 00:22 [[etherpad:p/TechDays-Mobile]] was archived as [[etherpadbackup:TechDays-Mobile]] by [[User:BryanDavis|BryanDavis]] * 00:22 [[etherpad:p/TargetLoader]] was archived as [[etherpadbackup:TargetLoader]] by [[User:BryanDavis|BryanDavis]] * 00:21 [[etherpad:p/T_movie]] was archived as [[etherpadbackup:T_movie]] by [[User:BryanDavis|BryanDavis]] * 00:21 [[etherpad:p/TFMQF]] was archived as [[etherpadbackup:TFMQF]] by [[User:BryanDavis|BryanDavis]] * 00:20 [[etherpad:p/T404584-announcement]] was archived as [[etherpadbackup:T404584-announcement]] by [[User:BryanDavis|BryanDavis]] * 00:20 [[etherpad:p/T395855]] was archived as [[etherpadbackup:T395855]] by [[User:BryanDavis|BryanDavis]] * 00:19 [[etherpad:p/T372912]] was archived as [[etherpadbackup:T372912]] by [[User:BryanDavis|BryanDavis]] * 00:19 [[etherpad:p/T364459]] was archived as [[etherpadbackup:T364459]] by [[User:BryanDavis|BryanDavis]] * 00:18 [[etherpad:p/T340241-new-fingerprints]] was archived as [[etherpadbackup:T340241-new-fingerprints]] by [[User:BryanDavis|BryanDavis]] * 00:18 [[etherpad:p/T340241-new-bastion]] was archived as [[etherpadbackup:T340241-new-bastion]] by [[User:BryanDavis|BryanDavis]] * 00:17 [[etherpad:p/T221119_-_log]] was archived as [[etherpadbackup:T221119_-_log]] by [[User:BryanDavis|BryanDavis]] * 00:17 [[etherpad:p/T191532]] was archived as [[etherpadbackup:T191532]] by [[User:BryanDavis|BryanDavis]] * 00:16 [[etherpad:p/T182722-outage]] was archived as [[etherpadbackup:T182722-outage]] by [[User:BryanDavis|BryanDavis]] * 00:16 [[etherpad:p/T152122_notes]] was archived as [[etherpadbackup:T152122_notes]] by [[User:BryanDavis|BryanDavis]] * 00:15 [[etherpad:p/T128190]] was archived as [[etherpadbackup:T128190]] by [[User:BryanDavis|BryanDavis]] * 00:15 [[etherpad:p/Swift-Switch-Originals]] was archived as [[etherpadbackup:Swift-Switch-Originals]] by [[User:BryanDavis|BryanDavis]] * 00:14 [[etherpad:p/Summary/_Hypothesis]] was archived as [[etherpadbackup:Summary/_Hypothesis]] by [[User:BryanDavis|BryanDavis]] * 00:14 [[etherpad:p/SquidToVarnish]] was archived as [[etherpadbackup:SquidToVarnish]] by [[User:BryanDavis|BryanDavis]] * 00:13 [[etherpad:p/Sqoop]] was archived as [[etherpadbackup:Sqoop]] by [[User:BryanDavis|BryanDavis]] * 00:13 [[etherpad:p/Sprint_D]] was archived as [[etherpadbackup:Sprint_D]] by [[User:BryanDavis|BryanDavis]] * 00:12 [[etherpad:p/ShutdownPreciseGrid]] was archived as [[etherpadbackup:ShutdownPreciseGrid]] by [[User:BryanDavis|BryanDavis]] * 00:12 [[etherpad:p/ServiceOps-AugustReboots]] was archived as [[etherpadbackup:ServiceOps-AugustReboots]] by [[User:BryanDavis|BryanDavis]] * 00:11 [[etherpad:p/ServiceOps]] was archived as [[etherpadbackup:ServiceOps]] by [[User:BryanDavis|BryanDavis]] * 00:11 [[etherpad:p/Searchfdbksurvey]] was archived as [[etherpadbackup:Searchfdbksurvey]] by [[User:BryanDavis|BryanDavis]] * 00:10 [[etherpad:p/Search_UserStories]] was archived as [[etherpadbackup:Search_UserStories]] by [[User:BryanDavis|BryanDavis]] * 00:10 [[etherpad:p/ScrummasterMeetup-2013-10-10]] was archived as [[etherpadbackup:ScrummasterMeetup-2013-10-10]] by [[User:BryanDavis|BryanDavis]] * 00:09 [[etherpad:p/Scrum-of-Scrums-copypaste]] was archived as [[etherpadbackup:Scrum-of-Scrums-copypaste]] by [[User:BryanDavis|BryanDavis]] * 00:09 [[etherpad:p/SandboxHatesMe]] was archived as [[etherpadbackup:SandboxHatesMe]] by [[User:BryanDavis|BryanDavis]] * 00:08 [[etherpad:p/SWAT_2016-04-14]] was archived as [[etherpadbackup:SWAT_2016-04-14]] by [[User:BryanDavis|BryanDavis]] * 00:08 [[etherpad:p/SWAT]] was archived as [[etherpadbackup:SWAT]] by [[User:BryanDavis|BryanDavis]] * 00:07 [[etherpad:p/SVImxzLIyw]] was archived as [[etherpadbackup:SVImxzLIyw]] by [[User:BryanDavis|BryanDavis]] * 00:07 [[etherpad:p/SRE-goals-FQ4-FY1819]] was archived as [[etherpadbackup:SRE-goals-FQ4-FY1819]] by [[User:BryanDavis|BryanDavis]] * 00:06 [[etherpad:p/SRE-goals-FQ3-FY1819]] was archived as [[etherpadbackup:SRE-goals-FQ3-FY1819]] by [[User:BryanDavis|BryanDavis]] * 00:05 [[etherpad:p/SRE-goals-FQ1-FY1920]] was archived as [[etherpadbackup:SRE-goals-FQ1-FY1920]] by [[User:BryanDavis|BryanDavis]] * 00:05 [[etherpad:p/SRE-ServiceOps-StatusMeeting]] was archived as [[etherpadbackup:SRE-ServiceOps-StatusMeeting]] by [[User:BryanDavis|BryanDavis]] * 00:04 [[etherpad:p/SRE-ServiceOps-2019-01-03]] was archived as [[etherpadbackup:SRE-ServiceOps-2019-01-03]] by [[User:BryanDavis|BryanDavis]] * 00:04 [[etherpad:p/SRE-Foundations-2021-10-25]] was archived as [[etherpadbackup:SRE-Foundations-2021-10-25]] by [[User:BryanDavis|BryanDavis]] * 00:03 [[etherpad:p/SRE-Foundations-2020-07-01]] was archived as [[etherpadbackup:SRE-Foundations-2020-07-01]] by [[User:BryanDavis|BryanDavis]] * 00:03 [[etherpad:p/SRE-Foundations-2020-03-11]] was archived as [[etherpadbackup:SRE-Foundations-2020-03-11]] by [[User:BryanDavis|BryanDavis]] * 00:02 [[etherpad:p/SRE-Foundations-2020-02-12]] was archived as [[etherpadbackup:SRE-Foundations-2020-02-12]] by [[User:BryanDavis|BryanDavis]] * 00:02 [[etherpad:p/SRE-Foundations-2019-11-13]] was archived as [[etherpadbackup:SRE-Foundations-2019-11-13]] by [[User:BryanDavis|BryanDavis]] * 00:01 [[etherpad:p/SRE-Foundations-2019-08-07]] was archived as [[etherpadbackup:SRE-Foundations-2019-08-07]] by [[User:BryanDavis|BryanDavis]] * 00:01 [[etherpad:p/SRE-Foundations-2019-07-31]] was archived as [[etherpadbackup:SRE-Foundations-2019-07-31]] by [[User:BryanDavis|BryanDavis]] * 00:00 [[etherpad:p/SOAAuth]] was archived as [[etherpadbackup:SOAAuth]] by [[User:BryanDavis|BryanDavis]] * 00:00 [[etherpad:p/SGE_rebuild]] was archived as [[etherpadbackup:SGE_rebuild]] by [[User:BryanDavis|BryanDavis]] === 2026-05-05 === * 23:59 [[etherpad:p/S]] was archived as [[etherpadbackup:S]] by [[User:BryanDavis|BryanDavis]] * 23:59 [[etherpad:p/RobotsMetaTag]] was archived as [[etherpadbackup:RobotsMetaTag]] by [[User:BryanDavis|BryanDavis]] * 23:58 [[etherpad:p/ResourceLoader-for-mobile]] was archived as [[etherpadbackup:ResourceLoader-for-mobile]] by [[User:BryanDavis|BryanDavis]] * 23:58 [[etherpad:p/ResourceLoader-Mobile]] was archived as [[etherpadbackup:ResourceLoader-Mobile]] by [[User:BryanDavis|BryanDavis]] * 23:57 [[etherpad:p/ResourceLoader]] was archived as [[etherpadbackup:ResourceLoader]] by [[User:BryanDavis|BryanDavis]] * 23:57 [[etherpad:p/ReleaseNotes]] was archived as [[etherpadbackup:ReleaseNotes]] by [[User:BryanDavis|BryanDavis]] * 23:56 [[etherpad:p/ReferenceInspector]] was archived as [[etherpadbackup:ReferenceInspector]] by [[User:BryanDavis|BryanDavis]] * 23:56 [[etherpad:p/Reading_web_sprint_2_names]] was archived as [[etherpadbackup:Reading_web_sprint_2_names]] by [[User:BryanDavis|BryanDavis]] * 23:55 [[etherpad:p/ReadingWebQ3Planning]] was archived as [[etherpadbackup:ReadingWebQ3Planning]] by [[User:BryanDavis|BryanDavis]] * 23:55 [[etherpad:p/ReadingWebCodeReviewSessions]] was archived as [[etherpadbackup:ReadingWebCodeReviewSessions]] by [[User:BryanDavis|BryanDavis]] * 23:54 [[etherpad:p/ReadingShowcase]] was archived as [[etherpadbackup:ReadingShowcase]] by [[User:BryanDavis|BryanDavis]] * 23:54 [[etherpad:p/Reading]] was archived as [[etherpadbackup:Reading]] by [[User:BryanDavis|BryanDavis]] * 23:53 [[etherpad:p/RdvI3eLIj5AMM4PN_hHz]] was archived as [[etherpadbackup:RdvI3eLIj5AMM4PN_hHz]] by [[User:BryanDavis|BryanDavis]] * 23:53 [[etherpad:p/RWW-WikipediaZero]] was archived as [[etherpadbackup:RWW-WikipediaZero]] by [[User:BryanDavis|BryanDavis]] * 23:52 [[etherpad:p/RLRLRL]] was archived as [[etherpadbackup:RLRLRL]] by [[User:BryanDavis|BryanDavis]] * 23:52 [[etherpad:p/RLFuncDeps]] was archived as [[etherpadbackup:RLFuncDeps]] by [[User:BryanDavis|BryanDavis]] * 23:51 [[etherpad:p/RLExceptions]] was archived as [[etherpadbackup:RLExceptions]] by [[User:BryanDavis|BryanDavis]] * 23:51 [[etherpad:p/RL-mwloader-testcases-tmp]] was archived as [[etherpadbackup:RL-mwloader-testcases-tmp]] by [[User:BryanDavis|BryanDavis]] * 23:50 [[etherpad:p/RL-mwloader-testcases]] was archived as [[etherpadbackup:RL-mwloader-testcases]] by [[User:BryanDavis|BryanDavis]] * 23:50 [[etherpad:p/RL-async-head]] was archived as [[etherpadbackup:RL-async-head]] by [[User:BryanDavis|BryanDavis]] * 23:49 [[etherpad:p/RD201412]] was archived as [[etherpadbackup:RD201412]] by [[User:BryanDavis|BryanDavis]] * 23:48 [[etherpad:p/RD201411]] was archived as [[etherpadbackup:RD201411]] by [[User:BryanDavis|BryanDavis]] * 23:48 [[etherpad:p/RD201409]] was archived as [[etherpadbackup:RD201409]] by [[User:BryanDavis|BryanDavis]] * 23:47 [[etherpad:p/RD2014]] was archived as [[etherpadbackup:RD2014]] by [[User:BryanDavis|BryanDavis]] * 23:47 [[etherpad:p/Quarterly_metrics_scorecard]] was archived as [[etherpadbackup:Quarterly_metrics_scorecard]] by [[User:BryanDavis|BryanDavis]] * 23:46 [[etherpad:p/QgiZvx5mWMQzlM03zCC1]] was archived as [[etherpadbackup:QgiZvx5mWMQzlM03zCC1]] by [[User:BryanDavis|BryanDavis]] * 23:46 [[etherpad:p/QR1]] was archived as [[etherpadbackup:QR1]] by [[User:BryanDavis|BryanDavis]] * 23:45 [[etherpad:p/QPP]] was archived as [[etherpadbackup:QPP]] by [[User:BryanDavis|BryanDavis]] * 23:45 [[etherpad:p/QA_Automation_sync_up]] was archived as [[etherpadbackup:QA_Automation_sync_up]] by [[User:BryanDavis|BryanDavis]] * 23:44 [[etherpad:p/Q]] was archived as [[etherpadbackup:Q]] by [[User:BryanDavis|BryanDavis]] * 23:44 [[etherpad:p/Puppet3]] was archived as [[etherpadbackup:Puppet3]] by [[User:BryanDavis|BryanDavis]] * 23:43 [[etherpad:p/ProofreadPageLua]] was archived as [[etherpadbackup:ProofreadPageLua]] by [[User:BryanDavis|BryanDavis]] * 23:43 [[etherpad:p/Phragile-requirements-2015-05]] was archived as [[etherpadbackup:Phragile-requirements-2015-05]] by [[User:BryanDavis|BryanDavis]] * 23:42 [[etherpad:p/Phabricator_Upgrade_Planning_20190123]] was archived as [[etherpadbackup:Phabricator_Upgrade_Planning_20190123]] by [[User:BryanDavis|BryanDavis]] * 23:42 [[etherpad:p/PhabricatorInstall]] was archived as [[etherpadbackup:PhabricatorInstall]] by [[User:BryanDavis|BryanDavis]] * 23:41 [[etherpad:p/PerformanceUpdate201601]] was archived as [[etherpadbackup:PerformanceUpdate201601]] by [[User:BryanDavis|BryanDavis]] * 23:41 [[etherpad:p/PerformanceDecemberUpdate]] was archived as [[etherpadbackup:PerformanceDecemberUpdate]] by [[User:BryanDavis|BryanDavis]] * 23:40 [[etherpad:p/Performance-2015-12-14]] was archived as [[etherpadbackup:Performance-2015-12-14]] by [[User:BryanDavis|BryanDavis]] * 23:40 [[etherpad:p/Performance]] was archived as [[etherpadbackup:Performance]] by [[User:BryanDavis|BryanDavis]] * 23:39 [[etherpad:p/PartnerEngineerTraining]] was archived as [[etherpadbackup:PartnerEngineerTraining]] by [[User:BryanDavis|BryanDavis]] * 23:39 [[etherpad:p/PV_Analytics_Infra]] was archived as [[etherpadbackup:PV_Analytics_Infra]] by [[User:BryanDavis|BryanDavis]] * 23:38 [[etherpad:p/PVTransition]] was archived as [[etherpadbackup:PVTransition]] by [[User:BryanDavis|BryanDavis]] * 23:38 [[etherpad:p/PAWS-2020-04-28]] was archived as [[etherpadbackup:PAWS-2020-04-28]] by [[User:BryanDavis|BryanDavis]] * 23:37 [[etherpad:p/P&EDashboard20160112]] was archived as [[etherpadbackup:P&EDashboard20160112]] by [[User:BryanDavis|BryanDavis]] * 23:37 [[etherpad:p/P]] was archived as [[etherpadbackup:P]] by [[User:BryanDavis|BryanDavis]] * 23:36 [[etherpad:p/OsJfzQH80T]] was archived as [[etherpadbackup:OsJfzQH80T]] by [[User:BryanDavis|BryanDavis]] * 23:36 [[etherpad:p/OpsSummit2013Day2]] was archived as [[etherpadbackup:OpsSummit2013Day2]] by [[User:BryanDavis|BryanDavis]] * 23:35 [[etherpad:p/OldWmfBranchCleanup]] was archived as [[etherpadbackup:OldWmfBranchCleanup]] by [[User:BryanDavis|BryanDavis]] * 23:35 [[etherpad:p/OctoberMetricsAgenda]] was archived as [[etherpadbackup:OctoberMetricsAgenda]] by [[User:BryanDavis|BryanDavis]] * 23:34 [[etherpad:p/OSM-Wikimedia]] was archived as [[etherpadbackup:OSM-Wikimedia]] by [[User:BryanDavis|BryanDavis]] * 23:34 [[etherpad:p/OOjsUI-TODO]] was archived as [[etherpadbackup:OOjsUI-TODO]] by [[User:BryanDavis|BryanDavis]] * 23:33 [[etherpad:p/NewMobileTech]] was archived as [[etherpadbackup:NewMobileTech]] by [[User:BryanDavis|BryanDavis]] * 23:33 [[etherpad:p/NewDeploysToday]] was archived as [[etherpadbackup:NewDeploysToday]] by [[User:BryanDavis|BryanDavis]] * 23:32 [[etherpad:p/N_sprint]] was archived as [[etherpadbackup:N_sprint]] by [[User:BryanDavis|BryanDavis]] * 23:32 [[etherpad:p/NQpvqJeiKg]] was archived as [[etherpadbackup:NQpvqJeiKg]] by [[User:BryanDavis|BryanDavis]] * 23:31 [[etherpad:p/N5q9EmRswClmLQyBOezv]] was archived as [[etherpadbackup:N5q9EmRswClmLQyBOezv]] by [[User:BryanDavis|BryanDavis]] * 23:31 [[etherpad:p/Mobile_at_Wikimania_2015]] was archived as [[etherpadbackup:Mobile_at_Wikimania_2015]] by [[User:BryanDavis|BryanDavis]] * 23:30 [[etherpad:p/Mobile_Parsoid]] was archived as [[etherpadbackup:Mobile_Parsoid]] by [[User:BryanDavis|BryanDavis]] * 23:29 [[etherpad:p/MobileWeb_Iteration_name]] was archived as [[etherpadbackup:MobileWeb_Iteration_name]] by [[User:BryanDavis|BryanDavis]] * 23:29 [[etherpad:p/MobileWebOnboarding]] was archived as [[etherpadbackup:MobileWebOnboarding]] by [[User:BryanDavis|BryanDavis]] * 23:28 [[etherpad:p/MobileWebDevelopment]] was archived as [[etherpadbackup:MobileWebDevelopment]] by [[User:BryanDavis|BryanDavis]] * 23:28 [[etherpad:p/MobileWeb-Retrospective]] was archived as [[etherpadbackup:MobileWeb-Retrospective]] by [[User:BryanDavis|BryanDavis]] * 23:27 [[etherpad:p/MobileWeb-Q4-Planning]] was archived as [[etherpadbackup:MobileWeb-Q4-Planning]] by [[User:BryanDavis|BryanDavis]] * 23:27 [[etherpad:p/MobileWeb-Q3-Planning]] was archived as [[etherpadbackup:MobileWeb-Q3-Planning]] by [[User:BryanDavis|BryanDavis]] * 23:26 [[etherpad:p/MobileWeb-Q2-Retrospective]] was archived as [[etherpadbackup:MobileWeb-Q2-Retrospective]] by [[User:BryanDavis|BryanDavis]] * 23:26 [[etherpad:p/MobileWeb-IterationName]] was archived as [[etherpadbackup:MobileWeb-IterationName]] by [[User:BryanDavis|BryanDavis]] * 23:25 [[etherpad:p/MobileWeb-CodeReview-Discussion]] was archived as [[etherpadbackup:MobileWeb-CodeReview-Discussion]] by [[User:BryanDavis|BryanDavis]] * 23:25 [[etherpad:p/MobileWatchlist]] was archived as [[etherpadbackup:MobileWatchlist]] by [[User:BryanDavis|BryanDavis]] * 23:24 [[etherpad:p/MobileUX]] was archived as [[etherpadbackup:MobileUX]] by [[User:BryanDavis|BryanDavis]] * 23:24 [[etherpad:p/MobileTutorial]] was archived as [[etherpadbackup:MobileTutorial]] by [[User:BryanDavis|BryanDavis]] * 23:23 [[etherpad:p/MobileToCore]] was archived as [[etherpadbackup:MobileToCore]] by [[User:BryanDavis|BryanDavis]] * 23:23 [[etherpad:p/MobileTechDays]] was archived as [[etherpadbackup:MobileTechDays]] by [[User:BryanDavis|BryanDavis]] * 23:22 [[etherpad:p/MobileTeamLunchHangout]] was archived as [[etherpadbackup:MobileTeamLunchHangout]] by [[User:BryanDavis|BryanDavis]] * 23:22 [[etherpad:p/MobileTeamHangout]] was archived as [[etherpadbackup:MobileTeamHangout]] by [[User:BryanDavis|BryanDavis]] * 23:21 [[etherpad:p/MobileShowcase]] was archived as [[etherpadbackup:MobileShowcase]] by [[User:BryanDavis|BryanDavis]] * 23:21 [[etherpad:p/MobileResourcePlanning]] was archived as [[etherpadbackup:MobileResourcePlanning]] by [[User:BryanDavis|BryanDavis]] * 23:20 [[etherpad:p/MobileQ1_2014_2015]] was archived as [[etherpadbackup:MobileQ1_2014_2015]] by [[User:BryanDavis|BryanDavis]] * 23:20 [[etherpad:p/MobileProjectPlanning-Sept2012]] was archived as [[etherpadbackup:MobileProjectPlanning-Sept2012]] by [[User:BryanDavis|BryanDavis]] * 23:19 [[etherpad:p/MobilePageLoadStats]] was archived as [[etherpadbackup:MobilePageLoadStats]] by [[User:BryanDavis|BryanDavis]] * 23:19 [[etherpad:p/MobileOps]] was archived as [[etherpadbackup:MobileOps]] by [[User:BryanDavis|BryanDavis]] * 23:18 [[etherpad:p/MobileOperatingSystems]] was archived as [[etherpadbackup:MobileOperatingSystems]] by [[User:BryanDavis|BryanDavis]] * 23:18 [[etherpad:p/MobileFrontendCore]] was archived as [[etherpadbackup:MobileFrontendCore]] by [[User:BryanDavis|BryanDavis]] * 23:17 [[etherpad:p/MobileFrontendChanges]] was archived as [[etherpadbackup:MobileFrontendChanges]] by [[User:BryanDavis|BryanDavis]] * 23:17 [[etherpad:p/MobileFrontend-core]] was archived as [[etherpadbackup:MobileFrontend-core]] by [[User:BryanDavis|BryanDavis]] * 23:16 [[etherpad:p/MobileFrontend-Deployment-Issues-20120423]] was archived as [[etherpadbackup:MobileFrontend-Deployment-Issues-20120423]] by [[User:BryanDavis|BryanDavis]] * 23:16 [[etherpad:p/MobileFrontend-20120507-deployment-issues]] was archived as [[etherpadbackup:MobileFrontend-20120507-deployment-issues]] by [[User:BryanDavis|BryanDavis]] * 23:15 [[etherpad:p/MobileDeploymentPainPoints]] was archived as [[etherpadbackup:MobileDeploymentPainPoints]] by [[User:BryanDavis|BryanDavis]] * 23:15 [[etherpad:p/MobileChangeLog]] was archived as [[etherpadbackup:MobileChangeLog]] by [[User:BryanDavis|BryanDavis]] * 23:14 [[etherpad:p/MobileAsks]] was archived as [[etherpadbackup:MobileAsks]] by [[User:BryanDavis|BryanDavis]] * 23:13 [[etherpad:p/MobileArchitectureReview]] was archived as [[etherpadbackup:MobileArchitectureReview]] by [[User:BryanDavis|BryanDavis]] * 23:13 [[etherpad:p/MobileAppTeam]] was archived as [[etherpadbackup:MobileAppTeam]] by [[User:BryanDavis|BryanDavis]] * 23:12 [[etherpad:p/MobileAppRetrospective]] was archived as [[etherpadbackup:MobileAppRetrospective]] by [[User:BryanDavis|BryanDavis]] * 23:12 [[etherpad:p/MobileAppReleaseManger]] was archived as [[etherpadbackup:MobileAppReleaseManger]] by [[User:BryanDavis|BryanDavis]] * 23:11 [[etherpad:p/Mobile23Oct12]] was archived as [[etherpadbackup:Mobile23Oct12]] by [[User:BryanDavis|BryanDavis]] * 23:11 [[etherpad:p/Mobile-WMDS]] was archived as [[etherpadbackup:Mobile-WMDS]] by [[User:BryanDavis|BryanDavis]] * 23:10 [[etherpad:p/Mobile-VE-Integration-Planning]] was archived as [[etherpadbackup:Mobile-VE-Integration-Planning]] by [[User:BryanDavis|BryanDavis]] * 23:10 [[etherpad:p/Mobile-RL]] was archived as [[etherpadbackup:Mobile-RL]] by [[User:BryanDavis|BryanDavis]] * 23:09 [[etherpad:p/Mobile-20navigation-20planning]] was archived as [[etherpadbackup:Mobile-20navigation-20planning]] by [[User:BryanDavis|BryanDavis]] * 23:09 [[etherpad:p/Mm3uNugLtk]] was archived as [[etherpadbackup:Mm3uNugLtk]] by [[User:BryanDavis|BryanDavis]] * 23:08 [[etherpad:p/Metrics_page]] was archived as [[etherpadbackup:Metrics_page]] by [[User:BryanDavis|BryanDavis]] * 23:08 [[etherpad:p/Meetup]] was archived as [[etherpadbackup:Meetup]] by [[User:BryanDavis|BryanDavis]] * 23:07 [[etherpad:p/MediaWiki_Workshop_Buea_-_Participants_emails]] was archived as [[etherpadbackup:MediaWiki_Workshop_Buea_-_Participants_emails]] by [[User:BryanDavis|BryanDavis]] * 23:07 [[etherpad:p/MediaWiki_UI_Standardization_hack4]] was archived as [[etherpadbackup:MediaWiki_UI_Standardization_hack4]] by [[User:BryanDavis|BryanDavis]] * 23:06 [[etherpad:p/McUX]] was archived as [[etherpadbackup:McUX]] by [[User:BryanDavis|BryanDavis]] * 23:06 [[etherpad:p/Maps_pad]] was archived as [[etherpadbackup:Maps_pad]] by [[User:BryanDavis|BryanDavis]] * 23:05 [[etherpad:p/MWDS2015_multi-device_world]] was archived as [[etherpadbackup:MWDS2015_multi-device_world]] by [[User:BryanDavis|BryanDavis]] * 23:05 [[etherpad:p/MWDS2015_developer_documentation]] was archived as [[etherpadbackup:MWDS2015_developer_documentation]] by [[User:BryanDavis|BryanDavis]] * 23:04 [[etherpad:p/MWCore2015Jan]] was archived as [[etherpadbackup:MWCore2015Jan]] by [[User:BryanDavis|BryanDavis]] * 23:04 [[etherpad:p/MFCommitGuidelines]] was archived as [[etherpadbackup:MFCommitGuidelines]] by [[User:BryanDavis|BryanDavis]] * 23:03 [[etherpad:p/MF-betalabs-planning]] was archived as [[etherpadbackup:MF-betalabs-planning]] by [[User:BryanDavis|BryanDavis]] * 23:03 [[etherpad:p/MDS_2015_SOA_plenary]] was archived as [[etherpadbackup:MDS_2015_SOA_plenary]] by [[User:BryanDavis|BryanDavis]] * 23:02 [[etherpad:p/MDS_2015_SOA_and_operations]] was archived as [[etherpadbackup:MDS_2015_SOA_and_operations]] by [[User:BryanDavis|BryanDavis]] * 23:02 [[etherpad:p/Luke081515]] was archived as [[etherpadbackup:Luke081515]] by [[User:BryanDavis|BryanDavis]] * 23:01 [[etherpad:p/Logout_Confirmation_DRC]] was archived as [[etherpadbackup:Logout_Confirmation_DRC]] by [[User:BryanDavis|BryanDavis]] * 23:01 [[etherpad:p/LibrarizationRFC]] was archived as [[etherpadbackup:LibrarizationRFC]] by [[User:BryanDavis|BryanDavis]] * 23:00 [[etherpad:p/Libera-channel-queue]] was archived as [[etherpadbackup:Libera-channel-queue]] by [[User:BryanDavis|BryanDavis]] * 23:00 [[etherpad:p/Legends_of_Lavanya]] was archived as [[etherpadbackup:Legends_of_Lavanya]] by [[User:BryanDavis|BryanDavis]] * 22:59 [[etherpad:p/LabsUpgradePlans]] was archived as [[etherpadbackup:LabsUpgradePlans]] by [[User:BryanDavis|BryanDavis]] * 22:59 [[etherpad:p/LabsIrcConf]] was archived as [[etherpadbackup:LabsIrcConf]] by [[User:BryanDavis|BryanDavis]] * 22:58 [[etherpad:p/LabsAPI]] was archived as [[etherpadbackup:LabsAPI]] by [[User:BryanDavis|BryanDavis]] * 22:58 [[etherpad:p/Labs-Incident-Report-20015-03-31]] was archived as [[etherpadbackup:Labs-Incident-Report-20015-03-31]] by [[User:BryanDavis|BryanDavis]] * 22:57 [[etherpad:p/Labs-2017-03-14]] was archived as [[etherpadbackup:Labs-2017-03-14]] by [[User:BryanDavis|BryanDavis]] * 22:57 [[etherpad:p/Labs-2017-03-06]] was archived as [[etherpadbackup:Labs-2017-03-06]] by [[User:BryanDavis|BryanDavis]] * 22:56 [[etherpad:p/Labs-2017-02-22]] was archived as [[etherpadbackup:Labs-2017-02-22]] by [[User:BryanDavis|BryanDavis]] * 22:56 [[etherpad:p/Labs-2017-01-23]] was archived as [[etherpadbackup:Labs-2017-01-23]] by [[User:BryanDavis|BryanDavis]] * 22:55 [[etherpad:p/Labs-2016-12-12]] was archived as [[etherpadbackup:Labs-2016-12-12]] by [[User:BryanDavis|BryanDavis]] * 22:54 [[etherpad:p/Labs-2016-11-07]] was archived as [[etherpadbackup:Labs-2016-11-07]] by [[User:BryanDavis|BryanDavis]] * 22:54 [[etherpad:p/Labs-2016-10-31]] was archived as [[etherpadbackup:Labs-2016-10-31]] by [[User:BryanDavis|BryanDavis]] * 22:53 [[etherpad:p/Labs-2016-09-19]] was archived as [[etherpadbackup:Labs-2016-09-19]] by [[User:BryanDavis|BryanDavis]] * 22:53 [[etherpad:p/Labs-2016-09-06]] was archived as [[etherpadbackup:Labs-2016-09-06]] by [[User:BryanDavis|BryanDavis]] * 22:52 [[etherpad:p/Labs-2016-08-29]] was archived as [[etherpadbackup:Labs-2016-08-29]] by [[User:BryanDavis|BryanDavis]] * 22:52 [[etherpad:p/Labs-2016-08-01]] was archived as [[etherpadbackup:Labs-2016-08-01]] by [[User:BryanDavis|BryanDavis]] * 22:51 [[etherpad:p/Labs-2016-07-25]] was archived as [[etherpadbackup:Labs-2016-07-25]] by [[User:BryanDavis|BryanDavis]] * 22:51 [[etherpad:p/Labs-2016-07-11]] was archived as [[etherpadbackup:Labs-2016-07-11]] by [[User:BryanDavis|BryanDavis]] * 22:50 [[etherpad:p/Labs-2016-06-06]] was archived as [[etherpadbackup:Labs-2016-06-06]] by [[User:BryanDavis|BryanDavis]] * 22:50 [[etherpad:p/Labs-2016-05-23]] was archived as [[etherpadbackup:Labs-2016-05-23]] by [[User:BryanDavis|BryanDavis]] * 22:49 [[etherpad:p/Labs-2016-05-16]] was archived as [[etherpadbackup:Labs-2016-05-16]] by [[User:BryanDavis|BryanDavis]] * 22:49 [[etherpad:p/Labs-2016-05-09]] was archived as [[etherpadbackup:Labs-2016-05-09]] by [[User:BryanDavis|BryanDavis]] * 22:48 [[etherpad:p/Labs-2016-05-02]] was archived as [[etherpadbackup:Labs-2016-05-02]] by [[User:BryanDavis|BryanDavis]] * 22:48 [[etherpad:p/Labs-2016-04-25]] was archived as [[etherpadbackup:Labs-2016-04-25]] by [[User:BryanDavis|BryanDavis]] * 22:47 [[etherpad:p/Labs-20150602]] was archived as [[etherpadbackup:Labs-20150602]] by [[User:BryanDavis|BryanDavis]] * 22:47 [[etherpad:p/L]] was archived as [[etherpadbackup:L]] by [[User:BryanDavis|BryanDavis]] * 22:46 [[etherpad:p/Kay@31]] was archived as [[etherpadbackup:Kay@31]] by [[User:BryanDavis|BryanDavis]] * 22:46 [[etherpad:p/Kafka-udp2log]] was archived as [[etherpadbackup:Kafka-udp2log]] by [[User:BryanDavis|BryanDavis]] * 22:45 [[etherpad:p/K]] was archived as [[etherpadbackup:K]] by [[User:BryanDavis|BryanDavis]] * 22:45 [[etherpad:p/J]] was archived as [[etherpadbackup:J]] by [[User:BryanDavis|BryanDavis]] * 22:44 [[etherpad:p/Introduction_to_Phabricator_(WikiCon)]] was archived as [[etherpadbackup:Introduction_to_Phabricator_(WikiCon)]] by [[User:BryanDavis|BryanDavis]] * 22:44 [[etherpad:p/Intro_a_las_tec_del_movimiento_Wikimedia]] was archived as [[etherpadbackup:Intro_a_las_tec_del_movimiento_Wikimedia]] by [[User:BryanDavis|BryanDavis]] * 22:43 [[etherpad:p/InlineScripting]] was archived as [[etherpadbackup:InlineScripting]] by [[User:BryanDavis|BryanDavis]] * 22:43 [[etherpad:p/Incident-Replay-Sessions]] was archived as [[etherpadbackup:Incident-Replay-Sessions]] by [[User:BryanDavis|BryanDavis]] * 22:42 [[etherpad:p/IlszcZZEEq]] was archived as [[etherpadbackup:IlszcZZEEq]] by [[User:BryanDavis|BryanDavis]] * 22:42 [[etherpad:p/Icon_standardisation]] was archived as [[etherpadbackup:Icon_standardisation]] by [[User:BryanDavis|BryanDavis]] * 22:41 [[etherpad:p/INGLW4jCQk]] was archived as [[etherpadbackup:INGLW4jCQk]] by [[User:BryanDavis|BryanDavis]] * 22:41 [[etherpad:p/IBrokeWikipediaList]] was archived as [[etherpadbackup:IBrokeWikipediaList]] by [[User:BryanDavis|BryanDavis]] * 22:40 [[etherpad:p/I]] was archived as [[etherpadbackup:I]] by [[User:BryanDavis|BryanDavis]] * 22:39 [[etherpad:p/HuggleCon-2-planning]] was archived as [[etherpadbackup:HuggleCon-2-planning]] by [[User:BryanDavis|BryanDavis]] * 22:39 [[etherpad:p/HuggleCon-1-participants-list]] was archived as [[etherpadbackup:HuggleCon-1-participants-list]] by [[User:BryanDavis|BryanDavis]] * 22:38 [[etherpad:p/Huggle-dev-pad]] was archived as [[etherpadbackup:Huggle-dev-pad]] by [[User:BryanDavis|BryanDavis]] * 22:38 [[etherpad:p/Huggle-IRC-conference]] was archived as [[etherpadbackup:Huggle-IRC-conference]] by [[User:BryanDavis|BryanDavis]] * 22:37 [[etherpad:p/Huggle]] was archived as [[etherpadbackup:Huggle]] by [[User:BryanDavis|BryanDavis]] * 22:37 [[etherpad:p/HowToReleaseAnApp]] was archived as [[etherpadbackup:HowToReleaseAnApp]] by [[User:BryanDavis|BryanDavis]] * 22:36 [[etherpad:p/Hiera]] was archived as [[etherpadbackup:Hiera]] by [[User:BryanDavis|BryanDavis]] * 22:36 [[etherpad:p/HadoopEtherpad]] was archived as [[etherpadbackup:HadoopEtherpad]] by [[User:BryanDavis|BryanDavis]] * 22:35 [[etherpad:p/Hackathon_Showcase_&_Coolest_Tool_Awards]] was archived as [[etherpadbackup:Hackathon_Showcase_&_Coolest_Tool_Awards]] by [[User:BryanDavis|BryanDavis]] * 22:35 [[etherpad:p/HKzL6x5e6z]] was archived as [[etherpadbackup:HKzL6x5e6z]] by [[User:BryanDavis|BryanDavis]] * 22:34 [[etherpad:p/HJRCD1b81bHXm-ZS-l9w]] was archived as [[etherpadbackup:HJRCD1b81bHXm-ZS-l9w]] by [[User:BryanDavis|BryanDavis]] * 22:34 [[etherpad:p/Gwods9UzuoEMR79NM8nl]] was archived as [[etherpadbackup:Gwods9UzuoEMR79NM8nl]] by [[User:BryanDavis|BryanDavis]] * 22:33 [[etherpad:p/GroupSelectOption]] was archived as [[etherpadbackup:GroupSelectOption]] by [[User:BryanDavis|BryanDavis]] * 22:33 [[etherpad:p/Google_Slack_Zoom_YouTube_bugs_in_Movement]] was archived as [[etherpadbackup:Google_Slack_Zoom_YouTube_bugs_in_Movement]] by [[User:BryanDavis|BryanDavis]] * 22:32 [[etherpad:p/GitWorkflow]] was archived as [[etherpadbackup:GitWorkflow]] by [[User:BryanDavis|BryanDavis]] * 22:32 [[etherpad:p/Git-migration-tools]] was archived as [[etherpadbackup:Git-migration-tools]] by [[User:BryanDavis|BryanDavis]] * 22:31 [[etherpad:p/Git-migration]] was archived as [[etherpadbackup:Git-migration]] by [[User:BryanDavis|BryanDavis]] * 22:31 [[etherpad:p/Git-Feb13]] was archived as [[etherpadbackup:Git-Feb13]] by [[User:BryanDavis|BryanDavis]] * 22:30 [[etherpad:p/Gerrit_login_problems]] was archived as [[etherpadbackup:Gerrit_login_problems]] by [[User:BryanDavis|BryanDavis]] * 22:30 [[etherpad:p/Gerrit_API_Wikimedia]] was archived as [[etherpadbackup:Gerrit_API_Wikimedia]] by [[User:BryanDavis|BryanDavis]] * 22:29 [[etherpad:p/GerritCleanupOrg]] was archived as [[etherpadbackup:GerritCleanupOrg]] by [[User:BryanDavis|BryanDavis]] * 22:29 [[etherpad:p/GerritAccounts]] was archived as [[etherpadbackup:GerritAccounts]] by [[User:BryanDavis|BryanDavis]] * 22:28 [[etherpad:p/GeoData]] was archived as [[etherpadbackup:GeoData]] by [[User:BryanDavis|BryanDavis]] * 22:28 [[etherpad:p/Gather_Q4]] was archived as [[etherpadbackup:Gather_Q4]] by [[User:BryanDavis|BryanDavis]] * 22:27 [[etherpad:p/G__________________]] was archived as [[etherpadbackup:G__________________]] by [[User:BryanDavis|BryanDavis]] * 22:27 [[etherpad:p/GWToolsetReview]] was archived as [[etherpadbackup:GWToolsetReview]] by [[User:BryanDavis|BryanDavis]] * 22:26 [[etherpad:p/G]] was archived as [[etherpadbackup:G]] by [[User:BryanDavis|BryanDavis]] * 22:26 [[etherpad:p/FutureOfEditingTalk]] was archived as [[etherpadbackup:FutureOfEditingTalk]] by [[User:BryanDavis|BryanDavis]] * 22:25 [[etherpad:p/FrontEndDevOnboarding-bahodir]] was archived as [[etherpadbackup:FrontEndDevOnboarding-bahodir]] by [[User:BryanDavis|BryanDavis]] * 22:25 [[etherpad:p/FrontEndDevOnboarding]] was archived as [[etherpadbackup:FrontEndDevOnboarding]] by [[User:BryanDavis|BryanDavis]] * 22:24 [[etherpad:p/France_-_communes_-_Data_.tab]] was archived as [[etherpadbackup:France_-_communes_-_Data_.tab]] by [[User:BryanDavis|BryanDavis]] * 22:24 [[etherpad:p/Foo]] was archived as [[etherpadbackup:Foo]] by [[User:BryanDavis|BryanDavis]] * 22:23 [[etherpad:p/FlowTodo]] was archived as [[etherpadbackup:FlowTodo]] by [[User:BryanDavis|BryanDavis]] * 22:23 [[etherpad:p/FlowAsAService]] was archived as [[etherpadbackup:FlowAsAService]] by [[User:BryanDavis|BryanDavis]] * 22:22 [[etherpad:p/Flow-ops-l_mail]] was archived as [[etherpadbackup:Flow-ops-l_mail]] by [[User:BryanDavis|BryanDavis]] * 22:21 [[etherpad:p/First_URLs]] was archived as [[etherpadbackup:First_URLs]] by [[User:BryanDavis|BryanDavis]] * 22:21 [[etherpad:p/Feb6Outage]] was archived as [[etherpadbackup:Feb6Outage]] by [[User:BryanDavis|BryanDavis]] * 22:20 [[etherpad:p/Feb-6-Issues]] was archived as [[etherpadbackup:Feb-6-Issues]] by [[User:BryanDavis|BryanDavis]] * 22:20 [[etherpad:p/FeaturesTeam2012-W15]] was archived as [[etherpadbackup:FeaturesTeam2012-W15]] by [[User:BryanDavis|BryanDavis]] * 22:19 [[etherpad:p/FeaturesTeam2012-W14]] was archived as [[etherpadbackup:FeaturesTeam2012-W14]] by [[User:BryanDavis|BryanDavis]] * 22:19 [[etherpad:p/FeaturesTeam2012-W13]] was archived as [[etherpadbackup:FeaturesTeam2012-W13]] by [[User:BryanDavis|BryanDavis]] * 22:18 [[etherpad:p/FeaturesTeam2012-W08]] was archived as [[etherpadbackup:FeaturesTeam2012-W08]] by [[User:BryanDavis|BryanDavis]] * 22:18 [[etherpad:p/FeaturesTeam2012-W07]] was archived as [[etherpadbackup:FeaturesTeam2012-W07]] by [[User:BryanDavis|BryanDavis]] * 22:17 [[etherpad:p/FeaturesTeam2012-W06]] was archived as [[etherpadbackup:FeaturesTeam2012-W06]] by [[User:BryanDavis|BryanDavis]] * 22:17 [[etherpad:p/FeaturesTeam2012-W03]] was archived as [[etherpadbackup:FeaturesTeam2012-W03]] by [[User:BryanDavis|BryanDavis]] * 22:16 [[etherpad:p/FeaturesTeam2012-W011]] was archived as [[etherpadbackup:FeaturesTeam2012-W011]] by [[User:BryanDavis|BryanDavis]] * 22:16 [[etherpad:p/FeaturesTeam20111227]] was archived as [[etherpadbackup:FeaturesTeam20111227]] by [[User:BryanDavis|BryanDavis]] * 22:15 [[etherpad:p/FeaturesTeam20111213]] was archived as [[etherpadbackup:FeaturesTeam20111213]] by [[User:BryanDavis|BryanDavis]] * 22:15 [[etherpad:p/FeaturesTeam20111206]] was archived as [[etherpadbackup:FeaturesTeam20111206]] by [[User:BryanDavis|BryanDavis]] * 22:14 [[etherpad:p/FeaturesTeam-Preload]] was archived as [[etherpadbackup:FeaturesTeam-Preload]] by [[User:BryanDavis|BryanDavis]] * 22:14 [[etherpad:p/Farm101]] was archived as [[etherpadbackup:Farm101]] by [[User:BryanDavis|BryanDavis]] * 22:13 [[etherpad:p/FRAnalyticsQ2FY16]] was archived as [[etherpadbackup:FRAnalyticsQ2FY16]] by [[User:BryanDavis|BryanDavis]] * 22:13 [[etherpad:p/FG3gMpB1qc]] was archived as [[etherpadbackup:FG3gMpB1qc]] by [[User:BryanDavis|BryanDavis]] * 22:12 [[etherpad:p/F2pDHw25sm7eJS1Q3Tk2]] was archived as [[etherpadbackup:F2pDHw25sm7eJS1Q3Tk2]] by [[User:BryanDavis|BryanDavis]] * 22:12 [[etherpad:p/F2gsk6WTo1]] was archived as [[etherpadbackup:F2gsk6WTo1]] by [[User:BryanDavis|BryanDavis]] * 22:11 [[etherpad:p/F]] was archived as [[etherpadbackup:F]] by [[User:BryanDavis|BryanDavis]] * 22:11 [[etherpad:p/ErAnycbwZA]] was archived as [[etherpadbackup:ErAnycbwZA]] by [[User:BryanDavis|BryanDavis]] * 22:10 [[etherpad:p/EpicIndiaItinerary]] was archived as [[etherpadbackup:EpicIndiaItinerary]] by [[User:BryanDavis|BryanDavis]] * 22:10 [[etherpad:p/Encuentro_técnico_hispanohablante_Wikimania_2022]] was archived as [[etherpadbackup:Encuentro_técnico_hispanohablante_Wikimania_2022]] by [[User:BryanDavis|BryanDavis]] * 22:09 [[etherpad:p/EchoNotification]] was archived as [[etherpadbackup:EchoNotification]] by [[User:BryanDavis|BryanDavis]] * 22:09 [[etherpad:p/EQIAD-rollout-sequence]] was archived as [[etherpadbackup:EQIAD-rollout-sequence]] by [[User:BryanDavis|BryanDavis]] * 22:08 [[etherpad:p/EL_python]] was archived as [[etherpadbackup:EL_python]] by [[User:BryanDavis|BryanDavis]] * 22:08 [[etherpad:p/E3Analytics]] was archived as [[etherpadbackup:E3Analytics]] by [[User:BryanDavis|BryanDavis]] * 22:07 [[etherpad:p/E3-2013-03-07-deploy]] was archived as [[etherpadbackup:E3-2013-03-07-deploy]] by [[User:BryanDavis|BryanDavis]] * 22:07 [[etherpad:p/DrinkingShoppingList]] was archived as [[etherpadbackup:DrinkingShoppingList]] by [[User:BryanDavis|BryanDavis]] * 22:06 [[etherpad:p/DownWithTwentyPercent]] was archived as [[etherpadbackup:DownWithTwentyPercent]] by [[User:BryanDavis|BryanDavis]] * 22:06 [[etherpad:p/Discussion__Redefine_Wikimedia_Hackathons]] was archived as [[etherpadbackup:Discussion__Redefine_Wikimedia_Hackathons]] by [[User:BryanDavis|BryanDavis]] * 22:05 [[etherpad:p/DevelopingMobileApps]] was archived as [[etherpadbackup:DevelopingMobileApps]] by [[User:BryanDavis|BryanDavis]] * 22:05 [[etherpad:p/DevSummitMain]] was archived as [[etherpadbackup:DevSummitMain]] by [[User:BryanDavis|BryanDavis]] * 22:04 [[etherpad:p/DevSum2015_designresearch]] was archived as [[etherpadbackup:DevSum2015_designresearch]] by [[User:BryanDavis|BryanDavis]] * 22:03 [[etherpad:p/DeploymentPrepReConfig]] was archived as [[etherpadbackup:DeploymentPrepReConfig]] by [[User:BryanDavis|BryanDavis]] * 22:03 [[etherpad:p/DSEorCDH4]] was archived as [[etherpadbackup:DSEorCDH4]] by [[User:BryanDavis|BryanDavis]] * 22:02 [[etherpad:p/DRworkshopParkingLot]] was archived as [[etherpadbackup:DRworkshopParkingLot]] by [[User:BryanDavis|BryanDavis]] * 22:02 [[etherpad:p/DRMethodsMenuSite]] was archived as [[etherpadbackup:DRMethodsMenuSite]] by [[User:BryanDavis|BryanDavis]] * 22:01 [[etherpad:p/ContinuousIntegration]] was archived as [[etherpadbackup:ContinuousIntegration]] by [[User:BryanDavis|BryanDavis]] * 22:01 [[etherpad:p/CollectionsNameA]] was archived as [[etherpadbackup:CollectionsNameA]] by [[User:BryanDavis|BryanDavis]] * 22:00 [[etherpad:p/Code-in_student_meeting]] was archived as [[etherpadbackup:Code-in_student_meeting]] by [[User:BryanDavis|BryanDavis]] * 22:00 [[etherpad:p/CmMGbzxbZw]] was archived as [[etherpadbackup:CmMGbzxbZw]] by [[User:BryanDavis|BryanDavis]] * 21:59 [[etherpad:p/Chromium-Upstream]] was archived as [[etherpadbackup:Chromium-Upstream]] by [[User:BryanDavis|BryanDavis]] * 21:59 [[etherpad:p/CentralNoticeDiscussion]] was archived as [[etherpadbackup:CentralNoticeDiscussion]] by [[User:BryanDavis|BryanDavis]] * 21:58 [[etherpad:p/CentralNotice]] was archived as [[etherpadbackup:CentralNotice]] by [[User:BryanDavis|BryanDavis]] * 21:58 [[etherpad:p/CampaignAssociatedMixins]] was archived as [[etherpadbackup:CampaignAssociatedMixins]] by [[User:BryanDavis|BryanDavis]] * 21:57 [[etherpad:p/C_________]] was archived as [[etherpadbackup:C_________]] by [[User:BryanDavis|BryanDavis]] * 21:57 [[etherpad:p/CZg-suME0Sx-nnCfKIEW]] was archived as [[etherpadbackup:CZg-suME0Sx-nnCfKIEW]] by [[User:BryanDavis|BryanDavis]] * 21:56 [[etherpad:p/CSF_Katie]] was archived as [[etherpadbackup:CSF_Katie]] by [[User:BryanDavis|BryanDavis]] * 21:56 [[etherpad:p/CEE-l10n-levels]] was archived as [[etherpadbackup:CEE-l10n-levels]] by [[User:BryanDavis|BryanDavis]] * 21:55 [[etherpad:p/C-Lessons]] was archived as [[etherpadbackup:C-Lessons]] by [[User:BryanDavis|BryanDavis]] * 21:55 [[etherpad:p/BotLicensing]] was archived as [[etherpadbackup:BotLicensing]] by [[User:BryanDavis|BryanDavis]] * 21:54 [[etherpad:p/BetaClusterpriorityworksync]] was archived as [[etherpadbackup:BetaClusterpriorityworksync]] by [[User:BryanDavis|BryanDavis]] * 21:54 [[etherpad:p/ArchitectureCommittee-2015-02-04]] was archived as [[etherpadbackup:ArchitectureCommittee-2015-02-04]] by [[User:BryanDavis|BryanDavis]] * 21:53 [[etherpad:p/ArchDoc]] was archived as [[etherpadbackup:ArchDoc]] by [[User:BryanDavis|BryanDavis]] * 21:53 [[etherpad:p/AppsReadership]] was archived as [[etherpadbackup:AppsReadership]] by [[User:BryanDavis|BryanDavis]] * 21:52 [[etherpad:p/AppsQ120142015]] was archived as [[etherpadbackup:AppsQ120142015]] by [[User:BryanDavis|BryanDavis]] * 21:52 [[etherpad:p/AppsPhabelloMigration]] was archived as [[etherpadbackup:AppsPhabelloMigration]] by [[User:BryanDavis|BryanDavis]] * 21:51 [[etherpad:p/AppsNav]] was archived as [[etherpadbackup:AppsNav]] by [[User:BryanDavis|BryanDavis]] * 21:51 [[etherpad:p/AppsContentService]] was archived as [[etherpadbackup:AppsContentService]] by [[User:BryanDavis|BryanDavis]] * 21:50 [[etherpad:p/AppUserTesting]] was archived as [[etherpadbackup:AppUserTesting]] by [[User:BryanDavis|BryanDavis]] * 21:50 [[etherpad:p/App-20design-20tweaks]] was archived as [[etherpadbackup:App-20design-20tweaks]] by [[User:BryanDavis|BryanDavis]] * 21:49 [[etherpad:p/Android_front-end_dev_task]] was archived as [[etherpadbackup:Android_front-end_dev_task]] by [[User:BryanDavis|BryanDavis]] * 21:48 [[etherpad:p/Analytics_all_staff]] was archived as [[etherpadbackup:Analytics_all_staff]] by [[User:BryanDavis|BryanDavis]] * 21:48 [[etherpad:p/Analytics_Wikimania]] was archived as [[etherpadbackup:Analytics_Wikimania]] by [[User:BryanDavis|BryanDavis]] * 21:47 [[etherpad:p/AnalyticsPlanningMeetingNotes]] was archived as [[etherpadbackup:AnalyticsPlanningMeetingNotes]] by [[User:BryanDavis|BryanDavis]] * 21:47 [[etherpad:p/AnalyticsMilestonesAndPriorities]] was archived as [[etherpadbackup:AnalyticsMilestonesAndPriorities]] by [[User:BryanDavis|BryanDavis]] * 21:46 [[etherpad:p/AnalyticsDevStaff]] was archived as [[etherpadbackup:AnalyticsDevStaff]] by [[User:BryanDavis|BryanDavis]] * 21:46 [[etherpad:p/Analytics-Reseach]] was archived as [[etherpadbackup:Analytics-Reseach]] by [[User:BryanDavis|BryanDavis]] * 21:45 [[etherpad:p/Analytics-Pixel-Service]] was archived as [[etherpadbackup:Analytics-Pixel-Service]] by [[User:BryanDavis|BryanDavis]] * 21:45 [[etherpad:p/Alpha-beta_cleaning]] was archived as [[etherpadbackup:Alpha-beta_cleaning]] by [[User:BryanDavis|BryanDavis]] * 21:44 [[etherpad:p/Adv_Browser_Tests_-_T86593]] was archived as [[etherpadbackup:Adv_Browser_Tests_-_T86593]] by [[User:BryanDavis|BryanDavis]] * 21:44 [[etherpad:p/Adedolapo_-_survey_for_content_Translation_users]] was archived as [[etherpadbackup:Adedolapo_-_survey_for_content_Translation_users]] by [[User:BryanDavis|BryanDavis]] * 21:43 [[etherpad:p/Accessibility_WMDS]] was archived as [[etherpadbackup:Accessibility_WMDS]] by [[User:BryanDavis|BryanDavis]] * 21:43 [[etherpad:p/AWMD_IRC-Meeting-13]] was archived as [[etherpadbackup:AWMD_IRC-Meeting-13]] by [[User:BryanDavis|BryanDavis]] * 21:42 [[etherpad:p/AWMD_IRC-Meeting-12]] was archived as [[etherpadbackup:AWMD_IRC-Meeting-12]] by [[User:BryanDavis|BryanDavis]] * 21:42 [[etherpad:p/AWMD_IRC-Meeting-11]] was archived as [[etherpadbackup:AWMD_IRC-Meeting-11]] by [[User:BryanDavis|BryanDavis]] * 21:41 [[etherpad:p/AWMD_IRC-Meeting-10]] was archived as [[etherpadbackup:AWMD_IRC-Meeting-10]] by [[User:BryanDavis|BryanDavis]] * 21:41 [[etherpad:p/API-blog-post]] was archived as [[etherpadbackup:API-blog-post]] by [[User:BryanDavis|BryanDavis]] * 21:40 [[etherpad:p/AFT5LaunchTests]] was archived as [[etherpadbackup:AFT5LaunchTests]] by [[User:BryanDavis|BryanDavis]] * 21:40 [[etherpad:p/AFT5]] was archived as [[etherpadbackup:AFT5]] by [[User:BryanDavis|BryanDavis]] * 21:39 [[etherpad:p/9RFl8QljDZ]] was archived as [[etherpadbackup:9RFl8QljDZ]] by [[User:BryanDavis|BryanDavis]] * 21:39 [[etherpad:p/92]] was archived as [[etherpadbackup:92]] by [[User:BryanDavis|BryanDavis]] * 21:38 [[etherpad:p/8r8ihQ7GpX]] was archived as [[etherpadbackup:8r8ihQ7GpX]] by [[User:BryanDavis|BryanDavis]] * 21:38 [[etherpad:p/8D1ml5Q1Sb]] was archived as [[etherpadbackup:8D1ml5Q1Sb]] by [[User:BryanDavis|BryanDavis]] * 21:37 [[etherpad:p/88952]] was archived as [[etherpadbackup:88952]] by [[User:BryanDavis|BryanDavis]] * 21:37 [[etherpad:p/7hDQxehZZW]] was archived as [[etherpadbackup:7hDQxehZZW]] by [[User:BryanDavis|BryanDavis]] * 21:36 [[etherpad:p/7YAmoRjGSAvfJrj9V9sD]] was archived as [[etherpadbackup:7YAmoRjGSAvfJrj9V9sD]] by [[User:BryanDavis|BryanDavis]] * 21:36 [[etherpad:p/7FtE43Tuhu]] was archived as [[etherpadbackup:7FtE43Tuhu]] by [[User:BryanDavis|BryanDavis]] * 21:35 [[etherpad:p/77SMYoPXek]] was archived as [[etherpadbackup:77SMYoPXek]] by [[User:BryanDavis|BryanDavis]] * 21:35 [[etherpad:p/6gR8aSREkz]] was archived as [[etherpadbackup:6gR8aSREkz]] by [[User:BryanDavis|BryanDavis]] * 21:34 [[etherpad:p/58040]] was archived as [[etherpadbackup:58040]] by [[User:BryanDavis|BryanDavis]] * 21:34 [[etherpad:p/503-eqiad-cirrussearch]] was archived as [[etherpadbackup:503-eqiad-cirrussearch]] by [[User:BryanDavis|BryanDavis]] * 21:33 [[etherpad:p/4pKxno7DAI]] was archived as [[etherpadbackup:4pKxno7DAI]] by [[User:BryanDavis|BryanDavis]] * 21:33 [[etherpad:p/4J8fHuL0yB]] was archived as [[etherpadbackup:4J8fHuL0yB]] by [[User:BryanDavis|BryanDavis]] * 21:32 [[etherpad:p/4.1_launch_post-mortem]] was archived as [[etherpadbackup:4.1_launch_post-mortem]] by [[User:BryanDavis|BryanDavis]] * 21:32 [[etherpad:p/20percent]] was archived as [[etherpadbackup:20percent]] by [[User:BryanDavis|BryanDavis]] * 21:31 [[etherpad:p/2026.03_Wikibase_SPARQL_Prefix_@_WMHack]] was archived as [[etherpadbackup:2026.03_Wikibase_SPARQL_Prefix_@_WMHack]] by [[User:BryanDavis|BryanDavis]] * 21:30 [[etherpad:p/2025-10-workers-reboot]] was archived as [[etherpadbackup:2025-10-workers-reboot]] by [[User:BryanDavis|BryanDavis]] * 21:30 [[etherpad:p/2025-04_toolforge_monthly_update]] was archived as [[etherpadbackup:2025-04_toolforge_monthly_update]] by [[User:BryanDavis|BryanDavis]] * 21:29 [[etherpad:p/2024_tekton_upgrade]] was archived as [[etherpadbackup:2024_tekton_upgrade]] by [[User:BryanDavis|BryanDavis]] * 21:29 [[etherpad:p/2021-switchdc-testing]] was archived as [[etherpadbackup:2021-switchdc-testing]] by [[User:BryanDavis|BryanDavis]] * 21:28 [[etherpad:p/2021-switchdc-prep]] was archived as [[etherpadbackup:2021-switchdc-prep]] by [[User:BryanDavis|BryanDavis]] * 21:28 [[etherpad:p/20205-06-components-api-reviews]] was archived as [[etherpadbackup:20205-06-components-api-reviews]] by [[User:BryanDavis|BryanDavis]] * 21:27 [[etherpad:p/2019-09-25-cloudvirt-load-rebalance]] was archived as [[etherpadbackup:2019-09-25-cloudvirt-load-rebalance]] by [[User:BryanDavis|BryanDavis]] * 21:27 [[etherpad:p/2019-07-23-eqiad-asw-a]] was archived as [[etherpadbackup:2019-07-23-eqiad-asw-a]] by [[User:BryanDavis|BryanDavis]] * 21:26 [[etherpad:p/2019-07-16-docker-registry]] was archived as [[etherpadbackup:2019-07-16-docker-registry]] by [[User:BryanDavis|BryanDavis]] * 21:26 [[etherpad:p/20180829-labstore-reboots]] was archived as [[etherpadbackup:20180829-labstore-reboots]] by [[User:BryanDavis|BryanDavis]] * 21:25 [[etherpad:p/20150519-Mailman]] was archived as [[etherpadbackup:20150519-Mailman]] by [[User:BryanDavis|BryanDavis]] * 21:25 [[etherpad:p/2014-antoine-june-checkin]] was archived as [[etherpadbackup:2014-antoine-june-checkin]] by [[User:BryanDavis|BryanDavis]] * 21:24 [[etherpad:p/2013-2014-Q2-MobileWeb]] was archived as [[etherpadbackup:2013-2014-Q2-MobileWeb]] by [[User:BryanDavis|BryanDavis]] * 21:24 [[etherpad:p/2012-2013-Q1-Mobile-Dept]] was archived as [[etherpadbackup:2012-2013-Q1-Mobile-Dept]] by [[User:BryanDavis|BryanDavis]] * 21:23 [[etherpad:p/19-jun-2014-parsercache-outage]] was archived as [[etherpadbackup:19-jun-2014-parsercache-outage]] by [[User:BryanDavis|BryanDavis]] * 21:23 [[etherpad:p/19-jun-2014-parsercache-outage]] was archived as [[etherpadbackup:19-jun-2014-parsercache-outage]] by [[User:BryanDavis|BryanDavis]] * 20:22 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]] === 2026-05-03 === * 13:56 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]] * 13:56 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]] * 13:56 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/123]] was archived as [[etherpadbackup:123]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/120wmf2deployment]] was archived as [[etherpadbackup:120wmf2deployment]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/119triage]] was archived as [[etherpadbackup:119triage]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/119deployment]] was archived as [[etherpadbackup:119deployment]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/119917]] was archived as [[etherpadbackup:119917]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/119-beta]] was archived as [[etherpadbackup:119-beta]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/1.36-release]] was archived as [[etherpadbackup:1.36-release]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/1.27.0-wmf.13-issues]] was archived as [[etherpadbackup:1.27.0-wmf.13-issues]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/1.25wmf15-perf-regression]] was archived as [[etherpadbackup:1.25wmf15-perf-regression]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/0WdKUDRA58]] was archived as [[etherpadbackup:0WdKUDRA58]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/123]] was archived as [[etherpadbackup:123]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/120wmf2deployment]] was archived as [[etherpadbackup:120wmf2deployment]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/119triage]] was archived as [[etherpadbackup:119triage]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/119deployment]] was archived as [[etherpadbackup:119deployment]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/119917]] was archived as [[etherpadbackup:119917]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/119-beta]] was archived as [[etherpadbackup:119-beta]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/1.36-release]] was archived as [[etherpadbackup:1.36-release]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/1.27.0-wmf.13-issues]] was archived as [[etherpadbackup:1.27.0-wmf.13-issues]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/1.25wmf15-perf-regression]] was archived as [[etherpadbackup:1.25wmf15-perf-regression]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/0WdKUDRA58]] was archived as [[etherpadbackup:0WdKUDRA58]] by [[User:BryanDavis|BryanDavis]] * 12:02 [[etherpad:p/WIKISOO3]] was archived as [[etherpadbackup:WIKISOO3]] by [[User:BryanDavis|BryanDavis]] * 12:02 [[etherpad:p/VertalingWikiIt]] was archived as [[etherpadbackup:VertalingWikiIt]] by [[User:BryanDavis|BryanDavis]] * 12:02 [[etherpad:p/WBUG_2023_09_28]] was archived as [[etherpadbackup:WBUG_2023_09_28]] by [[User:BryanDavis|BryanDavis]] * 12:02 [[etherpad:p/WBUG_2023_07_27]] was archived as [[etherpadbackup:WBUG_2023_07_27]] by [[User:BryanDavis|BryanDavis]] * 12:02 [[etherpad:p/IRCBot-Messages]] was archived as [[etherpadbackup:IRCBot-Messages]] by [[User:BryanDavis|BryanDavis]] * 12:02 [[etherpad:p/GLAMcampAmsterdamFri]] was archived as [[etherpadbackup:GLAMcampAmsterdamFri]] by [[User:BryanDavis|BryanDavis]] * 12:02 [[etherpad:p/GLAM-Wiki-CH_OpenRefine_PAWS]] was archived as [[etherpadbackup:GLAM-Wiki-CH_OpenRefine_PAWS]] by [[User:BryanDavis|BryanDavis]] * 12:02 [[etherpad:p/GD0bnahDcM]] was archived as [[etherpadbackup:GD0bnahDcM]] by [[User:BryanDavis|BryanDavis]] * 12:02 [[etherpad:p/DrTrigonbotfalsepositivesfilter]] was archived as [[etherpadbackup:DrTrigonbotfalsepositivesfilter]] by [[User:BryanDavis|BryanDavis]] * 12:00 [[etherpad:p/WBUG_2023_09_28]] was archived as [[etherpadbackup:WBUG_2023_09_28]] by [[User:BryanDavis|BryanDavis]] * 12:00 [[etherpad:p/WBUG_2023_07_27]] was archived as [[etherpadbackup:WBUG_2023_07_27]] by [[User:BryanDavis|BryanDavis]] * 12:00 [[etherpad:p/IRCBot-Messages]] was archived as [[etherpadbackup:IRCBot-Messages]] by [[User:BryanDavis|BryanDavis]] * 12:00 [[etherpad:p/GLAMcampAmsterdamFri]] was archived as [[etherpadbackup:GLAMcampAmsterdamFri]] by [[User:BryanDavis|BryanDavis]] * 12:00 [[etherpad:p/GLAM-Wiki-CH_OpenRefine_PAWS]] was archived as [[etherpadbackup:GLAM-Wiki-CH_OpenRefine_PAWS]] by [[User:BryanDavis|BryanDavis]] * 12:00 [[etherpad:p/GD0bnahDcM]] was archived as [[etherpadbackup:GD0bnahDcM]] by [[User:BryanDavis|BryanDavis]] * 12:00 [[etherpad:p/DrTrigonbotfalsepositivesfilter]] was archived as [[etherpadbackup:DrTrigonbotfalsepositivesfilter]] by [[User:BryanDavis|BryanDavis]] * 12:00 [[etherpad:p/BugTriage-Collection]] was archived as [[etherpadbackup:BugTriage-Collection]] by [[User:BryanDavis|BryanDavis]] * 09:25 [[etherpad:p/test-*,?,\,/,%,>,<,🤖💬]] was archived as [[etherpadbackup:test-*,?,\,/,%,>,<,🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 09:18 [[etherpad:p/test-*,?,\,/,%,>,<,🤖💬]] was archived as [[etherpadbackup:test-*,?,\,/,%,>,<,🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 07:46 [[etherpad:p/test-*,?,\,/,%,>,<,🤖💬]] was archived as [[etherpadbackup:test-*,?,\,/,%,>,<,🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 07:28 [[etherpad:p/prodlikevagrant-hackathon-planning]] was archived as [[etherpadbackup:prodlikevagrant-hackathon-planning]] by [[User:BDavis (WMF)|BDavis (WMF)]] === 2026-05-02 === * 13:01 [[etherpad:p/wpcon16_Wikipedianer_%26_%E2%80%9EMateriale_Textkultu]] was archived as [[etherpadbackup:p/wpcon16_Wikipedianer_%26_%E2%80%9EMateriale_Textkultu]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 07:18 [[etherpad:p/test-*%2C%3F%2C\%2C%2F%2C%25%2C>%2C<%2C🤖💬]] was archived as [[etherpadbackup:p/test-*%2C%3F%2C\%2C%2F%2C%25%2C>%2C<%2C🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 06:58 [[etherpad:p/test-*%2C%3F%2C\%2C%2F%2C%25%2C>%2C<%2C🤖💬]] was archived as [[etherpadbackup:p/test-*%2C%3F%2C\%2C%2F%2C%25%2C>%2C<%2C🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]] === 2026-05-01 === * 18:12 [[etherpad:p/10myths]] was archived as [[etherpadbackup:p/10myths]] by [[User:Krinkle|Krinkle]] * 15:30 [[etherpad:p/test-*,?,\,/,%,>,<,🤖💬]] was archived as [[etherpadbackup:p/test-*,?,\,/,%,>,<,🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 14:50 [[etherpad:p/PoGo_Friendship_Codes]] was archived as [[etherpadbackup:p/PoGo_Friendship_Codes]] by [[User:Jon Harald Søby|Jon Harald Søby]] * 14:42 [[etherpad:p/PoGo_Friendship_Codes]] was archived as [[etherpadbackup:p/PoGo_Friendship_Codes]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 13:51 [[etherpad:p/Wikimedia_Hackathon_2026_Opening]] was archived as [[etherpadbackup:p/Wikimedia_Hackathon_2026_Opening]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 13:16 [[etherpad:p/Wikimedia_Hackathon_2026_Opening]] was archived as [[etherpadbackup:p/Wikimedia_Hackathon_2026_Opening]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 13:07 [[etherpad:p/copyvios-todo]] was archived as [[etherpadbackup:p/copyvios-todo]] by [[User:Chlod|Chlod]] * 10:05 [[etherpad:p/hhvmdeploy]] was archived as [[etherpadbackup:p/hhvmdeploy]] by [[User:BDavis (WMF)|BDavis (WMF)]] === 2026-04-29 === * 00:49 [[etherpad:p/better-security-patch-management]] was archived as [[etherpadbackup:p/better-security-patch-management]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 00:49 [[etherpad:p/rooks-questions-to-alex]] was archived as [[etherpadbackup:p/rooks-questions-to-alex]] by [[User:BDavis (WMF)|BDavis (WMF)]] === 2026-04-27 === * 04:44 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]] * 03:46 [[etherpad:p/wmh2023-Overview_of_technical_areas_&_projects]] was archived as [[etherpadbackup:p/wmh2023-Overview_of_technical_areas_&_projects]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/open_meeting_COVID-19_18/04]] was archived as [[etherpadbackup:p/open_meeting_COVID-19_18/04]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/open_meeting_COVID-19_04/05]] was archived as [[etherpadbackup:p/open_meeting_COVID-19_04/05]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/aswikimeet/july-2024]] was archived as [[etherpadbackup:p/aswikimeet/july-2024]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/Wikisource_Meetup_29/09/2020]] was archived as [[etherpadbackup:p/Wikisource_Meetup_29/09/2020]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/Wikimedians_of_Nepal/Events/Meetup_February_2020]] was archived as [[etherpadbackup:p/Wikimedians_of_Nepal/Events/Meetup_February_2020]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/Wikimedia_LGBT+/Governance/2023-10-13]] was archived as [[etherpadbackup:p/Wikimedia_LGBT+/Governance/2023-10-13]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/WikiCendekia2026/WarkopD]] was archived as [[etherpadbackup:p/WikiCendekia2026/WarkopD]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/WikiCendekia2026/WarkopC]] was archived as [[etherpadbackup:p/WikiCendekia2026/WarkopC]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/WikiCendekia2026/WarkopB]] was archived as [[etherpadbackup:p/WikiCendekia2026/WarkopB]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/WikiCendekia2026/WarkopA]] was archived as [[etherpadbackup:p/WikiCendekia2026/WarkopA]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/Stolpersteine_nach_WikiData_(01/2025]] was archived as [[etherpadbackup:p/Stolpersteine_nach_WikiData_(01/2025]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/Sommer_24/8,_1._August]] was archived as [[etherpadbackup:p/Sommer_24/8,_1._August]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/SVG_Translation_Campaign_2019_India/Event/Patiala]] was archived as [[etherpadbackup:p/SVG_Translation_Campaign_2019_India/Event/Patiala]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/Meetup/Cape_Town/Cape_Town_22]] was archived as [[etherpadbackup:p/Meetup/Cape_Town/Cape_Town_22]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/Mach_mit!/2._Termin_Mentorinnen-Netzwerk]] was archived as [[etherpadbackup:p/Mach_mit!/2._Termin_Mentorinnen-Netzwerk]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/Kopdar_Wikisource_(Wikipabukon)_06/08/23]] was archived as [[etherpadbackup:p/Kopdar_Wikisource_(Wikipabukon)_06/08/23]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/Igbo_Wikimedians_Community/Movement_Charter_Commun]] was archived as [[etherpadbackup:p/Igbo_Wikimedians_Community/Movement_Charter_Commun]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/GLAM-Treffen_2020/Gruppe_C]] was archived as [[etherpadbackup:p/GLAM-Treffen_2020/Gruppe_C]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/GLAM-Treffen_2020/Gruppe_B]] was archived as [[etherpadbackup:p/GLAM-Treffen_2020/Gruppe_B]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/GLAM-Treffen_2020/Gruppe_A]] was archived as [[etherpadbackup:p/GLAM-Treffen_2020/Gruppe_A]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/CEE_Talks/Update_CEE_Hub_/_Outlook_CEE_Meeting_202]] was archived as [[etherpadbackup:p/CEE_Talks/Update_CEE_Hub_/_Outlook_CEE_Meeting_202]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/CEE_Talks/Outlook_CEE_Meeting_2022_V2]] was archived as [[etherpadbackup:p/CEE_Talks/Outlook_CEE_Meeting_2022_V2]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/CEE_Meeting_2020/Wikispore]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Wikispore]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/CEE_Meeting_2020/Translatable_modules]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Translatable_modules]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/CEE_Meeting_2020/Opportunities_for_1Lib1Ref]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Opportunities_for_1Lib1Ref]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/CEE_Meeting_2020/Movement_Strategy]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Movement_Strategy]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/CEE_Meeting_2020/Lua_templates_for_beginners]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Lua_templates_for_beginners]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/CEE_Meeting_2020/Help_your_community_grow]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Help_your_community_grow]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/CEE_Meeting_2020/101_Ways_to_Contribute]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/101_Ways_to_Contribute]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/AsWikiMentoringProject2024/Oct-Dec]] was archived as [[etherpadbackup:p/AsWikiMentoringProject2024/Oct-Dec]] by [[User:BryanDavis|BryanDavis]] * 03:16 [[etherpad:p/AsWikiMentoringProject2024%2FOct-Dec]] was deleted as [[etherpadbackup:p/AsWikiMentoringProject2024%2FOct-Dec]] by [[User:BryanDavis|BryanDavis]] * 03:15 [[etherpad:p/AsWikiMentoringProject2024%2FOct-Dec]] was deleted as [[etherpadbackup:p/AsWikiMentoringProject2024%2FOct-Dec]] by [[User:BryanDavis|BryanDavis]] * 03:12 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:wmh2023-Overview_of_technical_areas_%26_projects|wmh2023-Overview_of_technical_areas_%26_projects]] * 03:12 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:open_meeting_COVID-19_20%2F04|open_meeting_COVID-19_20%2F04]] * 03:12 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:open_meeting_COVID-19_18%2F04|open_meeting_COVID-19_18%2F04]] * 03:12 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:open_meeting_COVID-19_04%2F05|open_meeting_COVID-19_04%2F05]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:aswikimeet%2Fjuly-2024|aswikimeet%2Fjuly-2024]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Wikisource_Meetup_29%2F09%2F2020|Wikisource_Meetup_29%2F09%2F2020]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Wikimedians_of_Nepal%2FEvents%2FMeetup_February_2020|Wikimedians_of_Nepal%2FEvents%2FMeetup_February_2020]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Wikimedia_LGBT+%2FGovernance%2F2023-10-13|Wikimedia_LGBT+%2FGovernance%2F2023-10-13]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:WikiCendekia2026%2FWarkopD|WikiCendekia2026%2FWarkopD]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:WikiCendekia2026%2FWarkopC|WikiCendekia2026%2FWarkopC]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:WikiCendekia2026%2FWarkopB|WikiCendekia2026%2FWarkopB]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:WikiCendekia2026%2FWarkopA|WikiCendekia2026%2FWarkopA]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Summary%2F_Hypothesis|Summary%2F_Hypothesis]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Stolpersteine_nach_WikiData_(12%2F2024)|Stolpersteine_nach_WikiData_(12%2F2024)]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Stolpersteine_nach_WikiData_(01%2F2025|Stolpersteine_nach_WikiData_(01%2F2025]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Sommer_24%2F8,_1._August|Sommer_24%2F8,_1._August]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:SVG_Translation_Campaign_2019_India%2FEvent%2FPatiala|SVG_Translation_Campaign_2019_India%2FEvent%2FPatiala]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Meetup%2FCape_Town%2FCape_Town_22|Meetup%2FCape_Town%2FCape_Town_22]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Mach_mit!%2F2._Termin_Mentorinnen-Netzwerk|Mach_mit!%2F2._Termin_Mentorinnen-Netzwerk]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Kopdar_Wikisource_(Wikipabukon)_06%2F08%2F23|Kopdar_Wikisource_(Wikipabukon)_06%2F08%2F23]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Igbo_Wikimedians_Community%2FMovement_Charter_Commun|Igbo_Wikimedians_Community%2FMovement_Charter_Commun]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:GLAM-Treffen_2020%2FGruppe_C|GLAM-Treffen_2020%2FGruppe_C]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:GLAM-Treffen_2020%2FGruppe_B|GLAM-Treffen_2020%2FGruppe_B]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:GLAM-Treffen_2020%2FGruppe_A|GLAM-Treffen_2020%2FGruppe_A]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Commons_Photographers_30%2F05%2F2020|Commons_Photographers_30%2F05%2F2020]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:ChristianSW%2FNotizen|ChristianSW%2FNotizen]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Talks%2FUpdate_CEE_Hub_%2F_Outlook_CEE_Meeting_202|CEE_Talks%2FUpdate_CEE_Hub_%2F_Outlook_CEE_Meeting_202]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Talks%2FOutlook_CEE_Meeting_2022_V2|CEE_Talks%2FOutlook_CEE_Meeting_2022_V2]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FWikispore|CEE_Meeting_2020%2FWikispore]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FTranslatable_modules|CEE_Meeting_2020%2FTranslatable_modules]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FOpportunities_for_1Lib1Ref|CEE_Meeting_2020%2FOpportunities_for_1Lib1Ref]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FMovement_Strategy|CEE_Meeting_2020%2FMovement_Strategy]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FLua_templates_for_beginners|CEE_Meeting_2020%2FLua_templates_for_beginners]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FHelp_your_community_grow|CEE_Meeting_2020%2FHelp_your_community_grow]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2F101_Ways_to_Contribute|CEE_Meeting_2020%2F101_Ways_to_Contribute]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:AsWikiMentoringProject2024%2FOct-Dec|AsWikiMentoringProject2024%2FOct-Dec]] * 01:44 [[etherpad:p/foo]] was archived as [[etherpadbackup:p/foo]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:51 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:51 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:51 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:50 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:50 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:49 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:44 [[etherpad:p/foo]] was archived as [[etherpadbackup:p/foo]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:44 [[etherpad:p/foo]] was archived as [[etherpadbackup:p/foo]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:43 [[etherpad:p/foo]] was archived as [[etherpadbackup:p/foo]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:05 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]] === 2026-04-26 === * 23:25 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]] * 21:58 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]] * 17:51 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]] * 17:22 [[etherpad:p/bd808 test/path/?query=param]] was archived as [[etherpadbackup:p/bd808 test/path/?query=param]] by [[User:BryanDavis|BryanDavis]] * 17:22 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]] * 16:17 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]] === 2026-04-20 === * 02:42 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]] * 02:36 [[etherpad:p/bd808-etherpad-backup-test]] was archived by [[User:BryanDavis]] eqy9kicrik10aolg7jgiryuogjfetto 498 460173 2426630 2424453 2026-06-13T21:35:49Z ZI Jony 13427 Updated 2426630 wikitext text/x-wiki {{Shortcut|WIKILOUNGE/ADMIN}} This page contains the technical documentation and standard operating procedures for administering the '''WikiLounge''' server. This guide is intended only for server administrators with root/SSH access to the WMCS instance. == Server Access == WikiLounge is hosted on Wikimedia Cloud Services (WMCS). To administer the server, you must connect via SSH using your Wikitech developer account credentials. * '''Server Hostname:''' <code>lounge-server.eqiad1.wikimedia.cloud</code> (or your specific instance name) * '''SSH Command:''' <pre>ssh -J your_shell_username@bastion.wmcloud.org your_shell_username@lounge-server.thelounge.eqiad1.wikimedia.cloud</pre> Once logged in, the IRC bouncer configurations are managed under the user account where it was installed (typically your user directory, e.g., <code>~/.thelounge/</code>), and the Unified Web Portal is located in <code>~/wikilounge-admin/</code>. == Unified Web Dashboard == The primary method for administering WikiLounge is now the Unified Admin Dashboard, accessible at '''[https://wikilounge.wmcloud.org/admin https://wikilounge.wmcloud.org/admin]'''. * '''Access:''' Authenticates via Meta-Wiki OAuth. Only approved Super Admins and Admins can access the dashboard. * '''Inbox:''' Allows one-click approval of incoming account requests, which automatically provisions server space and sends temporary passwords via MediaWiki email. * '''Tracking:''' Tracks user inactivity and handles 150-day warnings and 180-day auto-suspensions via a background Cron job utilizing an SQLite database (<code>admin.db</code>). == User Management (CLI Fallbacks) == While the web dashboard automates the safe creation, renaming, resetting, and removal of users, administrators with SSH access can still perform these actions manually using The Lounge CLI if the web portal is down or requires backend maintenance. === Create a User === To manually create a new account and generate their initial password: <pre>thelounge add <username></pre> * The prompt will ask for a password (typing is invisible). * Choose "yes" to save logs to disk. * Press Enter to accept the default log location. === Remove a User === To permanently delete a user and immediately disconnect them from the IRC network: <pre>thelounge remove <username></pre> ''Note: This deletes their account file, but does not automatically delete their chat history logs. You must delete their logs manually if required for data retention compliance.'' === Reset a Password === If a user forgets their password or an account is compromised: <pre>thelounge reset <username></pre> === Rename a User === The Lounge does not have a native "rename" command. To rename a user manually, you must rename their configuration file and log directory, then restart the service: # Stop the server: <code>pm2 stop thelounge</code> # Rename the user config: <code>mv ~/.thelounge/users/OldName.json ~/.thelounge/users/NewName.json</code> # Rename their log database: <code>mv ~/.thelounge/logs/OldName.sqlite3 ~/.thelounge/logs/NewName.sqlite3</code> # Start the server: <code>pm2 start thelounge</code> == Configuration & Technical Details == === The Master Configuration File === All global IRC server settings (network lock, global bind IP, public mode) are stored in the master configuration file: <pre>nano ~/.thelounge/config.js</pre> Whenever you make a change to this file, you must restart the service for it to take effect. === Unified Portal Configuration (.env) === The Node.js web portal relies on an environment file for Meta-Wiki OAuth credentials, the MediaWiki bot password (used for automated emails and talk page warnings), and session security. <pre>nano ~/wikilounge-admin/.env</pre> If you update the OAuth or bot credentials in the <code>.env</code> file, you must force PM2 to reload the environment cache: <pre>pm2 restart wikilounge-admin --update-env</pre> === Service Management (PM2) === WikiLounge relies on two background processes managed by PM2: the IRC bouncer (<code>thelounge</code>) and the unified web portal (<code>wikilounge-admin</code>). Use these commands to control the server: * '''Restart web portal:''' <code>pm2 restart wikilounge-admin</code> * '''Restart IRC server:''' <code>pm2 restart thelounge</code> * '''Stop all services:''' <code>pm2 stop all</code> * '''View live console logs:''' <code>pm2 logs</code> * '''Check server status:''' <code>pm2 status</code> == Log Management & Abuse Investigations == {{Notice|'''Privacy Notice:''' Accessing a user's private message logs is a severe action. Logs should only be inspected during an active abuse investigation, a UCoC violation report, or at the explicit request of WMF Trust & Safety. Do not read user logs casually.}} === Locating Message Logs === To provide the offline "catch-up" feature, WikiLounge stores user chat history in SQLite databases. * '''Log Directory:''' <code>~/.thelounge/logs/</code> * '''Format:''' Each user has their own isolated database file (e.g., <code>~/.thelounge/logs/username.sqlite3</code>). === Investigating Logs === If you need to extract chat logs for a Phabricator abuse ticket or Trust & Safety investigation, you can query the SQLite database directly from the terminal. 1. Connect to the user's database: <pre>sqlite3 ~/.thelounge/logs/<username>.sqlite3</pre> 2. View the schema to understand the tables: <pre>.tables</pre> 3. Run a SQL query to find specific messages (example: searching for a specific word or user in the logs): <pre>SELECT time, network, target, text FROM messages WHERE text LIKE '%abuse_keyword%' LIMIT 50;</pre> 4. Type <code>.quit</code> to exit the SQLite prompt. === Deleting Logs === If a user requests the deletion of their data (Right to be Forgotten), or if an account is permanently removed: <pre>rm ~/.thelounge/logs/<username>.sqlite3</pre> == Bouncer Networking (IPv6) == To comply with Wikimedia Cloud Services and Libera.Chat policies, WikiLounge routes all IRC traffic through a designated IPv6 address. * The global bind address is hardcoded in <code>~/.thelounge/config.js</code>. * Do not change the <code>bind</code> property without approval from WMCS Cloud Services administrators. [[Category:IRC]] t0qfd2tdiabbc1g0l1kqd9242i5iqew 0 460323 2426646 2426152 2026-06-14T04:23:30Z Quiddity 1884 syntaxhighlight; linebreaks 2426646 wikitext text/x-wiki '''Lift Wing OpenAPI Server''' is a lightweight Apache httpd service that serves the Lift Wing OpenAPI specification files as static content. Unlike the other model servers in the <code>inference-services</code> repository, this is '''not''' a KServe model server — it does not use PyTorch, model volumes, or the KServe framework. == What it does == * Serves the aggregated <code>docs/openapi.yaml</code> and all per-model OpenAPI spec files * Adds [[:en:CORS|CORS]] headers so browser-based tools (Swagger UI, [[mw:API/REST_Sandbox|MediaWiki RestSandbox]]) can fetch the specs cross-origin * The <code>openapi.yaml</code> file uses <code>$ref</code> references to sibling files (e.g. <code>./langid.yaml</code>), so the entire <code>docs/</code> directory is served == Architecture == Based on the [[gitlab:repos/sre/miscweb/statictendril|statictendril/miscweb]] pattern. No custom Python code — just Apache httpd with config files. {| class="wikitable" !File !Purpose |- |<code>httpd.conf</code> |Main Apache config — loads modules (<code>mod_headers</code>, <code>mod_mime</code>, <code>mod_rewrite</code>, etc.) and includes the vhost |- |<code>liftwingspec.conf</code> |VirtualHost on <code>*:8080</code> — <code>DocumentRoot /srv/app/docs</code>, CORS headers, <code>ServerAlias api.wikimedia.org</code> |- |<code>entrypoint.sh</code> |Starts Apache with <code>apache2 -d /srv/app -f /srv/app/apache2.conf -DFOREGROUND</code> |- |<code>blubber.yaml</code> |[[mw:Blubber|Blubber]] build config — <code>docker-registry.wikimedia.org/httpd</code> base, copies <code>docs/</code> and config files into <code>/srv/app/</code> |} === How the config fits together === <pre> /srv/app/ ├── apache2.conf ← copied from httpd.conf │ ├── Loads mod_headers, mod_mime, mod_rewrite, ... │ └── IncludeOptional liftwingspec.conf ├── liftwingspec.conf ← VirtualHost with DocumentRoot, CORS, ServerAlias ├── docs/ ← the OpenAPI YAML files (openapi.yaml + per-model specs) │ ├── openapi.yaml │ ├── langid.yaml │ ├── edit-check.yaml │ └── ... ├── entrypoint.sh </pre> == Source code == * [[gerrit:plugins/gitiles/machinelearning/liftwing/inference-services/+/refs/heads/main/src/models/liftwing_openapi_server/|<code>src/models/liftwing_openapi_server/</code>]] * [[gerrit:plugins/gitiles/machinelearning/liftwing/inference-services/+/refs/heads/main/.pipeline/liftwing_openapi_server/blubber.yaml|Blubber pipeline config]] * [[gerrit:plugins/gitiles/machinelearning/liftwing/inference-services/+/refs/heads/main/docs/openapi.yaml|Aggregated OpenAPI spec]] == Running locally == Build and start the server: <pre> docker compose build liftwing-openapi-server docker compose up -d liftwing-openapi-server </pre> Test it: <pre> # Main aggregated spec curl http://localhost:8081/openapi.yaml </pre> === Local MediaWiki integration === With MediaWiki running at <code><nowiki>http://127.0.0.1:8080</nowiki></code>, add this to <code>LocalSettings.php</code>: <syntaxhighlight lang="php"> $wgRestSandboxSpecs['lw-openapi'] = [ 'url' => 'http://localhost:8081/openapi.yaml', 'name' => 'Lift Wing', 'group' => 'Lift Wing', ]; </syntaxhighlight> Then visit <code><nowiki>http://127.0.0.1:8080/w/index.php/Special:RestSandbox</nowiki></code>. == Deployment == The image is built via Jenkins and published as <code>inference-services-liftwing-openapi-server:stable</code>. Since this is not a KServe InferenceService, it is deployed as a plain Kubernetes Deployment using the [[gerrit:plugins/gitiles/operations/deployment-charts/+/refs/heads/master/helmfile.d/ml-services/liftwing-openapi-server/values.yaml|liftwing-openapi-server chart]] and follows the standard deployment procedure documented at [[Machine Learning/LiftWing/Deploy#How to deploy|Machine Learning/LiftWing/Deploy]]. The public URL is <code><nowiki>https://api.wikimedia.org/service/lw/specs/openapi.yaml</nowiki></code>, configured in the [[gerrit:plugins/gitiles/operations/mediawiki-config/+/refs/heads/master/wmf-config/InitialiseSettings.php#13154|<code>wgRestSandboxSpecs</code>]] setting in <code>mediawiki-config</code>. == See also == * [[Machine Learning/LiftWing/Inference Services]] * [[Machine Learning/LiftWing/API]] * [[Machine Learning/LiftWing/Deploy]] * {{Phabricator|T427902}} mhl08w2dmbe5zsyftwpk1bem1lylmxh 2 460332 2426647 2026-06-14T05:20:57Z 配合比全额更好（说说而已） 55856 Created page with "Hello!I'm 配合比全额更好（说说而已）." 2426647 wikitext text/x-wiki Hello!I'm 配合比全额更好（说说而已）. 8ku76nd4c1cb0o0z2e6q0rysvn6qd2h