Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.46.0-wmf.21 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2396717 2396309 2026-03-30T12:05:03Z Stashbot 7414 dcaro: removing wal from prometheus nodes to restart them 2396717 wikitext text/x-wiki b6kzktytf1loxf8sgjjqoxrt82b89ac 2396825 2396717 2026-03-30T14:38:39Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers 2396825 wikitext text/x-wiki gvxlqxqxdht28ou4p6fwqsbpvc0y8pc 2396826 2396825 2026-03-30T14:38:51Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers 2396826 wikitext text/x-wiki 0h6wlappicq4z1b8xqeuw1jq1ern5jw 0 3205 2396984 2395983 2026-03-30T19:29:21Z BCornwall-WMF 31521 /* Remove the service from the load-balancers and the backend servers */ fix mistaken cumin alias 2396984 wikitext text/x-wiki nwv3pt03krd84hd1d5712pw34ynt72p 0 4108 2396718 2396713 2026-03-30T12:14:13Z ScheduleDeploymentBot 37566 Add [[gerrit:1261477]] to Monday, March 30 UTC afternoon backport window 2396718 wikitext text/x-wiki jumrnehbcb8iozbyqnh2bcy1uy6czyx 2396719 2396718 2026-03-30T12:14:24Z ScheduleDeploymentBot 37566 Add [[gerrit:1262199]] to Monday, March 30 UTC afternoon backport window 2396719 wikitext text/x-wiki 0lz9rhxzeh3xep8ct5jp8h12cjufr9p 2396720 2396719 2026-03-30T12:15:03Z ScheduleDeploymentBot 37566 Add [[gerrit:1256432]] to Monday, March 30 UTC afternoon backport window 2396720 wikitext text/x-wiki d18b4qc64uekpnbxnhdvqx9aizwwhhk 2396721 2396720 2026-03-30T12:31:36Z ScheduleDeploymentBot 37566 Add [[gerrit:1264605]] to Monday, March 30 UTC afternoon backport window 2396721 wikitext text/x-wiki nrje4fxed9v1h1jmds5takgnfj8gxbd 2396723 2396721 2026-03-30T12:41:06Z ScheduleDeploymentBot 37566 Add [[gerrit:1264625]] to Monday, March 30 UTC afternoon backport window 2396723 wikitext text/x-wiki b5kpogp5odafp4f3gygaznoxa13uoge 2396742 2396723 2026-03-30T13:31:58Z ScheduleDeploymentBot 37566 Add [[gerrit:1262303]] to Monday, March 30 UTC late backport window 2396742 wikitext text/x-wiki cgj9nobiervjw5btpilrch51efgr4cg 2396752 2396742 2026-03-30T13:37:04Z Jdforrester (WMF) 468 /* {{Deployment_day|date=2026-03-30}} */ Three done. 2396752 wikitext text/x-wiki okh0pin0bqfyjh9ncvw2hleti15q6e4 2396766 2396752 2026-03-30T13:48:04Z Jdforrester (WMF) 468 /* {{Deployment_day|date=2026-03-30}} */ One not done. 2396766 wikitext text/x-wiki 99f0rtkz46r5hs0kyhfxwt075n1iufi 2396803 2396766 2026-03-30T14:18:01Z Jdforrester (WMF) 468 /* {{Deployment_day|date=2026-03-30}} */ Two done. 2396803 wikitext text/x-wiki 06x6tdlayqgxdipkka2kp7lhsl3gq9w 2396838 2396803 2026-03-30T14:58:39Z ScheduleDeploymentBot 37566 Add [[gerrit:1264652]] to Monday, March 30 UTC late backport window 2396838 wikitext text/x-wiki id9mv5uccib3zwlsibxo4rlrt0kszv6 2396840 2396838 2026-03-30T15:00:35Z KGraessle-WMF 37493 /* {{Deployment_day|date=2026-03-30}} */ 2396840 wikitext text/x-wiki h8164vja4zvjs57wk7ibku7nyfvgp45 2396845 2396840 2026-03-30T15:05:19Z ScheduleDeploymentBot 37566 Add [[gerrit:1264653]] to Monday, March 30 UTC late backport window 2396845 wikitext text/x-wiki eu7k9o5m21bomp0qt1kkj2i0ntzrnza 2396858 2396845 2026-03-30T15:29:52Z ScheduleDeploymentBot 37566 Add [[gerrit:1264662]] to Monday, March 30 UTC late backport window 2396858 wikitext text/x-wiki mklgqghil5avyxt2ryv36yun8au1mah 2396931 2396858 2026-03-30T17:40:15Z ScheduleDeploymentBot 37566 Add [[gerrit:1261732]] to Tuesday, March 31 UTC late backport window 2396931 wikitext text/x-wiki mbd2ahxnoac4qucwiqsozea1iouy4mt 2396940 2396931 2026-03-30T18:05:55Z ScheduleDeploymentBot 37566 Add [[gerrit:1264630]] to Monday, March 30 UTC late backport window 2396940 wikitext text/x-wiki k698h496gaauirvs933do5n3uh4lybs 2396985 2396940 2026-03-30T19:32:52Z CMassaro (WMF) 40102 Added an extra AW deployment for 2026-03-30 2396985 wikitext text/x-wiki chv0tezf3yog6gvkjlce0cxor4b8gif 2397182 2396985 2026-03-31T07:33:14Z SFaci-WMF 35637 /* {{Deployment_day|date=2026-03-31}} */ 2397182 wikitext text/x-wiki e1b0dg293jj1bzybz0jru8te6elhjjh 2397315 2397182 2026-03-31T10:28:13Z ScheduleDeploymentBot 37566 Add [[gerrit:1265367]] to Tuesday, March 31 UTC afternoon backport window 2397315 wikitext text/x-wiki i2sp958i6l849f56lsmkah252ojs3b5 2397316 2397315 2026-03-31T10:28:36Z ScheduleDeploymentBot 37566 Add [[gerrit:1265368]] to Tuesday, March 31 UTC afternoon backport window 2397316 wikitext text/x-wiki 4ansq61cl3wwyjkiqfed8t6jebkkqwl 2397317 2397316 2026-03-31T10:28:56Z ScheduleDeploymentBot 37566 Add [[gerrit:1265369]] to Tuesday, March 31 UTC afternoon backport window 2397317 wikitext text/x-wiki cxglnlu3wqcdy0ssaycxj13qynh9t1p 2397318 2397317 2026-03-31T10:29:21Z ScheduleDeploymentBot 37566 Add [[gerrit:1260006]] to Tuesday, March 31 UTC afternoon backport window 2397318 wikitext text/x-wiki 0ifr0b8eaqbqzwlfkinyy02bxevwlyt 0 7919 2396714 2396712 2026-03-30T12:01:01Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4006.ulsfo.wmnet 2396714 wikitext text/x-wiki bz1rl3ymc40364mdlv4lelzad9b4xnc 2396715 2396714 2026-03-30T12:01:06Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4006.ulsfo.wmnet 2396715 wikitext text/x-wiki cwxziy2050l5tn8yyyc64c7f3kj7pqy 2396716 2396715 2026-03-30T12:03:53Z Stashbot 7414 topranks: apply transport-in policy to core router transport peerings to prefer local anycast routes 2396716 wikitext text/x-wiki 304ucsty5eg0dzg0vg7mjruw4faka05 2396722 2396716 2026-03-30T12:34:59Z Stashbot 7414 moritzm: failover Ganeti master in ulsfo to ganeti4008 2396722 wikitext text/x-wiki k9rnvbzrycg43yejozrc0nhmez6ffcq 2396725 2396722 2026-03-30T13:05:15Z Stashbot 7414 jayme: disabling puppet on A:wikiube-worker-eqiad for T420436 2396725 wikitext text/x-wiki rjmlb4x9z4nl3qx6pxla5hf69sa9jpk 2396726 2396725 2026-03-30T13:05:24Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1264578|hCaptcha: Add APCu cache layer to health checker (T421204 T412947)]] 2396726 wikitext text/x-wiki 3fi2xh7m81e61hvsuruo5oozpyr094f 2396727 2396726 2026-03-30T13:05:56Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host pki-root1002.eqiad.wmnet 2396727 wikitext text/x-wiki l0ntcse50lgcxau4r6r3id9cxybh0o3 2396728 2396727 2026-03-30T13:07:15Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1264578|hCaptcha: Add APCu cache layer to health checker (T421204 T412947)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2396728 wikitext text/x-wiki p6x9xlctdy4ftps75ijr046pzroft1q 2396729 2396728 2026-03-30T13:09:46Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with sync 2396729 wikitext text/x-wiki 72i4yy1v2y8wesnaio0rdd1s213isiy 2396730 2396729 2026-03-30T13:10:51Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4005.ulsfo.wmnet 2396730 wikitext text/x-wiki raowl5ersz9db3jb1x1yxsxdzgrwu7h 2396732 2396730 2026-03-30T13:12:16Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki-root1002.eqiad.wmnet 2396732 wikitext text/x-wiki 4ras44nnhultmph1thf4rb2bz3l2q57 2396733 2396732 2026-03-30T13:12:34Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti4005.ulsfo.wmnet 2396733 wikitext text/x-wiki gio93thu9a86iu5f7fillnsq90hpuf5 2396734 2396733 2026-03-30T13:15:12Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1003.eqiad.wmnet 2396734 wikitext text/x-wiki 3eyn1nf4ifbbig4i7i9hnyvlaykluwx 2396735 2396734 2026-03-30T13:17:20Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1264578|hCaptcha: Add APCu cache layer to health checker (T421204 T412947)]] (duration: 11m 56s) 2396735 wikitext text/x-wiki b8ztbbp1rfc8txax71os92zg168lm0k 2396736 2396735 2026-03-30T13:19:01Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4005.ulsfo.wmnet 2396736 wikitext text/x-wiki sy8rs5k2riwzf3noxob2ifucsr3gndg 2396737 2396736 2026-03-30T13:19:15Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1003.eqiad.wmnet 2396737 wikitext text/x-wiki pb7v5yvoji04af4axrnnws8ql50xwsw 2396738 2396737 2026-03-30T13:19:32Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4005.ulsfo.wmnet 2396738 wikitext text/x-wiki ckrf9wn5jcvyzi0qog4wsxq7qlxc0h2 2396739 2396738 2026-03-30T13:30:09Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1004.eqiad.wmnet 2396739 wikitext text/x-wiki cf79ylczxgmn97etkhb9mdaqqhh6ymc 2396740 2396739 2026-03-30T13:30:57Z Stashbot 7414 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1264590|instrument(ReviseTone): record start of copyedit session (T419181)]], [[gerrit:1261477|Replace WANObjectCache with new MemcachedWrapper concept (T419666)]], [[gerrit:1262199|Fix match case for setting minute, week or month TTL on OrchestratorRequest (T421475)]] 2396740 wikitext text/x-wiki 3y797gr6y95ny9mnkwqh19pnrbcl1jh 2396741 2396740 2026-03-30T13:31:27Z Stashbot 7414 moritzm: rebalance Ganeti cluster in ulsfo following the completion of the migration to routed Ganeti T421044 2396741 wikitext text/x-wiki rovwpnfyeh9s73xzo1637dwoqvgrhfe 2396743 2396741 2026-03-30T13:32:51Z Stashbot 7414 jforrester@deploy1003: jforrester, migr: Backport for [[gerrit:1264590|instrument(ReviseTone): record start of copyedit session (T419181)]], [[gerrit:1261477|Replace WANObjectCache with new MemcachedWrapper concept (T419666)]], [[gerrit:1262199|Fix match case for setting minute, week or month TTL on OrchestratorRequest (T421475)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can 2396743 wikitext text/x-wiki 0njtl1mci8sc3c7nddof6nf6p0m21gg 2396745 2396743 2026-03-30T13:33:30Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti-jumbo1001.eqiad.wmnet with OS trixie 2396745 wikitext text/x-wiki h91blg6v2ago64vbr7b9cpy7cwcaej4 2396746 2396745 2026-03-30T13:34:02Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host ganeti-jumbo1001 2396746 wikitext text/x-wiki 1dwn0unqwqj6zbqg8ps9n58g50tzttc 2396747 2396746 2026-03-30T13:34:09Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1004.eqiad.wmnet 2396747 wikitext text/x-wiki 1sc9fktuo02agt1pnqob3ikd3qg7xle 2396749 2396747 2026-03-30T13:34:21Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.dns.netbox 2396749 wikitext text/x-wiki bhjg7vabiew5ilyg1qwxmc9mcsult3d 2396750 2396749 2026-03-30T13:35:47Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1005.eqiad.wmnet 2396750 wikitext text/x-wiki n7oor6u9t5qv17nb9raxfsbmzo436m9 2396751 2396750 2026-03-30T13:36:10Z Stashbot 7414 jforrester@deploy1003: jforrester, migr: Continuing with sync 2396751 wikitext text/x-wiki dmtb2679l2vaysi59sn5o4iprt04yfa 2396753 2396751 2026-03-30T13:38:33Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2396753 wikitext text/x-wiki h8ppofnjytzs2ahu5gxpyvkjspbd71v 2396754 2396753 2026-03-30T13:39:33Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.dns.netbox 2396754 wikitext text/x-wiki n9wp2b31l93wt06fdjrbqykxdahg430 2396755 2396754 2026-03-30T13:39:47Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1005.eqiad.wmnet 2396755 wikitext text/x-wiki 9vie7rwpl2816is7x0isrmwdvg0up40 2396756 2396755 2026-03-30T13:40:30Z Stashbot 7414 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1264590|instrument(ReviseTone): record start of copyedit session (T419181)]], [[gerrit:1261477|Replace WANObjectCache with new MemcachedWrapper concept (T419666)]], [[gerrit:1262199|Fix match case for setting minute, week or month TTL on OrchestratorRequest (T421475)]] (duration: 09m 33s) 2396756 wikitext text/x-wiki ad0re72fyh27no9jeq0pysgl9unoye6 2396757 2396756 2026-03-30T13:42:00Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast7002.wikimedia.org 2396757 wikitext text/x-wiki bkc9njhprgrdd3goxhilljuu6jmo1f7 2396758 2396757 2026-03-30T13:42:05Z Stashbot 7414 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1256432|Wikifunctions: Switch cache from mcrouter-wikifunctions to special access (T419666)]] 2396758 wikitext text/x-wiki ivjxtdbhdsf9no9s0kxwhrv3jdqqp8f 2396759 2396758 2026-03-30T13:42:13Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2396759 wikitext text/x-wiki 47sm2k1yggkzg4bbhkqnxosoq98pw0a 2396761 2396759 2026-03-30T13:43:49Z Stashbot 7414 jforrester@deploy1003: jforrester: Backport for [[gerrit:1256432|Wikifunctions: Switch cache from mcrouter-wikifunctions to special access (T419666)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2396761 wikitext text/x-wiki l1s9td74fn9wt9x0ixc353xfqmihzaz 2396762 2396761 2026-03-30T13:44:30Z Stashbot 7414 jforrester@deploy1003: Sync cancelled. 2396762 wikitext text/x-wiki kn1r6glppyhdenpbuify8qx8x83swa8 2396764 2396762 2026-03-30T13:47:28Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.loadbalancer.check-ipip 2396764 wikitext text/x-wiki 871grdvz00w70ncus9zbuumxssvtxsa 2396765 2396764 2026-03-30T13:47:59Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1328.eqiad.wmnet with OS trixie 2396765 wikitext text/x-wiki krvs9sn86u8d1dz7vryhpsjsuhl99oy 2396768 2396765 2026-03-30T13:48:16Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast7002.wikimedia.org 2396768 wikitext text/x-wiki mph4ehelq0885y8nt3hs8uvtgl6oxl1 2396769 2396768 2026-03-30T13:48:26Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt2002.wikimedia.org 2396769 wikitext text/x-wiki 0eenyf5nbher2helxlf26ca73pjnbs7 2396770 2396769 2026-03-30T13:48:30Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1328 2396770 wikitext text/x-wiki 7ajwpgutq7nf0hdd16fg6kjdry4t9hu 2396771 2396770 2026-03-30T13:48:40Z Stashbot 7414 jayme@cumin1003: END (FAIL) - Cookbook sre.loadbalancer.check-ipip (exit_code=99) 2396771 wikitext text/x-wiki eamuxddq4fztgxr3n8m2iak5k52c9mb 2396772 2396771 2026-03-30T13:48:43Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.netbox 2396772 wikitext text/x-wiki rbe0bxzibbz4ut1v5yryuockzbfc5t1 2396773 2396772 2026-03-30T13:49:02Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti-jumbo1001 - bking@cumin2002" 2396773 wikitext text/x-wiki esuggcfe0iupsyd3xsfxq7txozrr43z 2396774 2396773 2026-03-30T13:49:08Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ganeti-jumbo1001 - bking@cumin2002" 2396774 wikitext text/x-wiki jxad807rhsmw59w00oumzxv3rgek63j 2396775 2396774 2026-03-30T13:49:12Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2396775 wikitext text/x-wiki 08s3aq19go1x03hdfwuqon1e5rdhmnl 2396776 2396775 2026-03-30T13:49:16Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.dns.wipe-cache ganeti-jumbo1001.eqiad.wmnet 140.48.64.10.in-addr.arpa 0.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2396776 wikitext text/x-wiki ela6twngu0ienbsbt2ji2xnrx8fibzi 2396777 2396776 2026-03-30T13:49:19Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti-jumbo1001.eqiad.wmnet 140.48.64.10.in-addr.arpa 0.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2396777 wikitext text/x-wiki 7v657jimpmesgc2r5xbfpo0ta8yru84 2396778 2396777 2026-03-30T13:49:22Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti-jumbo1001 2396778 wikitext text/x-wiki ohvvc786a7wk8fp2ixveatfcuxcqt9i 2396779 2396778 2026-03-30T13:51:37Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti-jumbo1001 2396779 wikitext text/x-wiki h71b4qtbco37hpgvh5rc3100o5zjobf 2396780 2396779 2026-03-30T13:51:41Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ganeti-jumbo1001 2396780 wikitext text/x-wiki qjiagmlabglzm22r4bnirevjew047zu 2396782 2396780 2026-03-30T13:52:52Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.loadbalancer.check-ipip 2396782 wikitext text/x-wiki qtlbkfrey45ztodfsfj878xlhs4spnp 2396783 2396782 2026-03-30T13:54:03Z Stashbot 7414 jayme@cumin1003: END (FAIL) - Cookbook sre.loadbalancer.check-ipip (exit_code=99) 2396783 wikitext text/x-wiki eyn4qdmrdaf91g1zp2ne4tcwlxr6fdg 2396784 2396783 2026-03-30T13:54:08Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt2002.wikimedia.org 2396784 wikitext text/x-wiki nmvc07hkyojamg80a0loss15au74yue 2396785 2396784 2026-03-30T13:54:30Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1328 - ayounsi@cumin1003" 2396785 wikitext text/x-wiki 1lmxbkiasa4sakzbyoqe97wngds56he 2396786 2396785 2026-03-30T13:54:36Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1328 - ayounsi@cumin1003" 2396786 wikitext text/x-wiki a2xnz1eff4q2p9gxjvrpx4g8i969quw 2396787 2396786 2026-03-30T13:54:39Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2396787 wikitext text/x-wiki ksey23b8norpueshlqv85hgvkxz103z 2396788 2396787 2026-03-30T13:54:42Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1328.eqiad.wmnet 129.32.64.10.in-addr.arpa 9.2.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2396788 wikitext text/x-wiki gg0tiuyx6t1zin2a0c2eofrmz9rkznr 2396789 2396788 2026-03-30T13:54:45Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1328.eqiad.wmnet 129.32.64.10.in-addr.arpa 9.2.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2396789 wikitext text/x-wiki 4flmidu90dekcftst2mhjnzhs8r985s 2396790 2396789 2026-03-30T13:54:49Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1328 2396790 wikitext text/x-wiki 9efdbzx4zgctvdp17hbllpx8hjuxio5 2396791 2396790 2026-03-30T13:56:05Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1328 2396791 wikitext text/x-wiki g7c0y39xtypsq9f3gccbzyiwuak3597 2396792 2396791 2026-03-30T13:56:09Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1328 2396792 wikitext text/x-wiki djy8rka0n3n3eqw74k3tdcqmxkj5fx3 2396793 2396792 2026-03-30T13:59:20Z Stashbot 7414 jayme: enabling puppet on A:wikiube-worker-eqiad for T420436 2396793 wikitext text/x-wiki r2uqncj24d68wp82zesn2da45tcpdge 2396794 2396793 2026-03-30T13:59:49Z Stashbot 7414 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1264605|Instrumentation: Track clicks for user account menu experiment (T418053)]], [[gerrit:1264625|Display create account button in main menu when user is logged out. (T418053 T415647)]] 2396794 wikitext text/x-wiki rhpl38mk5ce9e8xnmbfai8s3ckk8dsr 2396795 2396794 2026-03-30T14:01:53Z Stashbot 7414 jforrester@deploy1003: emc-wmf, jforrester: Backport for [[gerrit:1264605|Instrumentation: Track clicks for user account menu experiment (T418053)]], [[gerrit:1264625|Display create account button in main menu when user is logged out. (T418053 T415647)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2396795 wikitext text/x-wiki mnu7dd64sodtfee7o3kp9qg85ih0go2 2396796 2396795 2026-03-30T14:02:22Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti-jumbo1001.eqiad.wmnet with reason: host reimage 2396796 wikitext text/x-wiki ilrkqwsmvd1iuib3h66ke2gqfnwruto 2396797 2396796 2026-03-30T14:08:03Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1328.eqiad.wmnet with reason: host reimage 2396797 wikitext text/x-wiki d0dwaninwblhs9cbpw3gyx6nkdn5sl8 2396798 2396797 2026-03-30T14:08:52Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti-jumbo1001.eqiad.wmnet with reason: host reimage 2396798 wikitext text/x-wiki tlz8iu6qt262bbups74g7mnpz6do36o 2396799 2396798 2026-03-30T14:12:32Z Stashbot 7414 jforrester@deploy1003: emc-wmf, jforrester: Continuing with sync 2396799 wikitext text/x-wiki lcxmddw9hryfq6mqkcpn6a78pr86qm3 2396800 2396799 2026-03-30T14:12:40Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1328.eqiad.wmnet with reason: host reimage 2396800 wikitext text/x-wiki c3k99h0htrzedeiudf4ifsyw8vgen29 2396801 2396800 2026-03-30T14:16:46Z Stashbot 7414 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1264605|Instrumentation: Track clicks for user account menu experiment (T418053)]], [[gerrit:1264625|Display create account button in main menu when user is logged out. (T418053 T415647)]] (duration: 16m 57s) 2396801 wikitext text/x-wiki 6ofe6o13obwz4fmn5kzh9e2zoj3ioi5 2396802 2396801 2026-03-30T14:17:53Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section es7 2396802 wikitext text/x-wiki pagbhawzn7akkdru6smjus8gcgrp87x 2396804 2396802 2026-03-30T14:18:21Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section es7 2396804 wikitext text/x-wiki 36o4d8q972k3kbodiiofwl9vtxki45u 2396805 2396804 2026-03-30T14:21:01Z Stashbot 7414 kamila@deploy1003: Started scap sync-world: Backport for [[gerrit:1262091|Enable $wgTempCategoryCollations for s3 wikis. (T419274 T419049)]] 2396805 wikitext text/x-wiki corw8y03suhumzpg9o2kamokg817avs 2396806 2396805 2026-03-30T14:22:46Z Stashbot 7414 kamila@deploy1003: kamila: Backport for [[gerrit:1262091|Enable $wgTempCategoryCollations for s3 wikis. (T419274 T419049)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2396806 wikitext text/x-wiki l3xn5rjl3g1zhrwjrxzvizh4vm6sok7 2396807 2396806 2026-03-30T14:23:11Z Stashbot 7414 cdanis: 💙cdanis@cumin1003.eqiad.wmnet ~ 🕥☕ sudo cumin 'A:cp' 'disable-puppet "cdanis CIDER 🍎"' 2396807 wikitext text/x-wiki kavhm459tf35k23xycdt7w317h723ko 2396808 2396807 2026-03-30T14:24:31Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti-jumbo1001.eqiad.wmnet with OS trixie 2396808 wikitext text/x-wiki 3lstoghg0a9hpfxjv51xec5znsymz4u 2396809 2396808 2026-03-30T14:26:46Z Stashbot 7414 kamila@deploy1003: kamila: Continuing with sync 2396809 wikitext text/x-wiki kkkotqaxae0tfxs4a2r9x7vvvsk7d4l 2396810 2396809 2026-03-30T14:31:00Z Stashbot 7414 kamila@deploy1003: Finished scap sync-world: Backport for [[gerrit:1262091|Enable $wgTempCategoryCollations for s3 wikis. (T419274 T419049)]] (duration: 09m 59s) 2396810 wikitext text/x-wiki 5bkyh52nmbwu96ot1r3c7lnoaorlalm 2396811 2396810 2026-03-30T14:31:19Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1328.eqiad.wmnet with OS trixie 2396811 wikitext text/x-wiki i861nihfrmxo23pwn2tnpc0c3ttjed8 2396812 2396811 2026-03-30T14:32:06Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1329.eqiad.wmnet with OS trixie 2396812 wikitext text/x-wiki soy75u3tbcomtp7kd6o3sizb934134k 2396813 2396812 2026-03-30T14:32:34Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1329 2396813 wikitext text/x-wiki ffcozc6428wrr2ba6o1al4h1lx3vr8k 2396814 2396813 2026-03-30T14:32:42Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.netbox 2396814 wikitext text/x-wiki ln6rizhy4krfb4xbqntnepac5nj82q1 2396815 2396814 2026-03-30T14:32:58Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) 2396815 wikitext text/x-wiki oyap98rhxicywr0hdwamw5cnzs6gymx 2396816 2396815 2026-03-30T14:36:31Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1329 - ayounsi@cumin1003" 2396816 wikitext text/x-wiki gty0zl0cdvvl36quia1miypu5gmro59 2396817 2396816 2026-03-30T14:36:37Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1329 - ayounsi@cumin1003" 2396817 wikitext text/x-wiki fur5qw7ep31s34hj2e8mtjyf7njr45e 2396818 2396817 2026-03-30T14:36:40Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2396818 wikitext text/x-wiki 4u0p83r14j688bneil62htelygype7b 2396819 2396818 2026-03-30T14:36:44Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1329.eqiad.wmnet 132.32.64.10.in-addr.arpa 2.3.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2396819 wikitext text/x-wiki mskgxp9tuwkmhd91kcilzl4owsmwtz2 2396820 2396819 2026-03-30T14:36:48Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1329.eqiad.wmnet 132.32.64.10.in-addr.arpa 2.3.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2396820 wikitext text/x-wiki ci9g6rx2fsyw1e0c6o7rw3nk6alsfmf 2396821 2396820 2026-03-30T14:36:51Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1329 2396821 wikitext text/x-wiki nll9294negtze5q0rbwulznvpg3kvdb 2396822 2396821 2026-03-30T14:37:01Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1329 2396822 wikitext text/x-wiki 00idn44spyr3mahps9ls3ol3yobi59o 2396823 2396822 2026-03-30T14:37:05Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1329 2396823 wikitext text/x-wiki fxbsslkar5ey33twn4uglcps82tvyd6 2396827 2396823 2026-03-30T14:39:26Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.loadbalancer.check-ipip 2396827 wikitext text/x-wiki rir66zkmbykn318mch29l4ae1unt4nt 2396828 2396827 2026-03-30T14:40:14Z Stashbot 7414 jayme@cumin1003: END (FAIL) - Cookbook sre.loadbalancer.check-ipip (exit_code=99) 2396828 wikitext text/x-wiki 8tvnv7olsdju0kp6j3akz98jgzqd1x2 2396829 2396828 2026-03-30T14:49:10Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1329.eqiad.wmnet with reason: host reimage 2396829 wikitext text/x-wiki ajexwut08tymqw9e2w0hg05ysgi8jxo 2396830 2396829 2026-03-30T14:50:00Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.loadbalancer.check-ipip 2396830 wikitext text/x-wiki kx3ibskk9o5c5jgvrbetdta3yitlmpg 2396831 2396830 2026-03-30T14:50:36Z Stashbot 7414 cdanis: CIDERGRINDER 🍎 now deployed globally 🚀🌍 2396831 wikitext text/x-wiki oqo09r3p8q6vc60bca4wzkjnuc59q46 2396832 2396831 2026-03-30T14:51:11Z Stashbot 7414 jayme@cumin1003: END (FAIL) - Cookbook sre.loadbalancer.check-ipip (exit_code=99) 2396832 wikitext text/x-wiki ttz7xg2c2j8gz2cmrnwthkb3o7up4a8 2396834 2396832 2026-03-30T14:52:17Z Stashbot 7414 kamila@deploy1003: Started scap sync-world: Backport for [[gerrit:1264651|Revert "Enable $wgTempCategoryCollations for s3 wikis."]] 2396834 wikitext text/x-wiki 359wn9fw3vety02b8cixwt5vs8iefex 2396835 2396834 2026-03-30T14:54:02Z Stashbot 7414 kamila@deploy1003: kamila: Backport for [[gerrit:1264651|Revert "Enable $wgTempCategoryCollations for s3 wikis."]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2396835 wikitext text/x-wiki aog2wr9f8pl57z5eueqwkoa6oy8zp64 2396836 2396835 2026-03-30T14:54:16Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1329.eqiad.wmnet with reason: host reimage 2396836 wikitext text/x-wiki a35p442d9yzm4d5zf5kaetny1ca08tv 2396837 2396836 2026-03-30T14:54:45Z Stashbot 7414 kamila@deploy1003: kamila: Continuing with sync 2396837 wikitext text/x-wiki asu22sgpc2u0bu58j2q5yxmyrygw95u 2396839 2396837 2026-03-30T14:59:00Z Stashbot 7414 kamila@deploy1003: Finished scap sync-world: Backport for [[gerrit:1264651|Revert "Enable $wgTempCategoryCollations for s3 wikis."]] (duration: 06m 42s) 2396839 wikitext text/x-wiki aehledaic8diqy4t1370hszgrtsze47 2396841 2396839 2026-03-30T15:01:45Z Stashbot 7414 fabfur: depooling cp6001 and cp6009 to upgrade haproxy to v 3.2 (T421402) 2396841 wikitext text/x-wiki 31gki132eoyea7ennefenafz9c2mlky 2396842 2396841 2026-03-30T15:02:59Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp6001.* 2396842 wikitext text/x-wiki fbvq4lfqmo743ng13yuckvs223vqdiy 2396843 2396842 2026-03-30T15:03:07Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp6009.* 2396843 wikitext text/x-wiki q26u2hthb6e47wh8xsgk1qvwb6i8p0g 2396844 2396843 2026-03-30T15:04:48Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host debmonitor1003.eqiad.wmnet 2396844 wikitext text/x-wiki l5s8vedbga5ygh2vd1xs97pxvtjz7g5 2396846 2396844 2026-03-30T15:06:12Z Stashbot 7414 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams-internal: apply 2396846 wikitext text/x-wiki 4yrjoneufbm627fpuxouhu2g3oj6704 2396847 2396846 2026-03-30T15:06:34Z Stashbot 7414 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams-internal: apply 2396847 wikitext text/x-wiki hijyalc5m2xbero8mk477vb4kp80s78 2396848 2396847 2026-03-30T15:06:43Z Stashbot 7414 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams-internal: apply 2396848 wikitext text/x-wiki gb11w9s4gjmyktqss0nemoug8r3cydr 2396849 2396848 2026-03-30T15:07:55Z Stashbot 7414 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams-internal: apply 2396849 wikitext text/x-wiki qelgei4u8tmwb6w4k1w3shokp79defn 2396850 2396849 2026-03-30T15:08:04Z Stashbot 7414 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams-internal: apply 2396850 wikitext text/x-wiki 7p3m8kv3dw38dz14lci45uwnf5g1py5 2396851 2396850 2026-03-30T15:08:40Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor1003.eqiad.wmnet 2396851 wikitext text/x-wiki 7vtmg6ff26tpi4o7ch49em8wkzesu82 2396852 2396851 2026-03-30T15:09:33Z Stashbot 7414 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams-internal: apply 2396852 wikitext text/x-wiki 309vw2wxl4yxjyvu11ip71ndr8x50b7 2396853 2396852 2026-03-30T15:09:59Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1329.eqiad.wmnet with OS trixie 2396853 wikitext text/x-wiki 8012o8jirwi346mrcsgr09b0a5id1fl 2396854 2396853 2026-03-30T15:11:28Z Stashbot 7414 dancy@deploy1003: Started deploy [releng/jenkins-deploy@6f6a192] (releasing): Grant Overall/Administer to Arnaudb 2396854 wikitext text/x-wiki ebiog7p3m9xzedlya3dmlofwhrjv7ag 2396856 2396854 2026-03-30T15:12:16Z Stashbot 7414 dancy@deploy1003: Finished deploy [releng/jenkins-deploy@6f6a192] (releasing): Grant Overall/Administer to Arnaudb (duration: 01m 01s) 2396856 wikitext text/x-wiki jbcpadlk3nstjsy6c7a8ikt4mpz35p2 2396857 2396856 2026-03-30T15:26:31Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org 2396857 wikitext text/x-wiki bixft339ektlzi8j0c2xdnrwt2m1iif 2396859 2396857 2026-03-30T15:31:50Z Stashbot 7414 urandom: upgrade cassandra-dev2001 to Cassandra 4.1.11 — T418417 2396859 wikitext text/x-wiki akeqzry6knh7am7i5npwoe2xpt0yl9z 2396860 2396859 2026-03-30T15:32:19Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org 2396860 wikitext text/x-wiki jhj2lichqow7mtrko48ie37awxzze77 2396862 2396860 2026-03-30T15:36:47Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org 2396862 wikitext text/x-wiki nti3ejouvlp80ypiyo2xjb844057ct0 2396863 2396862 2026-03-30T15:37:17Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp6001*} and A:cp - 3.2 test upgrade () 2396863 wikitext text/x-wiki hxu0m7j0e0fw2saxnfbtero1uwxi4oj 2396864 2396863 2026-03-30T15:38:51Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1330.eqiad.wmnet with OS trixie 2396864 wikitext text/x-wiki 4p1tjlwn5tymv1nw2s22mria8vkpr2n 2396865 2396864 2026-03-30T15:39:19Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1330 2396865 wikitext text/x-wiki p431hb6v1fj0hkb8o6nr4ygbdctuu44 2396866 2396865 2026-03-30T15:40:19Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.netbox 2396866 wikitext text/x-wiki 5wpvaprcxmith9e6i1sfyzpn6tw0tbw 2396867 2396866 2026-03-30T15:42:12Z Stashbot 7414 elukey: uploaded spicerack_12.3.0 to apt.wikimedia.org bookworm-wikimedia 2396867 wikitext text/x-wiki lmb8hqudriwpvu8q479ddx2yq5mgkxh 2396868 2396867 2026-03-30T15:42:26Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp6001*} and A:cp - 3.2 test upgrade () 2396868 wikitext text/x-wiki gn7ebb74edxc8l73sycxpgwn3v4bne9 2396869 2396868 2026-03-30T15:42:36Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org 2396869 wikitext text/x-wiki re3m0a1lj939fn8adjnjjzriac7rpvv 2396870 2396869 2026-03-30T15:42:40Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp6009*} and A:cp - 3.2 test upgrade () 2396870 wikitext text/x-wiki pukrt9s595gjp76m3w6wr1x3et7t8d3 2396871 2396870 2026-03-30T15:43:46Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2151.codfw.wmnet with reason: Maintenance 2396871 wikitext text/x-wiki n1xvec6df5w78xz6gl7hykxjzew9b20 2396872 2396871 2026-03-30T15:43:56Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2151 (T419635)', diff saved to https://phabricator.wikimedia.org/P89969 and previous config saved to /var/cache/conftool/dbconfig/20260330-154352-fceratto.json 2396872 wikitext text/x-wiki 3tqzgts454jmp5e2vg6qz0bhj42u220 2396873 2396872 2026-03-30T15:44:08Z Stashbot 7414 elukey: upgrade spicerack on cumin2002 2396873 wikitext text/x-wiki q3y3teg4e7oy22rqsofxrx0k3o7lzxn 2396875 2396873 2026-03-30T15:48:01Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp6009*} and A:cp - 3.2 test upgrade () 2396875 wikitext text/x-wiki 7f49pzbm6hwrr1gtjehvmopvz3exjj6 2396877 2396875 2026-03-30T15:51:53Z Stashbot 7414 fabfur: repooling cp6001 and cp6009 (T421402) 2396877 wikitext text/x-wiki obb8glw094mre39l48ezxki8v42mxnr 2396878 2396877 2026-03-30T15:51:57Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1330 - ayounsi@cumin1003" 2396878 wikitext text/x-wiki 10nnudkqmybiid9tsgu456i1ke0zvn8 2396879 2396878 2026-03-30T15:52:01Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1330 - ayounsi@cumin1003" 2396879 wikitext text/x-wiki azw1bno4m29cghyx530lbciq88boc43 2396880 2396879 2026-03-30T15:52:05Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2396880 wikitext text/x-wiki 9uopx8hn05zn7xsyz9e212xw4ugv17v 2396881 2396880 2026-03-30T15:52:10Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1330.eqiad.wmnet 163.48.64.10.in-addr.arpa 3.6.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2396881 wikitext text/x-wiki jof4p41m2qvmqniz198rrxlfvne465m 2396882 2396881 2026-03-30T15:52:14Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1330.eqiad.wmnet 163.48.64.10.in-addr.arpa 3.6.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2396882 wikitext text/x-wiki idud2nef94l3fehpgjslforagyn4jng 2396884 2396882 2026-03-30T15:52:16Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1330 2396884 wikitext text/x-wiki l6ob8f8hqbc6lc67xk6gogxpfp66fzi 2396885 2396884 2026-03-30T15:52:20Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp6009.* 2396885 wikitext text/x-wiki 1g0mui0zr8rnm4h7xj0ip07l4smxqb3 2396886 2396885 2026-03-30T15:52:24Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp6001.* 2396886 wikitext text/x-wiki 32e79rqgxudddgkbkgfy3u2ntunik33 2396887 2396886 2026-03-30T15:52:28Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1330 2396887 wikitext text/x-wiki fdinee07dkwi5h1ypw7eun3hytp3axa 2396888 2396887 2026-03-30T15:52:30Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1330 2396888 wikitext text/x-wiki 9gd4xamb5m7kzdudwmtr7pfdtkttvat 2396889 2396888 2026-03-30T15:52:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2151 (T419635)', diff saved to https://phabricator.wikimedia.org/P89970 and previous config saved to /var/cache/conftool/dbconfig/20260330-155242-fceratto.json 2396889 wikitext text/x-wiki eucfera4o5lhf1pmn3d0ldgwkf805if 2396890 2396889 2026-03-30T15:59:08Z Stashbot 7414 moritzm: rearmed keyholder on netmon* hosts following reboots 2396890 wikitext text/x-wiki j8ipy90fbevj3eozs8aecef18rj165f 2396891 2396890 2026-03-30T16:02:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P89971 and previous config saved to /var/cache/conftool/dbconfig/20260330-160251-fceratto.json 2396891 wikitext text/x-wiki 03vqa77o8harzq48gpr5v3ilfca0nzg 2396892 2396891 2026-03-30T16:04:01Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1330.eqiad.wmnet with reason: host reimage 2396892 wikitext text/x-wiki 4q1p9qw9pxqoyrw59pe1qbftm3q4dth 2396893 2396892 2026-03-30T16:09:03Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1330.eqiad.wmnet with reason: host reimage 2396893 wikitext text/x-wiki rzneukirpntcp9an54ehk6p8f1gsqjf 2396894 2396893 2026-03-30T16:11:47Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1111.eqiad.wmnet with OS trixie 2396894 wikitext text/x-wiki 817sbeecq8e6idopplvcn1fowzlqexd 2396895 2396894 2026-03-30T16:13:02Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P89972 and previous config saved to /var/cache/conftool/dbconfig/20260330-161259-fceratto.json 2396895 wikitext text/x-wiki hlseg9r7evsk6pxewz9ieb0t20jqduu 2396896 2396895 2026-03-30T16:23:10Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2151 (T419635)', diff saved to https://phabricator.wikimedia.org/P89973 and previous config saved to /var/cache/conftool/dbconfig/20260330-162307-fceratto.json 2396896 wikitext text/x-wiki 4ptjiilzote4otdgdrxw05k8s3xq448 2396897 2396896 2026-03-30T16:23:25Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2158.codfw.wmnet with reason: Maintenance 2396897 wikitext text/x-wiki h6h51romez5c4mgfzgexbz4t7orll7j 2396898 2396897 2026-03-30T16:23:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2158 (T419635)', diff saved to https://phabricator.wikimedia.org/P89974 and previous config saved to /var/cache/conftool/dbconfig/20260330-162331-fceratto.json 2396898 wikitext text/x-wiki 5l8e733dtof5ogevzxudpmusun0bhvz 2396899 2396898 2026-03-30T16:24:59Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1330.eqiad.wmnet with OS trixie 2396899 wikitext text/x-wiki kjp0dl4c8uquez3lm3otfpcyqkkf107 2396900 2396899 2026-03-30T16:32:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T419635)', diff saved to https://phabricator.wikimedia.org/P89975 and previous config saved to /var/cache/conftool/dbconfig/20260330-163239-fceratto.json 2396900 wikitext text/x-wiki 99onsqmeg28ys3a61unqleych2ck2ab 2396901 2396900 2026-03-30T16:37:44Z Stashbot 7414 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1111.eqiad.wmnet with OS trixie 2396901 wikitext text/x-wiki c09pdtcqkw8zl2mrardjdh88wkf6oox 2396902 2396901 2026-03-30T16:38:05Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1111.eqiad.wmnet with OS trixie 2396902 wikitext text/x-wiki e6lpvl7hn4a3rv0m999b3e1a4rbyxhw 2396904 2396902 2026-03-30T16:42:51Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P89976 and previous config saved to /var/cache/conftool/dbconfig/20260330-164248-fceratto.json 2396904 wikitext text/x-wiki je5b8ptx5tc9yhs45ko7kaem34hisrn 2396905 2396904 2026-03-30T16:52:59Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P89977 and previous config saved to /var/cache/conftool/dbconfig/20260330-165256-fceratto.json 2396905 wikitext text/x-wiki 63py444blbz17kbykwx0g5kcjorxcul 2396906 2396905 2026-03-30T16:54:43Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1111.eqiad.wmnet with reason: host reimage 2396906 wikitext text/x-wiki 5014gpjczl28sj26604l6n85c56sini 2396907 2396906 2026-03-30T16:58:43Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1111.eqiad.wmnet with reason: host reimage 2396907 wikitext text/x-wiki crz9coqdmseollp03iqts7pq3ol1p51 2396908 2396907 2026-03-30T17:00:39Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.loadbalancer.check-ipip 2396908 wikitext text/x-wiki aykfg7xcpzoxvudnbuv86kj2u9pa01f 2396909 2396908 2026-03-30T17:00:51Z Stashbot 7414 jayme@cumin1003: END (FAIL) - Cookbook sre.loadbalancer.check-ipip (exit_code=99) 2396909 wikitext text/x-wiki ffpzywv0qkjpyxijbjiqw0zcnyoa5bv 2396910 2396909 2026-03-30T17:03:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T419635)', diff saved to https://phabricator.wikimedia.org/P89978 and previous config saved to /var/cache/conftool/dbconfig/20260330-170305-fceratto.json 2396910 wikitext text/x-wiki 0lo3oj70rxxdto8cxrdgrbe1gc3s7a1 2396911 2396910 2026-03-30T17:03:24Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2169.codfw.wmnet with reason: Maintenance 2396911 wikitext text/x-wiki d8cjut67hkxtbb4h01ikzrdej0io945 2396912 2396911 2026-03-30T17:03:33Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2169 (T419635)', diff saved to https://phabricator.wikimedia.org/P89979 and previous config saved to /var/cache/conftool/dbconfig/20260330-170329-fceratto.json 2396912 wikitext text/x-wiki kv7qqfq2ojt5amxn3d0s2mmxe8g567r 2396913 2396912 2026-03-30T17:05:58Z Stashbot 7414 kamila@deploy1003: Started scap sync-world: Backport for [[gerrit:1264670|Enable $wgTempCategoryCollations for s3 wikis. (T419274 T419049)]] 2396913 wikitext text/x-wiki oz9nq9ud2ocl6wj9t07vbdp5rwairiu 2396914 2396913 2026-03-30T17:07:44Z Stashbot 7414 kamila@deploy1003: kamila: Backport for [[gerrit:1264670|Enable $wgTempCategoryCollations for s3 wikis. (T419274 T419049)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2396914 wikitext text/x-wiki h9a9i69tckcwdyn5ltj86mse8o7mtko 2396915 2396914 2026-03-30T17:11:54Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host aqs1023.eqiad.wmnet with OS bullseye 2396915 wikitext text/x-wiki 5jwzok5agxlc8ino5vih0a96iqi5qic 2396916 2396915 2026-03-30T17:12:03Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.move-vlan for host aqs1023 2396916 wikitext text/x-wiki 0ry4ouks3342psxqvi1dfch4bdlg11m 2396917 2396916 2026-03-30T17:12:12Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host aqs1023 2396917 wikitext text/x-wiki 8s2585aid5dvo2pf9cxnmgnxqhernhp 2396918 2396917 2026-03-30T17:12:16Z Stashbot 7414 jgreen@dns1004: START - running authdns-update 2396918 wikitext text/x-wiki 5ot915ucd5rbmf7bx3jj4ul9g42giws 2396919 2396918 2026-03-30T17:12:22Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T419635)', diff saved to https://phabricator.wikimedia.org/P89980 and previous config saved to /var/cache/conftool/dbconfig/20260330-171215-fceratto.json 2396919 wikitext text/x-wiki soy688ef9kizvega938fla1my1vdwgw 2396920 2396919 2026-03-30T17:13:53Z Stashbot 7414 jgreen@dns1004: END - running authdns-update 2396920 wikitext text/x-wiki 11c3f2ezxhwsbyy8fs1fs9b1i43mmpq 2396921 2396920 2026-03-30T17:14:19Z Stashbot 7414 kamila@deploy1003: kamila: Continuing with sync 2396921 wikitext text/x-wiki 6nqla1du1uootaikbkas00jexrshzco 2396922 2396921 2026-03-30T17:18:34Z Stashbot 7414 kamila@deploy1003: Finished scap sync-world: Backport for [[gerrit:1264670|Enable $wgTempCategoryCollations for s3 wikis. (T419274 T419049)]] (duration: 12m 36s) 2396922 wikitext text/x-wiki b1ak1ayx8x5ogi1tsle7h2237mj4qef 2396923 2396922 2026-03-30T17:21:42Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1111.eqiad.wmnet with OS trixie 2396923 wikitext text/x-wiki knvcnlx5cahr15sbs8ql0qwjy5zy1id 2396924 2396923 2026-03-30T17:22:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P89981 and previous config saved to /var/cache/conftool/dbconfig/20260330-172224-fceratto.json 2396924 wikitext text/x-wiki crqcuyz4bwv1zture11vfnuzaauu7zo 2396925 2396924 2026-03-30T17:22:41Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1023.eqiad.wmnet with reason: host reimage 2396925 wikitext text/x-wiki ajvm5vr88qlkir9zscileinebpitakj 2396927 2396925 2026-03-30T17:29:34Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1023.eqiad.wmnet with reason: host reimage 2396927 wikitext text/x-wiki sh9fg9mcbpagyr9y212wh4uimby4b6v 2396928 2396927 2026-03-30T17:29:41Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1111.* 2396928 wikitext text/x-wiki n4oac7zz69e4dav45e44c0c7ish9qc6 2396929 2396928 2026-03-30T17:32:35Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P89982 and previous config saved to /var/cache/conftool/dbconfig/20260330-173232-fceratto.json 2396929 wikitext text/x-wiki e8ban25w4qelocvssj5vs9ekmstttu6 2396930 2396929 2026-03-30T17:37:51Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling reboot on A:hcaptcha-proxy and not A:hcaptcha-proxy-ulsfo and not P{hcaptcha-proxy7002.wikimedia.org} and A:hcaptcha-proxy 2396930 wikitext text/x-wiki rcy4phq9m6xr5jexovdlkzugjx40fym 2396932 2396930 2026-03-30T17:42:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T419635)', diff saved to https://phabricator.wikimedia.org/P89983 and previous config saved to /var/cache/conftool/dbconfig/20260330-174240-fceratto.json 2396932 wikitext text/x-wiki b59jhvesnhdy89hdfusk4tie37s2u5o 2396933 2396932 2026-03-30T17:42:58Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2180.codfw.wmnet with reason: Maintenance 2396933 wikitext text/x-wiki qhq9w1vf6h3mvv6bifsxxccsqeybk5o 2396934 2396933 2026-03-30T17:43:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2180 (T419635)', diff saved to https://phabricator.wikimedia.org/P89984 and previous config saved to /var/cache/conftool/dbconfig/20260330-174305-fceratto.json 2396934 wikitext text/x-wiki 625c5iooq59fd3e3sw6068rhjy1yq1x 2396935 2396934 2026-03-30T17:44:04Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - eevans@cumin1003" 2396935 wikitext text/x-wiki 3djszermmpsrledbp4b6297oxppba3d 2396936 2396935 2026-03-30T17:44:22Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T419635)', diff saved to https://phabricator.wikimedia.org/P89985 and previous config saved to /var/cache/conftool/dbconfig/20260330-174419-fceratto.json 2396936 wikitext text/x-wiki iggnhcdlxmf3z7gs3kvin2l3qagql32 2396937 2396936 2026-03-30T17:54:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P89986 and previous config saved to /var/cache/conftool/dbconfig/20260330-175427-fceratto.json 2396937 wikitext text/x-wiki fz9lqueisofsm6ue6pf1z6jct9htka4 2396938 2396937 2026-03-30T17:56:55Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1109.eqiad.wmnet with OS trixie 2396938 wikitext text/x-wiki mb40dozc0vsispeqhkbwus6hpfrcr5h 2396939 2396938 2026-03-30T18:04:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P89987 and previous config saved to /var/cache/conftool/dbconfig/20260330-180434-fceratto.json 2396939 wikitext text/x-wiki erzmrwhkz28g76agdk6ztel91ozbwfz 2396941 2396939 2026-03-30T18:07:07Z Stashbot 7414 jforrester@deploy2002: mwscript-k8s job started: maintenance/namespaceDupes.php --wiki=frwiki --source-pseudo-namespace=Abstract_ --fix --move-talk # T420654 abstract: is now an interwiki 2396941 wikitext text/x-wiki rprqagipwd8jr3h0w2ju3scac3k0h7m 2396943 2396941 2026-03-30T18:14:47Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T419635)', diff saved to https://phabricator.wikimedia.org/P89988 and previous config saved to /var/cache/conftool/dbconfig/20260330-181443-fceratto.json 2396943 wikitext text/x-wiki qdllu2uoneq78nogis30y68q9275on8 2396944 2396943 2026-03-30T18:14:51Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2193.codfw.wmnet with reason: Maintenance 2396944 wikitext text/x-wiki gji5gzipsz1t9hx1xigoha0dw8fjr1s 2396945 2396944 2026-03-30T18:15:02Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2193 (T419635)', diff saved to https://phabricator.wikimedia.org/P89989 and previous config saved to /var/cache/conftool/dbconfig/20260330-181457-fceratto.json 2396945 wikitext text/x-wiki qix7votzr6d5xnb56lixxv25hcvgk90 2396946 2396945 2026-03-30T18:15:29Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - eevans@cumin1003" 2396946 wikitext text/x-wiki 5hwi516rty4lwp8z43ytdp9ehfsfe21 2396947 2396946 2026-03-30T18:15:32Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1023.eqiad.wmnet with OS bullseye 2396947 wikitext text/x-wiki ax0o813b07nujpqc31o980dj9xywaji 2396948 2396947 2026-03-30T18:16:15Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T419635)', diff saved to https://phabricator.wikimedia.org/P89990 and previous config saved to /var/cache/conftool/dbconfig/20260330-181611-fceratto.json 2396948 wikitext text/x-wiki k9km3mw7fbs5d4k0nsxrnt3q0illszj 2396949 2396948 2026-03-30T18:18:36Z Stashbot 7414 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1109.eqiad.wmnet with OS trixie 2396949 wikitext text/x-wiki 62eymaqrg99qmxi3t7nbqwdf3i00758 2396950 2396949 2026-03-30T18:18:51Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1109.eqiad.wmnet with OS trixie 2396950 wikitext text/x-wiki 7unnvl11uj1bvus3au9rmmruxop4bic 2396951 2396950 2026-03-30T18:19:09Z Stashbot 7414 cdobbins@cumin2002: conftool action : set/pooled=no; selector: name=cp1106.eqiad.wmnet [reason: trixie reimaging] 2396951 wikitext text/x-wiki 7nb7b010t9kb8pp82n1t1kvikko93iy 2396952 2396951 2026-03-30T18:19:56Z Stashbot 7414 cdobbins@cumin2002: START - Cookbook sre.hosts.reimage for host cp1106.eqiad.wmnet with OS trixie 2396952 wikitext text/x-wiki r3fvipjwchdoe29h47sweabc0qrm7gk 2396953 2396952 2026-03-30T18:26:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P89991 and previous config saved to /var/cache/conftool/dbconfig/20260330-182618-fceratto.json 2396953 wikitext text/x-wiki brp17qe5mdvivwibjiuj6wf39aevxm2 2396954 2396953 2026-03-30T18:31:23Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling reboot on A:hcaptcha-proxy and not A:hcaptcha-proxy-ulsfo and not P{hcaptcha-proxy7002.wikimedia.org} and A:hcaptcha-proxy 2396954 wikitext text/x-wiki g2pa8uunf0b4a4p1jkwygafm2mqfvtl 2396955 2396954 2026-03-30T18:35:31Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1109.eqiad.wmnet with reason: host reimage 2396955 wikitext text/x-wiki oxrl5f2d6vh9cvn1eiezfw53fsoj9hv 2396956 2396955 2026-03-30T18:36:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P89992 and previous config saved to /var/cache/conftool/dbconfig/20260330-183626-fceratto.json 2396956 wikitext text/x-wiki mcnti6k6x0vxrkadousvglkaf99q06w 2396957 2396956 2026-03-30T18:39:43Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1109.eqiad.wmnet with reason: host reimage 2396957 wikitext text/x-wiki 71hq11tyjgwy6vriq8s8usokimahve1 2396958 2396957 2026-03-30T18:46:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T419635)', diff saved to https://phabricator.wikimedia.org/P89993 and previous config saved to /var/cache/conftool/dbconfig/20260330-184635-fceratto.json 2396958 wikitext text/x-wiki jbnv24k347ytojbc38jl4rqwg4iqkxp 2396959 2396958 2026-03-30T18:46:53Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2197.codfw.wmnet with reason: Maintenance 2396959 wikitext text/x-wiki auywd3vy2mcxqvvusiaiv1stzrssvgv 2396960 2396959 2026-03-30T18:50:03Z Stashbot 7414 brett@dns1006: START - running authdns-update 2396960 wikitext text/x-wiki 3uxmspjph8fy1wpgd9t7pprzt5zztwp 2396961 2396960 2026-03-30T18:51:39Z Stashbot 7414 brett@dns1006: END - running authdns-update 2396961 wikitext text/x-wiki t5y70se57kmf5tp8le5xucmnoq68u2v 2396962 2396961 2026-03-30T18:52:29Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host aqs1024.eqiad.wmnet with OS bullseye 2396962 wikitext text/x-wiki 34tqalwqoawr0uldpihgyv9iyqxkh2q 2396963 2396962 2026-03-30T18:52:44Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.move-vlan for host aqs1024 2396963 wikitext text/x-wiki q0ez8pm81ig543y54xgu9p2xz06srdk 2396964 2396963 2026-03-30T18:52:47Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host aqs1024 2396964 wikitext text/x-wiki 5ha8w32gzsvefk39w39s8ml69i4gnp5 2396965 2396964 2026-03-30T18:54:16Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2214.codfw.wmnet with reason: Maintenance 2396965 wikitext text/x-wiki lu5fvg8ikjvbhue57og7t2n1kymomz4 2396966 2396965 2026-03-30T18:54:25Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2214 (T419635)', diff saved to https://phabricator.wikimedia.org/P89994 and previous config saved to /var/cache/conftool/dbconfig/20260330-185422-fceratto.json 2396966 wikitext text/x-wiki kk3w7cd353l21216gtv4grsqnd58my4 2396967 2396966 2026-03-30T19:00:02Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1109.eqiad.wmnet with OS trixie 2396967 wikitext text/x-wiki dji4kn5227wadotb35nqzymr0rz8vpm 2396968 2396967 2026-03-30T19:02:12Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T419635)', diff saved to https://phabricator.wikimedia.org/P89995 and previous config saved to /var/cache/conftool/dbconfig/20260330-190208-fceratto.json 2396968 wikitext text/x-wiki 0dlk4ne86ae46td9iqinfa4vv4mrozb 2396969 2396968 2026-03-30T19:04:01Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1024.eqiad.wmnet with reason: host reimage 2396969 wikitext text/x-wiki jfu7hp74cqmaaynjc0ygjy95z0im6ab 2396970 2396969 2026-03-30T19:06:02Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary-codfw and A:lvs (T420468) 2396970 wikitext text/x-wiki d135n76bjjed3myq2satd1an465d08t 2396971 2396970 2026-03-30T19:06:33Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-secondary-codfw and A:lvs (T420468) 2396971 wikitext text/x-wiki qdstw4pn4tg2r41d6y9l1k76irf6xg3 2396972 2396971 2026-03-30T19:08:09Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary-eqiad and A:lvs (T420468) 2396972 wikitext text/x-wiki dagcrxoeiwmk2b73g1lp54nxcepgpy9 2396973 2396972 2026-03-30T19:10:05Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1024.eqiad.wmnet with reason: host reimage 2396973 wikitext text/x-wiki rao5c1kugdzac7p0yupgurfv8ite2jh 2396974 2396973 2026-03-30T19:10:35Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-secondary-eqiad and A:lvs (T420468) 2396974 wikitext text/x-wiki kdaqdgej8uzlah52l4zy3rs3ns3kdal 2396975 2396974 2026-03-30T19:11:31Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-low-traffic-codfw and A:lvs (T420468) 2396975 wikitext text/x-wiki 3vhfdhgoka2mrf220etidj1usjmbpsi 2396976 2396975 2026-03-30T19:12:02Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-low-traffic-codfw and A:lvs (T420468) 2396976 wikitext text/x-wiki kxy5zb0qtfg5u7liyzqx4q3dyw43bse 2396977 2396976 2026-03-30T19:12:18Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-low-traffic-eqiad and A:lvs (T420468) 2396977 wikitext text/x-wiki 2t6bss5451gsu7crwul9m8zt6tsto07 2396978 2396977 2026-03-30T19:12:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P89996 and previous config saved to /var/cache/conftool/dbconfig/20260330-191216-fceratto.json 2396978 wikitext text/x-wiki em1g7wu6nzhc7m51qzvqb2f0162q2lr 2396979 2396978 2026-03-30T19:13:00Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-low-traffic-eqiad and A:lvs (T420468) 2396979 wikitext text/x-wiki akb4putejhp9yej1jive1tevabyggxk 2396980 2396979 2026-03-30T19:15:18Z Stashbot 7414 brett: Delete mw-parsoid lvs service via sudo -i cumin A:lvs-low-traffic-codfw 'ipvsadm --delete-service --tcp-service 10.2.1.92:4452' - T420468 2396980 wikitext text/x-wiki 5kmr3s9e0lqaqnkzisgdr996h942qeh 2396981 2396980 2026-03-30T19:16:21Z Stashbot 7414 brett: Delete mw-parsoid lvs service via sudo -i cumin A:lvs-secondary-codfw 'ipvsadm --delete-service --tcp-service 10.2.1.92:4452' - T420468 2396981 wikitext text/x-wiki b6t017er938prv1hx73t9a0u9k3s86j 2396982 2396981 2026-03-30T19:18:12Z Stashbot 7414 brett: Delete mw-parsoid lvs service via sudo -i cumin A:lvs-secondary-eqiad 'ipvsadm --delete-service --tcp-service 10.2.2.92:4452' - T420468 2396982 wikitext text/x-wiki lhjc33jry41dzug9k7e7fjr8d8vugri 2396983 2396982 2026-03-30T19:18:21Z Stashbot 7414 brett: Delete mw-parsoid lvs service via sudo -i cumin A:lvs-low-traffic-eqiad 'ipvsadm --delete-service --tcp-service 10.2.2.92:4452' - T420468 2396983 wikitext text/x-wiki o93lgy32f66nnc1nfq0kmjufrymuavb 2396986 2396983 2026-03-30T19:40:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T419635)', diff saved to https://phabricator.wikimedia.org/P90000 and previous config saved to /var/cache/conftool/dbconfig/20260330-194025-fceratto.json 2396986 wikitext text/x-wiki 3h65f5g0yysbq6zk201dzdmge239ff3 2396987 2396986 2026-03-30T19:40:50Z Stashbot 7414 cdobbins@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp1106.eqiad.wmnet with OS trixie 2396987 wikitext text/x-wiki q9vfjwfi6bo5jzx0z8u58ot0ewosbf0 2396988 2396987 2026-03-30T19:44:36Z Stashbot 7414 cdobbins@cumin2002: START - Cookbook sre.hosts.reimage for host cp1106.eqiad.wmnet with OS trixie 2396988 wikitext text/x-wiki n6mgosczywjaf7q9i7rzt8kvlmxb9k7 2396989 2396988 2026-03-30T19:44:55Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host aqs1025.eqiad.wmnet with OS bullseye 2396989 wikitext text/x-wiki hrctfcf6aewkbtn5ccxm1m0qedjzpsk 2396990 2396989 2026-03-30T19:45:22Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.move-vlan for host aqs1025 2396990 wikitext text/x-wiki tjcry5zt86vvztwd29w1s7zxjwf7y7b 2396991 2396990 2026-03-30T19:46:02Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.dns.netbox 2396991 wikitext text/x-wiki aujuf7kg1r52tv1klwz4j2nruasez34 2396992 2396991 2026-03-30T19:48:16Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1107.eqiad.wmnet with OS trixie 2396992 wikitext text/x-wiki bq9fwdknszaroq0p16nyn98dws2xs2d 2396993 2396992 2026-03-30T19:48:33Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host restbase2039.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2396993 wikitext text/x-wiki 2809k8h3s5f6la12b6187l75pcfw638 2396994 2396993 2026-03-30T19:49:37Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host restbase2039.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2396994 wikitext text/x-wiki kt16u2asexez8ctcdvyn2icgpdjogbz 2396995 2396994 2026-03-30T19:50:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P90001 and previous config saved to /var/cache/conftool/dbconfig/20260330-195033-fceratto.json 2396995 wikitext text/x-wiki bskj6uxu2an8ht6evsnx2gusxwnbtob 2396996 2396995 2026-03-30T20:00:11Z Stashbot 7414 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1107.eqiad.wmnet with OS trixie 2396996 wikitext text/x-wiki 757uwl5dzm475z8e2af7vqvlrs40tf3 2396997 2396996 2026-03-30T20:00:31Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1107.eqiad.wmnet with OS trixie 2396997 wikitext text/x-wiki 3ng5iki2pmwklbd9iy9jawejbzmgmre 2396998 2396997 2026-03-30T20:00:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P90002 and previous config saved to /var/cache/conftool/dbconfig/20260330-200042-fceratto.json 2396998 wikitext text/x-wiki haqlrs0pspghk1w3qmqneplebg4h9uk 2396999 2396998 2026-03-30T20:01:21Z Stashbot 7414 cdobbins@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1106.eqiad.wmnet with reason: host reimage 2396999 wikitext text/x-wiki tde1cq1xzxnkvl0qa5mdjolgbmzzspo 2397000 2396999 2026-03-30T20:02:15Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host aqs1025 - eevans@cumin1003" 2397000 wikitext text/x-wiki lbouqxxu2f78v98unt242ps1wqo7t8x 2397001 2397000 2026-03-30T20:02:20Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host aqs1025 - eevans@cumin1003" 2397001 wikitext text/x-wiki afui7v661iviasqs6i0k31o5yyh1i20 2397002 2397001 2026-03-30T20:02:23Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2397002 wikitext text/x-wiki 1unkzl5uofi2vezntbkkxf6dvaj511t 2397003 2397002 2026-03-30T20:02:26Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.dns.wipe-cache aqs1025.eqiad.wmnet 24.32.64.10.in-addr.arpa 4.2.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2397003 wikitext text/x-wiki ak3ljthl9y85zgis9agiozk9shgzsng 2397004 2397003 2026-03-30T20:02:30Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) aqs1025.eqiad.wmnet 24.32.64.10.in-addr.arpa 4.2.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2397004 wikitext text/x-wiki b7008zn9jfzt1d86xfav3t5wibkjjy3 2397005 2397004 2026-03-30T20:02:33Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host aqs1025 2397005 wikitext text/x-wiki 2qbh51rgn5zz1n3y8k2sus4ahj8m3sk 2397006 2397005 2026-03-30T20:02:42Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host aqs1025 2397006 wikitext text/x-wiki 9286k5ro9tq2rywtk46rwws7cz9tdqw 2397007 2397006 2026-03-30T20:02:45Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host aqs1025 2397007 wikitext text/x-wiki 12j0qeff182ek6popl2xw6ucbon6or7 2397008 2397007 2026-03-30T20:06:03Z Stashbot 7414 cdobbins@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1106.eqiad.wmnet with reason: host reimage 2397008 wikitext text/x-wiki 2ktn1g9ox2chgmt1sjau59l7aoaqq6w 2397009 2397008 2026-03-30T20:06:24Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching cassandra-dev200[2-3].codfw.wmnet: Applying upgrade to Cassandra 4.1.11 — T418417 - eevans@cumin1003 2397009 wikitext text/x-wiki swibxp6ld9vq3v6tocy2ealykgy4f5r 2397010 2397009 2026-03-30T20:10:22Z Stashbot 7414 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1264662|config: Enable EmailConfirmationBanner on testwiki (T421366)]] 2397010 wikitext text/x-wiki triake30y0in7rzakce36jfm63c9485 2397011 2397010 2026-03-30T20:10:54Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T419635)', diff saved to https://phabricator.wikimedia.org/P90003 and previous config saved to /var/cache/conftool/dbconfig/20260330-201050-fceratto.json 2397011 wikitext text/x-wiki 6yscr1nelimrrsdiqbjmk68xia5gvy1 2397012 2397011 2026-03-30T20:11:09Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2224.codfw.wmnet with reason: Maintenance 2397012 wikitext text/x-wiki k2xlafuzfqtr55414dpnogo1l3ufgn5 2397013 2397012 2026-03-30T20:11:25Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2224 (T419635)', diff saved to https://phabricator.wikimedia.org/P90004 and previous config saved to /var/cache/conftool/dbconfig/20260330-201116-fceratto.json 2397013 wikitext text/x-wiki 0ooi1598npel1z3gfth7lp3ghsvvfzn 2397014 2397013 2026-03-30T20:12:12Z Stashbot 7414 cjming@deploy1003: cjming, mmartorana: Backport for [[gerrit:1264662|config: Enable EmailConfirmationBanner on testwiki (T421366)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2397014 wikitext text/x-wiki asqw5u0d6c2g7aiiljm9xdn1hac4gbt 2397015 2397014 2026-03-30T20:14:02Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1025.eqiad.wmnet with reason: host reimage 2397015 wikitext text/x-wiki mojqk71vdtey99bn1dp39i2yoxpuyqc 2397016 2397015 2026-03-30T20:17:14Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1107.eqiad.wmnet with reason: host reimage 2397016 wikitext text/x-wiki 7ucy9bifvzt4pzqskjiov3aqcpl7lb3 2397017 2397016 2026-03-30T20:18:14Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1025.eqiad.wmnet with reason: host reimage 2397017 wikitext text/x-wiki rd07sutfmjjucmavouyevkt2ghy64eu 2397018 2397017 2026-03-30T20:18:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 (T419635)', diff saved to https://phabricator.wikimedia.org/P90005 and previous config saved to /var/cache/conftool/dbconfig/20260330-201831-fceratto.json 2397018 wikitext text/x-wiki pc0oljs5tvaew3zw5pdx0f6giocbn4p 2397019 2397018 2026-03-30T20:19:58Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching cassandra-dev200[2-3].codfw.wmnet: Applying upgrade to Cassandra 4.1.11 — T418417 - eevans@cumin1003 2397019 wikitext text/x-wiki doa4jf6ar1b5pogle49ww0gvev7klqn 2397020 2397019 2026-03-30T20:22:18Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1107.eqiad.wmnet with reason: host reimage 2397020 wikitext text/x-wiki com0baewjhlnifo7nj1brk2dbj6ot9x 2397021 2397020 2026-03-30T20:25:20Z Stashbot 7414 cjming@deploy1003: cjming, mmartorana: Continuing with sync 2397021 wikitext text/x-wiki nc9m7ej8zt6nz1m0z56mypezv19x3z4 2397022 2397021 2026-03-30T20:26:35Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2397022 wikitext text/x-wiki o4nb3r1zzhajga5tkj11jxhn3jabmn6 2397023 2397022 2026-03-30T20:28:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P90006 and previous config saved to /var/cache/conftool/dbconfig/20260330-202840-fceratto.json 2397023 wikitext text/x-wiki cnhxdtzz2x0u7b85x9y0fm988o3pj3f 2397025 2397023 2026-03-30T20:29:38Z Stashbot 7414 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1264662|config: Enable EmailConfirmationBanner on testwiki (T421366)]] (duration: 19m 16s) 2397025 wikitext text/x-wiki 9fu91iq4bcngjef2g7v310sb9nc20s6 2397026 2397025 2026-03-30T20:30:34Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding kafka-logging2006 to codfw - jhancock@cumin2002" 2397026 wikitext text/x-wiki do745wpmzbvllzl3o10xc3suo3yeuxf 2397027 2397026 2026-03-30T20:30:39Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding kafka-logging2006 to codfw - jhancock@cumin2002" 2397027 wikitext text/x-wiki 3a2n7jfg3v59xi6ivn46dmfku9gow69 2397028 2397027 2026-03-30T20:30:43Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2397028 wikitext text/x-wiki jpwbxzdobozowga3gxlhz9svi277h42 2397029 2397028 2026-03-30T20:31:53Z Stashbot 7414 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1262303|[EventStreamConfig] Add product_metrics.web_base.active_reader_baseline stream and product_metrics.web_base.attribution_research stream (T420621)]] 2397029 wikitext text/x-wiki 7abx1891si7fvqhtiwjgk8iawxd8wpd 2397030 2397029 2026-03-30T20:31:57Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1025.eqiad.wmnet with OS bullseye 2397030 wikitext text/x-wiki gm8wotw92lkbj1evov0s67979kylfvy 2397031 2397030 2026-03-30T20:33:44Z Stashbot 7414 cjming@deploy1003: tchin, cjming: Backport for [[gerrit:1262303|[EventStreamConfig] Add product_metrics.web_base.active_reader_baseline stream and product_metrics.web_base.attribution_research stream (T420621)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2397031 wikitext text/x-wiki 2773z0u5e8ka568yttb6svmba8sarxz 2397032 2397031 2026-03-30T20:34:50Z Stashbot 7414 cdobbins@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1106.eqiad.wmnet with OS trixie 2397032 wikitext text/x-wiki scj1eisu63jjfomd3y4gf523ye9opf8 2397033 2397032 2026-03-30T20:35:28Z Stashbot 7414 cdobbins@cumin2002: conftool action : set/pooled=yes; selector: name=cp1106.eqiad.wmnet [reason: trixie reimaging] 2397033 wikitext text/x-wiki k5nx47p20m73sw41zclesfc460dhg29 2397034 2397033 2026-03-30T20:35:44Z Stashbot 7414 cdobbins@cumin2002: conftool action : set/pooled=no; selector: name=cp1108.eqiad.wmnet [reason: trixie reimaging] 2397034 wikitext text/x-wiki 16254lhit6mcvjvlon6tbcovtavlg3r 2397035 2397034 2026-03-30T20:35:52Z Stashbot 7414 cjming@deploy1003: tchin, cjming: Continuing with sync 2397035 wikitext text/x-wiki tq2119x7i5cwfl1fxy17kvkj512fcqd 2397036 2397035 2026-03-30T20:36:12Z Stashbot 7414 cdobbins@cumin2002: START - Cookbook sre.hosts.reimage for host cp1108.eqiad.wmnet with OS trixie 2397036 wikitext text/x-wiki 2vti33u8mnk8tow7cowgou031fyjn92 2397037 2397036 2026-03-30T20:38:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P90007 and previous config saved to /var/cache/conftool/dbconfig/20260330-203848-fceratto.json 2397037 wikitext text/x-wiki 718gw8r10rz08zzkq5t2cmxeo7c5m71 2397038 2397037 2026-03-30T20:40:08Z Stashbot 7414 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1262303|[EventStreamConfig] Add product_metrics.web_base.active_reader_baseline stream and product_metrics.web_base.attribution_research stream (T420621)]] (duration: 08m 15s) 2397038 wikitext text/x-wiki acscj1pv13rfsp778toozlny7i8bh73 2397040 2397038 2026-03-30T20:41:36Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host aqs1026.eqiad.wmnet with OS bullseye 2397040 wikitext text/x-wiki nmrto7q9rsu8fulw5ezp8ne7bszd6ov 2397041 2397040 2026-03-30T20:41:56Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.move-vlan for host aqs1026 2397041 wikitext text/x-wiki mhc2grffrukdvc4wx5up5qidlt41r8s 2397042 2397041 2026-03-30T20:41:59Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host aqs1026 2397042 wikitext text/x-wiki pizvw8gtyoq157n2arsp68uxab2vl5r 2397043 2397042 2026-03-30T20:42:53Z Stashbot 7414 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1264652|Add delete-redirect to filemovers on Wikimedia Commons (T421373)]], [[gerrit:1264653|Add TestKitchenExposureResetEpoch config variable (T414738)]] 2397043 wikitext text/x-wiki 5xskzmvg35un478cmi93sowrdbwglpo 2397044 2397043 2026-03-30T20:44:29Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1107.eqiad.wmnet with OS trixie 2397044 wikitext text/x-wiki ecg9j2y94ikq234q9wmuk6qjvfp5nde 2397045 2397044 2026-03-30T20:44:38Z Stashbot 7414 cjming@deploy1003: cjming, nmw03: Backport for [[gerrit:1264652|Add delete-redirect to filemovers on Wikimedia Commons (T421373)]], [[gerrit:1264653|Add TestKitchenExposureResetEpoch config variable (T414738)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2397045 wikitext text/x-wiki erpqxptyzswyhx3qhzm73ccr0tmaej4 2397046 2397045 2026-03-30T20:45:28Z Stashbot 7414 cjming@deploy1003: cjming, nmw03: Continuing with sync 2397046 wikitext text/x-wiki muthf4ibb9u7ieihop7qrdtaxkfquw5 2397047 2397046 2026-03-30T20:45:43Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1107.* 2397047 wikitext text/x-wiki j8wtjzi3eio496u71dxvhomv7ljbz9r 2397048 2397047 2026-03-30T20:49:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 (T419635)', diff saved to https://phabricator.wikimedia.org/P90008 and previous config saved to /var/cache/conftool/dbconfig/20260330-204856-fceratto.json 2397048 wikitext text/x-wiki s0ae7hr3ovztnw0bffuoamt3z5cesw0 2397050 2397048 2026-03-30T20:49:42Z Stashbot 7414 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1264652|Add delete-redirect to filemovers on Wikimedia Commons (T421373)]], [[gerrit:1264653|Add TestKitchenExposureResetEpoch config variable (T414738)]] (duration: 06m 49s) 2397050 wikitext text/x-wiki owcnghfq6q9bfje8c637yyn3sye10f8 2397051 2397050 2026-03-30T20:51:31Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1105.eqiad.wmnet with OS trixie 2397051 wikitext text/x-wiki bovnxnjlv9ccyrobcijbzx1y6fs7i9v 2397052 2397051 2026-03-30T20:53:12Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1026.eqiad.wmnet with reason: host reimage 2397052 wikitext text/x-wiki 26s8zwl0zcs76jcinin6ap4wbd2cry1 2397053 2397052 2026-03-30T20:59:56Z Stashbot 7414 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1264630|Add event stream for logged-in reader retention experiment (T420490)]] 2397053 wikitext text/x-wiki diyojduc2o8ijhqjkboy0jdhyo5ippd 2397054 2397053 2026-03-30T21:00:12Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1026.eqiad.wmnet with reason: host reimage 2397054 wikitext text/x-wiki rwuw14pqeg2cbeaie676xtv1b9sunjx 2397055 2397054 2026-03-30T21:02:05Z Stashbot 7414 cjming@deploy1003: cjming, annet: Backport for [[gerrit:1264630|Add event stream for logged-in reader retention experiment (T420490)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2397055 wikitext text/x-wiki ct7zr7kksab40e2p0c16vs230jnsdwx 2397056 2397055 2026-03-30T21:02:26Z Stashbot 7414 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1105.eqiad.wmnet with OS trixie 2397056 wikitext text/x-wiki 3i8lzie5i75d8f9d7dv30776yhiinoj 2397057 2397056 2026-03-30T21:02:41Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1105.eqiad.wmnet with OS trixie 2397057 wikitext text/x-wiki 273x4fcm8wrr82qyuxhf4bvz64gkp56 2397058 2397057 2026-03-30T21:02:49Z Stashbot 7414 cjming@deploy1003: cjming, annet: Continuing with sync 2397058 wikitext text/x-wiki 6mh3yubzifglq2e962ivn8pte3h2yqa 2397060 2397058 2026-03-30T21:07:00Z Stashbot 7414 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1264630|Add event stream for logged-in reader retention experiment (T420490)]] (duration: 07m 04s) 2397060 wikitext text/x-wiki iv0sice0o5shjskmpgmx4kiflcdkovi 2397061 2397060 2026-03-30T21:07:43Z Stashbot 7414 cjming: end of UTC late backport window 2397061 wikitext text/x-wiki ritsiuvg0hx8n6c4rqtfi2fje7nfdsa 2397063 2397061 2026-03-30T21:14:35Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1026.eqiad.wmnet with OS bullseye 2397063 wikitext text/x-wiki jrrvf6ilhehdvu4cpn75d9xiwtrzcmj 2397064 2397063 2026-03-30T21:15:40Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host aqs1027.eqiad.wmnet with OS bullseye 2397064 wikitext text/x-wiki 7kdhpoesl8s2tn5y8crnwrch1saraqq 2397065 2397064 2026-03-30T21:16:07Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.move-vlan for host aqs1027 2397065 wikitext text/x-wiki 7lkkojy43qsg58p6l9a9u5zp66whhtp 2397066 2397065 2026-03-30T21:19:12Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1105.eqiad.wmnet with reason: host reimage 2397066 wikitext text/x-wiki mphoxmibttn5rm7lnts4xwxshpkyxt2 2397067 2397066 2026-03-30T21:23:12Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1105.eqiad.wmnet with reason: host reimage 2397067 wikitext text/x-wiki gpsgmb9nbdgrze19yw833dncxxy3zn5 2397069 2397067 2026-03-30T21:35:14Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.dns.netbox 2397069 wikitext text/x-wiki 68a4vdylvp7jgwpr1jggsm6cj85j0ey 2397070 2397069 2026-03-30T21:44:43Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1105.eqiad.wmnet with OS trixie 2397070 wikitext text/x-wiki tqbhm2uzlu9pe7xfqmangpi6jjvdnbv 2397073 2397070 2026-03-30T21:56:32Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging2006 2397073 wikitext text/x-wiki 654dshi6j3qe54tj3i3kmd35o0y4jby 2397074 2397073 2026-03-30T21:56:42Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging2006 2397074 wikitext text/x-wiki 22abhobhov3ji5pmrkakv9dfovjkeq5 2397075 2397074 2026-03-30T21:56:47Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging2007 2397075 wikitext text/x-wiki rgpgui92c0pkrft6lzb7hv0ejfk91oh 2397076 2397075 2026-03-30T21:56:55Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging2007 2397076 wikitext text/x-wiki 7hmsjgwrlizrrobd91vrk767rnagnmi 2397077 2397076 2026-03-30T21:56:59Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging2008 2397077 wikitext text/x-wiki svluxshf6ypapa9al2pm6sfsf34gaga 2397078 2397077 2026-03-30T21:57:07Z Stashbot 7414 cdobbins@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp1108.eqiad.wmnet with OS trixie 2397078 wikitext text/x-wiki l5bj2ukgvvg4t4aajzxzicjxipafb2r 2397079 2397078 2026-03-30T21:57:10Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging2008 2397079 wikitext text/x-wiki ba3iy3st4x13lxz01dqrpeak6a3do04 2397080 2397079 2026-03-30T21:57:58Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host kafka-logging2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2397080 wikitext text/x-wiki 4lx0rexzttjssx0o7pn1g5n61qfoxcm 2397081 2397080 2026-03-30T21:59:08Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2397081 wikitext text/x-wiki kbq9ezk5shh4aavrne3pzq1g91q525v 2397083 2397081 2026-03-30T22:02:51Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host kafka-logging2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2397083 wikitext text/x-wiki bc6s9eom5920ituxegm3c2jbfqbuzdo 2397084 2397083 2026-03-30T22:03:00Z Stashbot 7414 cdobbins@cumin2002: START - Cookbook sre.hosts.reimage for host cp1108.eqiad.wmnet with OS trixie 2397084 wikitext text/x-wiki ftx3sdc3kyeg4fcbusolrr0vapg0e5d 2397085 2397084 2026-03-30T22:03:04Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2397085 wikitext text/x-wiki iagzppsn2x6ddf6jtcy8qsqa68oyt17 2397088 2397085 2026-03-30T22:19:49Z Stashbot 7414 cdobbins@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1108.eqiad.wmnet with reason: host reimage 2397088 wikitext text/x-wiki s7uwsadwpwpq0zj6wt9kf17voco8j1q 2397089 2397088 2026-03-30T22:24:08Z Stashbot 7414 cdobbins@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1108.eqiad.wmnet with reason: host reimage 2397089 wikitext text/x-wiki g2chszwmx0e1zktgh1pqfltcl6m2a7z 2397090 2397089 2026-03-30T22:24:58Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host aqs1027 - eevans@cumin1003" 2397090 wikitext text/x-wiki ixengj1s8u4hkal80omb6ts46euqwxo 2397091 2397090 2026-03-30T22:25:03Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host aqs1027 - eevans@cumin1003" 2397091 wikitext text/x-wiki cgt0m1c3usyrmzg78641pv6r921jiu5 2397092 2397091 2026-03-30T22:25:07Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2397092 wikitext text/x-wiki 67rlobjloc8ejj0wxy6s0tx461wcf5e 2397094 2397092 2026-03-30T22:25:11Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.dns.wipe-cache aqs1027.eqiad.wmnet 26.32.64.10.in-addr.arpa 6.2.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2397094 wikitext text/x-wiki 6uukihtxor5x204a26zp4r5sr0vyt2m 2397095 2397094 2026-03-30T22:25:15Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) aqs1027.eqiad.wmnet 26.32.64.10.in-addr.arpa 6.2.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2397095 wikitext text/x-wiki 3h6cg28ox6lolrwrkadf4tghc64u7k6 2397096 2397095 2026-03-30T22:25:18Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host aqs1027 2397096 wikitext text/x-wiki 9xis8tpdsncr91a9mtyifqok8b0olpl 2397098 2397096 2026-03-30T22:33:20Z Stashbot 7414 cdobbins@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on lvs6002.drmrs.wmnet with reason: planned reboot 2397098 wikitext text/x-wiki pwh709qodx1jk6rm5nmsyosbnjdulas 2397100 2397098 2026-03-30T22:47:07Z Stashbot 7414 cdobbins@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs6002.drmrs.wmnet 2397100 wikitext text/x-wiki mmdhdkz16kblkxkrxzllz3khdw3m19k 2397101 2397100 2026-03-30T22:47:11Z Stashbot 7414 cdobbins@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs6002.drmrs.wmnet 2397101 wikitext text/x-wiki ffrvi28tulsgd2eavl3ve9eltogbosd 2397102 2397101 2026-03-30T22:47:32Z Stashbot 7414 cdobbins@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1108.eqiad.wmnet with OS trixie 2397102 wikitext text/x-wiki h0qgbc2p75kjeey1bw401wswqhil3a2 2397103 2397102 2026-03-30T22:48:05Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host aqs1027 2397103 wikitext text/x-wiki qzbr4t8rjdholpy01530o1f9de5ddzt 2397104 2397103 2026-03-30T22:48:08Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host aqs1027 2397104 wikitext text/x-wiki ordiseclyvuyt93pdwagf7yej19n01w 2397105 2397104 2026-03-30T22:48:54Z Stashbot 7414 cdobbins@cumin2002: conftool action : set/pooled=yes; selector: name=cp1108.eqiad.wmnet [reason: trixie reimaging] 2397105 wikitext text/x-wiki gyo3o60zkkeagnw658f18g8yebxrkr1 2397106 2397105 2026-03-30T22:49:15Z Stashbot 7414 cdobbins@cumin2002: conftool action : set/pooled=no; selector: name=cp1110.eqiad.wmnet [reason: trixie reimaging] 2397106 wikitext text/x-wiki 6rw0zggwczx3tt89ns9eibler5vituh 2397107 2397106 2026-03-30T22:54:22Z Stashbot 7414 cdobbins@cumin2002: conftool action : set/pooled=yes; selector: name=cp1110.eqiad.wmnet [reason: trixie reimaging] 2397107 wikitext text/x-wiki e8rs6atoo4w4nl1lmyfss64g0qd5fve 2397108 2397107 2026-03-30T22:54:34Z Stashbot 7414 cdobbins@cumin2002: conftool action : set/pooled=no; selector: name=cp1112.eqiad.wmnet [reason: trixie reimaging] 2397108 wikitext text/x-wiki sca12633c8xfbqn8ibz5xxlgdjqor7a 2397109 2397108 2026-03-30T22:55:22Z Stashbot 7414 cdobbins@cumin2002: conftool action : set/pooled=yes; selector: name=cp1112.eqiad.wmnet [reason: trixie reimaging] 2397109 wikitext text/x-wiki iydz8dcntl50n66qgygfcz2jmnnusck 2397110 2397109 2026-03-30T22:55:30Z Stashbot 7414 cdobbins@cumin2002: conftool action : set/pooled=no; selector: name=cp1114.eqiad.wmnet [reason: trixie reimaging] 2397110 wikitext text/x-wiki djxw01vqvz7cjj7shcco43r12b0vffs 2397111 2397110 2026-03-30T22:59:19Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1027.eqiad.wmnet with reason: host reimage 2397111 wikitext text/x-wiki 1b176q15ozzez4r0qyumhr04c4d0tjt 2397112 2397111 2026-03-30T23:00:39Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1105.* 2397112 wikitext text/x-wiki 855kp4kpcbkuflmysqpiowdf3p1u6gv 2397113 2397112 2026-03-30T23:04:56Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1027.eqiad.wmnet with reason: host reimage 2397113 wikitext text/x-wiki hduml1jzifamo2mfchnp2cbclckb8qd 2397114 2397113 2026-03-30T23:19:20Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1027.eqiad.wmnet with OS bullseye 2397114 wikitext text/x-wiki 2qfgo3elce0nkrtjgoa3tm32p0ccxzc 2397117 2397114 2026-03-31T00:15:23Z Stashbot 7414 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2397117 wikitext text/x-wiki pz53zoi3d6sa42pwyrtzhkm1q0jed0u 2397118 2397117 2026-03-31T00:25:31Z Stashbot 7414 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2397118 wikitext text/x-wiki q9h68q61ne4nqfoo1tfmnd9v4a8it4z 2397119 2397118 2026-03-31T00:26:29Z Stashbot 7414 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2397119 wikitext text/x-wiki g5fadjmbgl325hywe47efz0mhv849rb 2397120 2397119 2026-03-31T00:36:42Z Stashbot 7414 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2397120 wikitext text/x-wiki g4nrmgi3r55w2uqgyhf1y1c8rtqgcxn 2397121 2397120 2026-03-31T00:38:29Z Stashbot 7414 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2397121 wikitext text/x-wiki 44lnpne5x7dsg9oumj199u7qrd0olcz 2397122 2397121 2026-03-31T00:48:36Z Stashbot 7414 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2397122 wikitext text/x-wiki ton066r26nf95hzu47yd3amw69u5puj 2397126 2397122 2026-03-31T02:01:04Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2397126 wikitext text/x-wiki 5xru6tqa6n89asfb9kw40eliogp7gw5 2397127 2397126 2026-03-31T02:08:17Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 13s) 2397127 wikitext text/x-wiki iormo8yc6h83ocduwfar930433rdhd0 2397128 2397127 2026-03-31T02:38:12Z Stashbot 7414 swfrench@deploy1003: Started scap sync-world: Backport for [[gerrit:1264897|Revert "Enable $wgTempCategoryCollations for s3 wikis." (T419274)]] 2397128 wikitext text/x-wiki 43u2puhhy2x03bcesbwk4outsodirx6 2397129 2397128 2026-03-31T02:40:23Z Stashbot 7414 swfrench@deploy1003: swfrench: Backport for [[gerrit:1264897|Revert "Enable $wgTempCategoryCollations for s3 wikis." (T419274)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2397129 wikitext text/x-wiki s3sx00y377ris12kbb31vlf2ekvwfbm 2397130 2397129 2026-03-31T02:41:23Z Stashbot 7414 swfrench@deploy1003: swfrench: Continuing with sync 2397130 wikitext text/x-wiki ic187h4scwxvvfhrw6ljuabt07qx8bg 2397131 2397130 2026-03-31T02:45:59Z Stashbot 7414 swfrench@deploy1003: Finished scap sync-world: Backport for [[gerrit:1264897|Revert "Enable $wgTempCategoryCollations for s3 wikis." (T419274)]] (duration: 07m 27s) 2397131 wikitext text/x-wiki 05d8d616lja2t0f0elkh69y8up8buq1 2397133 2397131 2026-03-31T03:03:43Z Stashbot 7414 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.46.0-wmf.22 refs T420480 2397133 wikitext text/x-wiki lsd9g0r7hg50q2zduiasczfm7zzkbo2 2397134 2397133 2026-03-31T03:41:18Z Stashbot 7414 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.46.0-wmf.22 refs T420480 (duration: 37m 41s) 2397134 wikitext text/x-wiki 9x7euoii7khk47stfr83kcjlf1p128j 2397136 2397134 2026-03-31T04:02:33Z Stashbot 7414 mwpresync@deploy1003: Pruned MediaWiki: 1.46.0-wmf.19 (duration: 02m 29s) 2397136 wikitext text/x-wiki nnkyewl21bjv2w63hzdk3vsn3m5712a 2397137 2397136 2026-03-31T05:19:55Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org 2397137 wikitext text/x-wiki 44c9oi9pod6l4mn78ocmc2j425cdl1t 2397138 2397137 2026-03-31T05:26:06Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org 2397138 wikitext text/x-wiki qb0gaiy6afuw7chglvb3whphmpg5ib4 2397139 2397138 2026-03-31T05:28:43Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org 2397139 wikitext text/x-wiki 5lj18x4bups04nncbvraoiyhclmtzcf 2397140 2397139 2026-03-31T05:30:35Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org 2397140 wikitext text/x-wiki 5r7z4p9hj7fmnpgsc7jaey535r4gxrq 2397142 2397140 2026-03-31T06:36:05Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2157.codfw.wmnet with reason: Maintenance 2397142 wikitext text/x-wiki 70pv9o0oiajvcne1piq5xsgxzjrr5rs 2397143 2397142 2026-03-31T06:36:14Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2157 (T419635)', diff saved to https://phabricator.wikimedia.org/P90011 and previous config saved to /var/cache/conftool/dbconfig/20260331-063611-fceratto.json 2397143 wikitext text/x-wiki abim0utkirotci1r4pspj8xpwbmct30 2397144 2397143 2026-03-31T06:37:24Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host titan1001.eqiad.wmnet 2397144 wikitext text/x-wiki 7wmj6le2ab1li5lpwau44hrc6e1k35v 2397145 2397144 2026-03-31T06:37:40Z Stashbot 7414 tappof@cumin1003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host titan1001.eqiad.wmnet 2397145 wikitext text/x-wiki gmmzrac3u5b9xi03dsn4pak8mwsj8zb 2397146 2397145 2026-03-31T06:38:11Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host titan1001.eqiad.wmnet 2397146 wikitext text/x-wiki 6kfjix0m8kyjh1okdgs8fkwuonmwtsp 2397147 2397146 2026-03-31T06:38:14Z Stashbot 7414 tappof@cumin1003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host titan1001.eqiad.wmnet 2397147 wikitext text/x-wiki 3gdywn6xefc7wkfop7g5hyvacsfo5wl 2397149 2397147 2026-03-31T06:38:54Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host titan1001.eqiad.wmnet 2397149 wikitext text/x-wiki l36a0w5vkfn116nffu953ohx66lzj3s 2397150 2397149 2026-03-31T06:46:23Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host titan1001.eqiad.wmnet 2397150 wikitext text/x-wiki cx2uuz75qnxfp15egruizpkjrm7zq18 2397151 2397150 2026-03-31T06:50:01Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1165.eqiad.wmnet with reason: Maintenance 2397151 wikitext text/x-wiki 949hewpixop3628lapgds0omsitcqna 2397152 2397151 2026-03-31T06:50:21Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance 2397152 wikitext text/x-wiki bccp9sp81atvczjg07s39k4cv6fzwzy 2397153 2397152 2026-03-31T06:50:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1165 (T419635)', diff saved to https://phabricator.wikimedia.org/P90012 and previous config saved to /var/cache/conftool/dbconfig/20260331-065027-fceratto.json 2397153 wikitext text/x-wiki 6vw9jnu9mv2c17qy9hmcdwe5uhh6fw9 2397155 2397153 2026-03-31T06:55:11Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 12 hosts with reason: Downgrade to 10.11.13 2397155 wikitext text/x-wiki g1kkewij82uclw2mw8z0m7bmup7ux6s 2397156 2397155 2026-03-31T06:55:19Z Stashbot 7414 moritzm: installing postgresql-15 security updates 2397156 wikitext text/x-wiki 6rm2c6etxrnbxahdybzj2g91hnqse13 2397157 2397156 2026-03-31T06:55:39Z Stashbot 7414 ryankemper@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster search_codfw: test reboot boottime check T410577 - ryankemper@cumin2002 - T410577 2397157 wikitext text/x-wiki 694wwbkiit1pi05f4rsnuv49iyti1ey 2397158 2397157 2026-03-31T06:56:47Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2011.codfw.wmnet 2397158 wikitext text/x-wiki jr6r1sldkilaud82ezaec7o0my7x6u7 2397159 2397158 2026-03-31T06:58:39Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host titan1002.eqiad.wmnet 2397159 wikitext text/x-wiki 00z4sf3wwfj3dplkgr2qsjd1uyr914a 2397161 2397159 2026-03-31T07:02:44Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2011.codfw.wmnet 2397161 wikitext text/x-wiki 6mw2zb58vq87e9kpptqyoraqlgy5u4l 2397162 2397161 2026-03-31T07:06:06Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host titan1002.eqiad.wmnet 2397162 wikitext text/x-wiki ik2s6f5z3h2pu04i20lk1wr72q6i28z 2397163 2397162 2026-03-31T07:11:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T419635)', diff saved to https://phabricator.wikimedia.org/P90013 and previous config saved to /var/cache/conftool/dbconfig/20260331-071140-fceratto.json 2397163 wikitext text/x-wiki 732g63nbh99f1je9v6xk5lxkbpxhh9x 2397165 2397163 2026-03-31T07:14:55Z Stashbot 7414 moritzm: installing mongo-c-driver security updates 2397165 wikitext text/x-wiki h2kuepnyl5j48wl2mr3lkxkdhf7h9gc 2397166 2397165 2026-03-31T07:16:24Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1331.eqiad.wmnet with OS trixie 2397166 wikitext text/x-wiki 0c6wnizg6lpqtttybk7aoqsru8l006f 2397167 2397166 2026-03-31T07:16:52Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1331 2397167 wikitext text/x-wiki bxr09eahfmmje9835cye6vefarir1kz 2397168 2397167 2026-03-31T07:17:00Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.netbox 2397168 wikitext text/x-wiki nqpcz8liwprdea3mms5s8o4gpua8e20 2397169 2397168 2026-03-31T07:21:36Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1331 - ayounsi@cumin1003" 2397169 wikitext text/x-wiki koei20mpwczmfjn8i1z9xhm8fp60pbq 2397170 2397169 2026-03-31T07:21:42Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1331 - ayounsi@cumin1003" 2397170 wikitext text/x-wiki ocpgfnniv8zidfcqmm2z22um7vi9vga 2397171 2397170 2026-03-31T07:21:46Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2397171 wikitext text/x-wiki 1x4b7vfnvzwxdc44fhpkxsvxxpjw3p0 2397172 2397171 2026-03-31T07:21:48Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1331.eqiad.wmnet 172.48.64.10.in-addr.arpa 2.7.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2397172 wikitext text/x-wiki my05f2wk3iu44ytoz0thpv6ssxzabhs 2397173 2397172 2026-03-31T07:21:51Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1331.eqiad.wmnet 172.48.64.10.in-addr.arpa 2.7.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2397173 wikitext text/x-wiki 4254q9z6z7i3yenstrt6dyvc7bc6m3p 2397174 2397173 2026-03-31T07:21:54Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1331 2397174 wikitext text/x-wiki 4yo8udg1h79ekjqt18lnwsjelq2v6mt 2397175 2397174 2026-03-31T07:21:58Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P90014 and previous config saved to /var/cache/conftool/dbconfig/20260331-072147-fceratto.json 2397175 wikitext text/x-wiki i3ypnorokki42gy7qnrzt5c26lw9pvb 2397176 2397175 2026-03-31T07:22:49Z Stashbot 7414 ryankemper@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster search_codfw: test reboot boottime check T410577 - ryankemper@cumin2002 - T410577 2397176 wikitext text/x-wiki m61j81f73n7x60d3t7ommi77hf6gf11 2397177 2397176 2026-03-31T07:23:13Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1331 2397177 wikitext text/x-wiki 14kyyfvf9p3v1m1luhg449ni43efkz3 2397178 2397177 2026-03-31T07:23:17Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1331 2397178 wikitext text/x-wiki pchohg2onmrtnrqjcr0hjr9ut9v6rzf 2397179 2397178 2026-03-31T07:23:22Z Stashbot 7414 ryankemper: T410577 ^ cookbook did its job, ctrl+c'd after one host was rebooted. new spicerack upgrade confirmed working 2397179 wikitext text/x-wiki ksgmgaeb3kf9i617easupve4ewj51xn 2397181 2397179 2026-03-31T07:31:57Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P90015 and previous config saved to /var/cache/conftool/dbconfig/20260331-073155-fceratto.json 2397181 wikitext text/x-wiki pku36hweimtpplhtdk4cffgpa1jz1tl 2397183 2397181 2026-03-31T07:34:55Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1331.eqiad.wmnet with reason: host reimage 2397183 wikitext text/x-wiki i0xgwtcrf09jn42145t264q7z2whruy 2397184 2397183 2026-03-31T07:35:29Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.loadbalancer.check-ipip 2397184 wikitext text/x-wiki 1kfqmgewaz5pve99o2pnpiu5zyfcbzx 2397185 2397184 2026-03-31T07:35:40Z Stashbot 7414 jayme@cumin1003: END (FAIL) - Cookbook sre.loadbalancer.check-ipip (exit_code=99) 2397185 wikitext text/x-wiki jp8r534pv7c6llb46g8yr0nuxh7j8jh 2397186 2397185 2026-03-31T07:35:43Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "New deploy for MR 152 - fabfur@cumin1003" 2397186 wikitext text/x-wiki 0qhqls3z9u6ig4yuwvvqx8w6r0xqp8s 2397187 2397186 2026-03-31T07:35:46Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: New deploy for MR 152 - fabfur@cumin1003 2397187 wikitext text/x-wiki q6r6s3a61ei3okyw5wjysn24eg110un 2397188 2397187 2026-03-31T07:36:30Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: New deploy for MR 152 - fabfur@cumin1003 2397188 wikitext text/x-wiki am2pnrfc83adzk311567m9rt74ubom7 2397189 2397188 2026-03-31T07:36:36Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "New deploy for MR 152 - fabfur@cumin1003" 2397189 wikitext text/x-wiki mpthyaxr274iwb4ejro0d8gdrmodiqu 2397190 2397189 2026-03-31T07:36:44Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "New deploy for MR 152 - fabfur@cumin1003" 2397190 wikitext text/x-wiki rk26t163ji53cqfpcpz0jtoqc0lop1r 2397191 2397190 2026-03-31T07:36:48Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: New deploy for MR 152 - fabfur@cumin1003 2397191 wikitext text/x-wiki frub22k6oguujpppdvzyo18qgqmc2t9 2397192 2397191 2026-03-31T07:36:51Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.loadbalancer.check-ipip 2397192 wikitext text/x-wiki hd0ja5fr752c354s67oc5w8hr3k28hz 2397193 2397192 2026-03-31T07:37:00Z Stashbot 7414 jayme@cumin1003: END (FAIL) - Cookbook sre.loadbalancer.check-ipip (exit_code=99) 2397193 wikitext text/x-wiki c1ug4oqqxwgi6n0lp9oyh5svx7ud9xf 2397194 2397193 2026-03-31T07:37:29Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: New deploy for MR 152 - fabfur@cumin1003 2397194 wikitext text/x-wiki ozfhmqxfhyfuvcunv8bt64d3a2qjqaw 2397195 2397194 2026-03-31T07:37:33Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "New deploy for MR 152 - fabfur@cumin1003" 2397195 wikitext text/x-wiki kpxgjbq59079vt7zb2p79zeo6rk8nf1 2397196 2397195 2026-03-31T07:39:00Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1331.eqiad.wmnet with reason: host reimage 2397196 wikitext text/x-wiki qfdtznos4637pnt0cxsefn9tcllmrqo 2397197 2397196 2026-03-31T07:42:06Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T419635)', diff saved to https://phabricator.wikimedia.org/P90016 and previous config saved to /var/cache/conftool/dbconfig/20260331-074202-fceratto.json 2397197 wikitext text/x-wiki n2ml8qbiv98ly2320iscgiodt4d7akd 2397198 2397197 2026-03-31T07:42:21Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1168.eqiad.wmnet with reason: Maintenance 2397198 wikitext text/x-wiki chdlmh7gpzpv1nq2f4y9ehk5orv6vpo 2397199 2397198 2026-03-31T07:42:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1168 (T419635)', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20260331-074227-fceratto.json 2397199 wikitext text/x-wiki h7npnutoef3izd8wgyitv1ukdearesp 2397200 2397199 2026-03-31T07:44:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T419635)', diff saved to https://phabricator.wikimedia.org/P90018 and previous config saved to /var/cache/conftool/dbconfig/20260331-074442-fceratto.json 2397200 wikitext text/x-wiki 6h6clyp74yd1j60me2vs53nhyblicz4 2397201 2397200 2026-03-31T07:46:32Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2012.codfw.wmnet 2397201 wikitext text/x-wiki 9oo2tyaepkat4vw6tjv5kmjewolmsyi 2397202 2397201 2026-03-31T07:52:48Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2012.codfw.wmnet 2397202 wikitext text/x-wiki r20lx4ccqrbc1k6rqwsm73jtjt2u9sh 2397204 2397202 2026-03-31T07:54:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P90019 and previous config saved to /var/cache/conftool/dbconfig/20260331-075450-fceratto.json 2397204 wikitext text/x-wiki n5a6df18psok5pbb9tqvdf4clnfgz29 2397205 2397204 2026-03-31T07:55:02Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1331.eqiad.wmnet with OS trixie 2397205 wikitext text/x-wiki 4taeewcfkvexrl1rmwa9k60rzuk7nza 2397207 2397205 2026-03-31T07:56:10Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2013.codfw.wmnet 2397207 wikitext text/x-wiki 1q6yz5qlw72xt5n0mohsy0cg4vhmb34 2397208 2397207 2026-03-31T08:01:42Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1332.eqiad.wmnet with OS trixie 2397208 wikitext text/x-wiki 58hw2pz2rmm9bjm98hqqou4nc50a0tm 2397209 2397208 2026-03-31T08:02:11Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1332 2397209 wikitext text/x-wiki aauvzkwjkepy7b50mpt4jy93qkgjmt2 2397210 2397209 2026-03-31T08:02:33Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2013.codfw.wmnet 2397210 wikitext text/x-wiki mcp1bzp7npurcv9w6lxfh9sqwm7lrxw 2397211 2397210 2026-03-31T08:03:54Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.netbox 2397211 wikitext text/x-wiki i5oabzfkwuatbm6pxerplcexdpr1dux 2397212 2397211 2026-03-31T08:05:06Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20260331-080459-fceratto.json 2397212 wikitext text/x-wiki oit5f7aabxy4jnuiyo1ltz6de9g5mpt 2397213 2397212 2026-03-31T08:06:41Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.decommission for hosts an-tool1007.eqiad.wmnet 2397213 wikitext text/x-wiki 4bn513x7mwb6y4so2ysc2x1hbouggzo 2397214 2397213 2026-03-31T08:07:35Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.hosts.decommission for hosts an-tool1011.eqiad.wmnet 2397214 wikitext text/x-wiki god5z78x0jb9pn2po9etap7au6h3894 2397215 2397214 2026-03-31T08:07:38Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1332 - ayounsi@cumin1003" 2397215 wikitext text/x-wiki k30ah8xvlze2s0dw0f2tfdi9z9esxq4 2397216 2397215 2026-03-31T08:07:42Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1332 - ayounsi@cumin1003" 2397216 wikitext text/x-wiki pza9slt7w50nj6ad3jg05ja0trsz73f 2397217 2397216 2026-03-31T08:07:46Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2397217 wikitext text/x-wiki sswua3a2tmnk37oo959ipxxgd0d1k6r 2397218 2397217 2026-03-31T08:07:50Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1332.eqiad.wmnet 190.48.64.10.in-addr.arpa 0.9.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2397218 wikitext text/x-wiki 1g04myfcm3h9r7hkh1u5nq9aaorrn2m 2397219 2397218 2026-03-31T08:07:53Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1332.eqiad.wmnet 190.48.64.10.in-addr.arpa 0.9.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2397219 wikitext text/x-wiki i8okacvvqlj4ohgq4tzghssse82gtav 2397220 2397219 2026-03-31T08:07:56Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1332 2397220 wikitext text/x-wiki nzdydrrlp8e5wvle2243gx59wfxdlip 2397221 2397220 2026-03-31T08:10:50Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host titan2002.codfw.wmnet 2397221 wikitext text/x-wiki 034ivlby2p60b2o6cux25pcdz7489yt 2397222 2397221 2026-03-31T08:11:03Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1332 2397222 wikitext text/x-wiki 72gl8w6pmr397h69jfla31jjb05vsb8 2397223 2397222 2026-03-31T08:11:07Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1332 2397223 wikitext text/x-wiki 8rnqa942zp9r4eloq9juikz500ryx8z 2397224 2397223 2026-03-31T08:12:15Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.dns.netbox 2397224 wikitext text/x-wiki a4u6y1apd5r4hpytha8delid5s04dn7 2397225 2397224 2026-03-31T08:15:30Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T419635)', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20260331-081511-fceratto.json 2397225 wikitext text/x-wiki shi0mjr75n75uw6w864pitlg7mtco38 2397226 2397225 2026-03-31T08:15:35Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1180.eqiad.wmnet with reason: Maintenance 2397226 wikitext text/x-wiki 99qtyry5e941r81rwc96uag140cem7h 2397227 2397226 2026-03-31T08:15:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1180 (T419635)', diff saved to https://phabricator.wikimedia.org/P90020 and previous config saved to /var/cache/conftool/dbconfig/20260331-081541-fceratto.json 2397227 wikitext text/x-wiki 7lzx4ugft69ae9kasp7r8qyoim3ekmt 2397228 2397227 2026-03-31T08:16:11Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-tool1007.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brouberol@cumin1003" 2397228 wikitext text/x-wiki n4qq3i8dlbxkcleweez63hbucm7v6c6 2397229 2397228 2026-03-31T08:16:45Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-tool1007.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brouberol@cumin1003" 2397229 wikitext text/x-wiki fg5dmtpev8xo2h0hopduvbo4zw870c8 2397230 2397229 2026-03-31T08:16:47Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2397230 wikitext text/x-wiki plqhypmf6hrd7q6cexlvv8i6gbn5kmv 2397231 2397230 2026-03-31T08:16:51Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-tool1007.eqiad.wmnet 2397231 wikitext text/x-wiki sda5eybc7obz1bfrdtidb1n46snqoyb 2397232 2397231 2026-03-31T08:16:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T419635)', diff saved to https://phabricator.wikimedia.org/P90021 and previous config saved to /var/cache/conftool/dbconfig/20260331-081651-fceratto.json 2397232 wikitext text/x-wiki 82sn8lqbo5x15ii4zpnjurzcrjqanxm 2397233 2397232 2026-03-31T08:17:01Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.dns.netbox 2397233 wikitext text/x-wiki 7ygu1tu98kvye2m25z1xt7bjq4t0o5l 2397234 2397233 2026-03-31T08:17:10Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host titan2002.codfw.wmnet 2397234 wikitext text/x-wiki 08u1dt39sygqboc5q32pv9oru6ay0lf 2397235 2397234 2026-03-31T08:19:46Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2397235 wikitext text/x-wiki 5tvbek37xbf30j795l3gqo10jgjqtiw 2397236 2397235 2026-03-31T08:19:49Z Stashbot 7414 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-tool1011.eqiad.wmnet 2397236 wikitext text/x-wiki n4tglu6reyd1ugv6gowkqjdqihli9am 2397237 2397236 2026-03-31T08:23:01Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1332.eqiad.wmnet with reason: host reimage 2397237 wikitext text/x-wiki ejm577ytvg98wgj8av1u4ktf1h7weqq 2397238 2397237 2026-03-31T08:24:40Z Stashbot 7414 elukey: upgrade spicerack on cumin1003 2397238 wikitext text/x-wiki 85futg5ih5gqe3bqrjbn4orn0ub3fpj 2397239 2397238 2026-03-31T08:27:02Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P90022 and previous config saved to /var/cache/conftool/dbconfig/20260331-082700-fceratto.json 2397239 wikitext text/x-wiki lo9qtyqy4nf1a0277d72vcnhgkvha02 2397240 2397239 2026-03-31T08:29:33Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1332.eqiad.wmnet with reason: host reimage 2397240 wikitext text/x-wiki 8gnutqasgyxzyrzo44vnqe5ocnecrxo 2397241 2397240 2026-03-31T08:37:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P90023 and previous config saved to /var/cache/conftool/dbconfig/20260331-083707-fceratto.json 2397241 wikitext text/x-wiki qgz12occgxmw42y7zz654vw0f0p0zif 2397244 2397241 2026-03-31T08:44:41Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.loadbalancer.check-ipip 2397244 wikitext text/x-wiki 7tzlrw9l7p7rimxm33cd4fgyulj6j0v 2397245 2397244 2026-03-31T08:44:45Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.loadbalancer.check-ipip (exit_code=0) 2397245 wikitext text/x-wiki k4irauzwvdg5bhibf8pbudchi0xjc5o 2397248 2397245 2026-03-31T08:45:39Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1332.eqiad.wmnet with OS trixie 2397248 wikitext text/x-wiki 95h55zorj1c49ueki5km3bi89ce7uvb 2397249 2397248 2026-03-31T08:47:22Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T419635)', diff saved to https://phabricator.wikimedia.org/P90024 and previous config saved to /var/cache/conftool/dbconfig/20260331-084714-fceratto.json 2397249 wikitext text/x-wiki 0ad4ppi4wv0nb8t88l8enhqx3zsxy53 2397250 2397249 2026-03-31T08:47:36Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1187.eqiad.wmnet with reason: Maintenance 2397250 wikitext text/x-wiki 0qpsahysihl9dpff2oezmk5qr0pplgl 2397251 2397250 2026-03-31T08:47:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1187 (T419635)', diff saved to https://phabricator.wikimedia.org/P90025 and previous config saved to /var/cache/conftool/dbconfig/20260331-084742-fceratto.json 2397251 wikitext text/x-wiki n6z9htxionyd6olmse4hryn2judatga 2397252 2397251 2026-03-31T08:49:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T419635)', diff saved to https://phabricator.wikimedia.org/P90026 and previous config saved to /var/cache/conftool/dbconfig/20260331-084951-fceratto.json 2397252 wikitext text/x-wiki glo77420g831ke0r5qn6wtnlo6z5i5g 2397253 2397252 2026-03-31T08:53:12Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps2014.codfw.wmnet 2397253 wikitext text/x-wiki 7cnfkhaluhxkfymihksn55sqdiazsh4 2397254 2397253 2026-03-31T08:56:54Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1333.eqiad.wmnet with OS trixie 2397254 wikitext text/x-wiki elrifu8bh6ppe3ddvbqmdxg9r3crvc1 2397255 2397254 2026-03-31T08:57:23Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1333 2397255 wikitext text/x-wiki 4tqfp1bkqq17kzpahfwkiomznhxvv5k 2397256 2397255 2026-03-31T08:57:31Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.netbox 2397256 wikitext text/x-wiki dqwfeb7xdchikqts060kvsljwo2usba 2397257 2397256 2026-03-31T08:59:53Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2014.codfw.wmnet 2397257 wikitext text/x-wiki l0db6y1ywb659uuog5yjam5hhyufjzp 2397258 2397257 2026-03-31T09:00:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P90027 and previous config saved to /var/cache/conftool/dbconfig/20260331-085958-fceratto.json 2397258 wikitext text/x-wiki kur9uwodl3orjrysw9koedtpke7vpz9 2397259 2397258 2026-03-31T09:03:10Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "remove body from patterns - oblivian@cumin1003" 2397259 wikitext text/x-wiki 3yh11ux230nr60mzfdrphmxzan2rdku 2397260 2397259 2026-03-31T09:03:14Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: remove body from patterns - oblivian@cumin1003 2397260 wikitext text/x-wiki 1t4uc8b841scrtjubbu7vwdkrxaoaju 2397261 2397260 2026-03-31T09:04:06Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: remove body from patterns - oblivian@cumin1003 2397261 wikitext text/x-wiki 8f8j6fw065qy5tw137165ity97ik1nz 2397262 2397261 2026-03-31T09:04:09Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "remove body from patterns - oblivian@cumin1003" 2397262 wikitext text/x-wiki mjcyeel8b7ggsxc3afpmb0xc0aa134j 2397263 2397262 2026-03-31T09:07:57Z Stashbot 7414 XioNoX: pfw1-eqiad - add NAT rule - T421750 2397263 wikitext text/x-wiki 7q0cl2uehuamffql42m4yc6rmh39jok 2397266 2397263 2026-03-31T09:10:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P90029 and previous config saved to /var/cache/conftool/dbconfig/20260331-091007-fceratto.json 2397266 wikitext text/x-wiki fau3ghct3n8235tg8ev94fz0xpn09hm 2397267 2397266 2026-03-31T09:13:33Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host maps-test2001.codfw.wmnet 2397267 wikitext text/x-wiki btsp3op2evwlt5smxyy9shj34lx6vc8 2397268 2397267 2026-03-31T09:14:42Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1333 - ayounsi@cumin1003" 2397268 wikitext text/x-wiki dvcd3jou19xmxy0i0ite9o0x3qkm8aq 2397269 2397268 2026-03-31T09:14:47Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1333 - ayounsi@cumin1003" 2397269 wikitext text/x-wiki tmwknqh5mfqw2b3tgqodc3yz61bxq3h 2397270 2397269 2026-03-31T09:14:50Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2397270 wikitext text/x-wiki rrcbwk2eysjchi9k4lg7qe9pi41wy6x 2397271 2397270 2026-03-31T09:14:53Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1333.eqiad.wmnet 191.48.64.10.in-addr.arpa 1.9.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2397271 wikitext text/x-wiki ckdfilvb5e3m7bx5sgva4s45myu7bln 2397272 2397271 2026-03-31T09:14:55Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1333.eqiad.wmnet 191.48.64.10.in-addr.arpa 1.9.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2397272 wikitext text/x-wiki 0i9rlek8a7cav4dpegzfthen6i8qr3g 2397273 2397272 2026-03-31T09:14:58Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1333 2397273 wikitext text/x-wiki 2zmi4q50civnn7q3anff1tpn05xx3x6 2397274 2397273 2026-03-31T09:15:43Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1333 2397274 wikitext text/x-wiki 0bct6wormnxignsv83sjw2bkzu9edsd 2397275 2397274 2026-03-31T09:15:46Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1333 2397275 wikitext text/x-wiki ktfcms1lgssehvwi2og0sdtltjhs3s5 2397276 2397275 2026-03-31T09:20:18Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T419635)', diff saved to https://phabricator.wikimedia.org/P90030 and previous config saved to /var/cache/conftool/dbconfig/20260331-092014-fceratto.json 2397276 wikitext text/x-wiki 53o39faonhgwgbjoabbq4v82tm8tkq4 2397277 2397276 2026-03-31T09:20:33Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1201.eqiad.wmnet with reason: Maintenance 2397277 wikitext text/x-wiki 96uzfi479pbx3o0w34coo4g42qewi8b 2397278 2397277 2026-03-31T09:20:42Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1201 (T419635)', diff saved to https://phabricator.wikimedia.org/P90031 and previous config saved to /var/cache/conftool/dbconfig/20260331-092038-fceratto.json 2397278 wikitext text/x-wiki 4t7qeyvtbffcely9gnx11uoja67xfpm 2397279 2397278 2026-03-31T09:21:39Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps-test2001.codfw.wmnet 2397279 wikitext text/x-wiki lm95o93amfln9jae0eok2j84amvtnk9 2397280 2397279 2026-03-31T09:21:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T419635)', diff saved to https://phabricator.wikimedia.org/P90032 and previous config saved to /var/cache/conftool/dbconfig/20260331-092148-fceratto.json 2397280 wikitext text/x-wiki ilt09q2rc9fhwmoukwszwqj4lvgladd 2397281 2397280 2026-03-31T09:27:23Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1333.eqiad.wmnet with reason: host reimage 2397281 wikitext text/x-wiki 08fuvpgw35g4lu07hr8pc83wowisrv1 2397282 2397281 2026-03-31T09:31:59Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P90033 and previous config saved to /var/cache/conftool/dbconfig/20260331-093156-fceratto.json 2397282 wikitext text/x-wiki 1seva58vbzxfj9rku5zs9ifpwz7vs1n 2397283 2397282 2026-03-31T09:35:03Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1333.eqiad.wmnet with reason: host reimage 2397283 wikitext text/x-wiki te920ymym19xwygzcjwe804h2su1m55 2397284 2397283 2026-03-31T09:42:07Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P90034 and previous config saved to /var/cache/conftool/dbconfig/20260331-094205-fceratto.json 2397284 wikitext text/x-wiki rkw3misktuotkoamidltiisbrwx2wam 2397285 2397284 2026-03-31T09:45:24Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc2003.wikimedia.org 2397285 wikitext text/x-wiki csjd8is55b6x60hzr57vtt0i1otd2o4 2397286 2397285 2026-03-31T09:49:03Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2003.wikimedia.org 2397286 wikitext text/x-wiki 3xtmoo6h70zbqwtgx3u1n6srnlb4xry 2397287 2397286 2026-03-31T09:51:43Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1333.eqiad.wmnet with OS trixie 2397287 wikitext text/x-wiki 91kk1sfti7j6j055168j6gkpgnf3ne7 2397288 2397287 2026-03-31T09:52:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T419635)', diff saved to https://phabricator.wikimedia.org/P90035 and previous config saved to /var/cache/conftool/dbconfig/20260331-095213-fceratto.json 2397288 wikitext text/x-wiki e2gni6chiumrsofqhojifhh4ok18snd 2397289 2397288 2026-03-31T09:52:20Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{wikikube-worker1347.eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) 2397289 wikitext text/x-wiki 0rtzg7fmz5b2cx1rev1wm0xrtzdktwl 2397290 2397289 2026-03-31T09:52:24Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1347.eqiad.wmnet 2397290 wikitext text/x-wiki sb5nmgrkd7bb9tz51jw3ae1wmdfsv7d 2397291 2397290 2026-03-31T09:52:33Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance 2397291 wikitext text/x-wiki ijijw8qo477m5xsr2h5qkbfzrl8axxu 2397293 2397291 2026-03-31T09:52:56Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1347.eqiad.wmnet 2397293 wikitext text/x-wiki l9sjpd96zes0vijudbqk0vtivmen5vo 2397295 2397293 2026-03-31T09:58:16Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1347.eqiad.wmnet 2397295 wikitext text/x-wiki 9lnuc0dxm2u4ufr142i2rxsp1uz4e1f 2397296 2397295 2026-03-31T09:58:19Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1347.eqiad.wmnet 2397296 wikitext text/x-wiki 8zg48llyuirasmayftnx4qbbz2o4cfq 2397297 2397296 2026-03-31T09:58:22Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{wikikube-worker1347.eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) 2397297 wikitext text/x-wiki f2ohkpl5ylkrmofynpk3kh2m1yw57gi 2397298 2397297 2026-03-31T09:58:26Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1334.eqiad.wmnet with OS trixie 2397298 wikitext text/x-wiki ph4h2jufdsdll7b3x3og4hm8bne25ko 2397299 2397298 2026-03-31T09:58:51Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1334 2397299 wikitext text/x-wiki 5rxd5dbpm9f0tub6gzxw6g85l6mogc0 2397300 2397299 2026-03-31T09:59:01Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.netbox 2397300 wikitext text/x-wiki j4oqurqqi3ksrorza6wghe4ducqekkq 2397301 2397300 2026-03-31T09:59:57Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance 2397301 wikitext text/x-wiki 4u98hv779h52vvs7gzdj0irgrvnvpu2 2397302 2397301 2026-03-31T10:02:55Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1334 - ayounsi@cumin1003" 2397302 wikitext text/x-wiki mv61jku9qea9xn4m1hkkuuogg2sg87j 2397303 2397302 2026-03-31T10:03:00Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1334 - ayounsi@cumin1003" 2397303 wikitext text/x-wiki j5ns4nbdu98vdw2lzq75ennszfn7r6l 2397304 2397303 2026-03-31T10:03:03Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2397304 wikitext text/x-wiki omx5prf2ld0fqnq4aic166g3j3v4gj3 2397305 2397304 2026-03-31T10:03:07Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1334.eqiad.wmnet 192.48.64.10.in-addr.arpa 2.9.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2397305 wikitext text/x-wiki efqi5rz94druckspa4uteqjupes57qp 2397306 2397305 2026-03-31T10:03:10Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1334.eqiad.wmnet 192.48.64.10.in-addr.arpa 2.9.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2397306 wikitext text/x-wiki c4n6yv6abrle0arg7fz0dees6y9qchv 2397307 2397306 2026-03-31T10:03:14Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1334 2397307 wikitext text/x-wiki s76e8earrmqkya94ack4y6ozp53cgfo 2397308 2397307 2026-03-31T10:03:57Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1334 2397308 wikitext text/x-wiki mszk4wp4u03svp73fiae76phi0xkkow 2397309 2397308 2026-03-31T10:04:00Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1334 2397309 wikitext text/x-wiki fucwxt0gd9mi0h1yzaap5fspzracw51 2397310 2397309 2026-03-31T10:15:59Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1334.eqiad.wmnet with reason: host reimage 2397310 wikitext text/x-wiki 91e5pc30gbtnh9soagt0fzaggvjtbi2 2397311 2397310 2026-03-31T10:19:15Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1334.eqiad.wmnet with reason: host reimage 2397311 wikitext text/x-wiki 60hs34jqutwty6t7z7haau8k8op27ce 2397312 2397311 2026-03-31T10:21:07Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1159.eqiad.wmnet with reason: Maintenance 2397312 wikitext text/x-wiki koyu079g8r9zspwbgunlwvinkhzrrxr 2397313 2397312 2026-03-31T10:21:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1159 (T419635)', diff saved to https://phabricator.wikimedia.org/P90036 and previous config saved to /var/cache/conftool/dbconfig/20260331-102112-fceratto.json 2397313 wikitext text/x-wiki p2x1xkz84ybni78615udocw9dgfn2rb 2397319 2397313 2026-03-31T10:36:03Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1334.eqiad.wmnet with OS trixie 2397319 wikitext text/x-wiki 9jrjks4aifag2k557blo1mggyatkxm6 2397320 2397319 2026-03-31T10:38:04Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1362.eqiad.wmnet with OS trixie 2397320 wikitext text/x-wiki bjiqmjbegniwl7h5reae8ihwyelsmq0 2397321 2397320 2026-03-31T10:38:21Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1362 2397321 wikitext text/x-wiki ceo4q5grmsjos1oximbd4zsbsbpllwh 2397322 2397321 2026-03-31T10:38:29Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.netbox 2397322 wikitext text/x-wiki 3i0np0aptierec2oqqst6qlqhi3o2qx 2397323 2397322 2026-03-31T10:44:33Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2397323 wikitext text/x-wiki 028yr9mdg5bxixyht9lucdi7kryyahh 2397324 2397323 2026-03-31T10:44:38Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2397324 wikitext text/x-wiki a7gyk2rymku6zplq9i96ns9f6ypuv4z 2397325 2397324 2026-03-31T10:48:19Z Stashbot 7414 marostegui: rename table global_block_whitelist on s3 and s5 for closed.dblist wikis T420525 2397325 wikitext text/x-wiki 10b4mwv0vxr4s00fr4nij8yw39irt4o 2397326 2397325 2026-03-31T10:48:26Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1362 - ayounsi@cumin1003" 2397326 wikitext text/x-wiki 0w9aqkpbwd7p55xj6grcvcicfjimncy 2397327 2397326 2026-03-31T10:48:32Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1362 - ayounsi@cumin1003" 2397327 wikitext text/x-wiki 0xe0ijhyuxr9z6ptb023zsjx9ibo89p 2397328 2397327 2026-03-31T10:48:36Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2397328 wikitext text/x-wiki 1rt39ogjioaojt4vvxcz34tzr4wkzsw 2397329 2397328 2026-03-31T10:48:39Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1362.eqiad.wmnet 134.32.64.10.in-addr.arpa 4.3.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2397329 wikitext text/x-wiki d1fir5tgtnawwse8v93owxfcwxrbo4f 2397330 2397329 2026-03-31T10:48:42Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1362.eqiad.wmnet 134.32.64.10.in-addr.arpa 4.3.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2397330 wikitext text/x-wiki 4ywdwwiz1fnkdvn2bazehff17r9vgpo 2397331 2397330 2026-03-31T10:48:45Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1362 2397331 wikitext text/x-wiki nnumz7iaot50u69xmxqcqoh2jd3ge80 2397332 2397331 2026-03-31T10:49:26Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1362 2397332 wikitext text/x-wiki kvlpgq01mdxp43b62ybxak76bf6k1q2 2397333 2397332 2026-03-31T10:49:30Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1362 2397333 wikitext text/x-wiki 9toeugg89r7fyie04ktmrmqch160rqd 2397334 2397333 2026-03-31T10:56:45Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host titan2001.codfw.wmnet 2397334 wikitext text/x-wiki rxngi9d8am5itnbjpjv87y7xc4wwhy8 2397335 2397334 2026-03-31T10:58:09Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1003.eqiad.wmnet with OS bookworm 2397335 wikitext text/x-wiki 3bo3yyjicajigkhitrgpq1pqk55g2rf 2397336 2397335 2026-03-31T10:58:36Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.move-vlan for host dse-k8s-worker1003 2397336 wikitext text/x-wiki 999oxqp3iksa0i4szo4cko6yqanuzzx 2397338 2397336 2026-03-31T11:01:27Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1362.eqiad.wmnet with reason: host reimage 2397338 wikitext text/x-wiki cbajozavyqbahmku2k03jncsp7dosae 2397339 2397338 2026-03-31T11:01:42Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.netbox 2397339 wikitext text/x-wiki trklg2mrne4yknpbawkw75cnvo0d5kz 2397340 2397339 2026-03-31T11:04:46Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host titan2001.codfw.wmnet 2397340 wikitext text/x-wiki njc91qkixc8l5epcq67gbtojqysuj1x 2397341 2397340 2026-03-31T11:08:02Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1362.eqiad.wmnet with reason: host reimage 2397341 wikitext text/x-wiki mseqdxvs7n24tubf3hwgunb3mo5bflc 2397342 2397341 2026-03-31T11:10:41Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host dse-k8s-worker1003 - btullis@cumin1003" 2397342 wikitext text/x-wiki rs82y63sgemhm6c8efqlgib1pm6ygpa 2397343 2397342 2026-03-31T11:10:47Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host dse-k8s-worker1003 - btullis@cumin1003" 2397343 wikitext text/x-wiki on9z1uzs00lj4atffqc9h2d2zy253rl 2397344 2397343 2026-03-31T11:10:50Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2397344 wikitext text/x-wiki hp6vf5747x9wyq819dvrxketm0dmerf 2397345 2397344 2026-03-31T11:10:54Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1003.eqiad.wmnet 178.32.64.10.in-addr.arpa 8.7.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2397345 wikitext text/x-wiki exjj354ell6l3mkkkh4d9cb4lmwqmdw 2397346 2397345 2026-03-31T11:10:56Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1003.eqiad.wmnet 178.32.64.10.in-addr.arpa 8.7.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2397346 wikitext text/x-wiki 6siluvscyvlclad1sxva8tv33nu0nwy 2397347 2397346 2026-03-31T11:10:59Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1003 2397347 wikitext text/x-wiki rbmovucf97rpn8qno01gncip18zyh44 2397348 2397347 2026-03-31T11:12:43Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2397348 wikitext text/x-wiki eu1z5lxnlp7wicg6t3np9miu4t2mz6g 2397349 2397348 2026-03-31T11:12:53Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2397349 wikitext text/x-wiki 54mlhpeluf3s4j6j1yy9tdb8bv2v872 2397350 2397349 2026-03-31T11:13:15Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2397350 wikitext text/x-wiki oslqvfh74ckwvj8o7m1u83na70ywxlt 2397351 2397350 2026-03-31T11:13:23Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2397351 wikitext text/x-wiki g98xsjpeikaum0gxtulwjf3vt40goms 2397352 2397351 2026-03-31T11:16:19Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1003 2397352 wikitext text/x-wiki o41zigzakkbnfkbsz5udu4b2gitqw87 2397353 2397352 2026-03-31T11:16:23Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host dse-k8s-worker1003 2397353 wikitext text/x-wiki 2il14d43ytg93t35d8ue1tolokq4fng 2397354 2397353 2026-03-31T11:22:55Z Stashbot 7414 moritzm: installing gnupg2 security updates 2397354 wikitext text/x-wiki b036k21c9yjh4mm6fxa4x4bjyqvnwc8 2397355 2397354 2026-03-31T11:24:48Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1362.eqiad.wmnet with OS trixie 2397355 wikitext text/x-wiki atdnrzyifv836zvauw9ani4ynlb7apz 2397356 2397355 2026-03-31T11:32:33Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1003.eqiad.wmnet with reason: host reimage 2397356 wikitext text/x-wiki pijkw2uq24xmygvzcqd8ux4dgitqoyw 2397357 2397356 2026-03-31T11:34:01Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2148.codfw.wmnet with reason: Maintenance 2397357 wikitext text/x-wiki j00zewpxs5tjjsi46mcv4768g1bdg1k 2397358 2397357 2026-03-31T11:34:11Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2148 (T419635)', diff saved to https://phabricator.wikimedia.org/P90042 and previous config saved to /var/cache/conftool/dbconfig/20260331-113407-fceratto.json 2397358 wikitext text/x-wiki o5sxh6vrsicvp3c18x35r223zz9myp2 2397360 2397358 2026-03-31T11:37:13Z Stashbot 7414 moritzm: upgrade Envoy on IDM to 1.35.9 T419637 2397360 wikitext text/x-wiki o4r4zx60o8iqw8yyxmsibjlcor4cwjo 2397361 2397360 2026-03-31T11:39:00Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1003.eqiad.wmnet with reason: host reimage 2397361 wikitext text/x-wiki jdq56uljcb1k2dytf18rlt10mlq03u4 2397362 2397361 2026-03-31T11:44:32Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1264581|Switch from InterwikiSortingPrepend to the ULS config]] 2397362 wikitext text/x-wiki j7ulkf9zpltcmk1ohrra5a0wx5shpa4 2397363 2397362 2026-03-31T11:49:06Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1264581|Switch from InterwikiSortingPrepend to the ULS config]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2397363 wikitext text/x-wiki kt3okw1jq1qso949rbd8e6zhbjo9lvq 2397364 2397363 2026-03-31T11:50:13Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with sync 2397364 wikitext text/x-wiki 6vfc2hrrkkd04ff9mnm9qy2jy97b2f4 2397365 2397364 2026-03-31T11:50:20Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T419635)', diff saved to https://phabricator.wikimedia.org/P90044 and previous config saved to /var/cache/conftool/dbconfig/20260331-115016-fceratto.json 2397365 wikitext text/x-wiki sv5hphgc0l4v1sg71xn9a12nr10gj45 2397366 2397365 2026-03-31T11:57:52Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1264581|Switch from InterwikiSortingPrepend to the ULS config]] (duration: 13m 19s) 2397366 wikitext text/x-wiki 9gdv7m8hqopuu9otdtwyv0mclmzktrd 2397367 2397366 2026-03-31T11:58:04Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1003.eqiad.wmnet with OS bookworm 2397367 wikitext text/x-wiki 8labi2u8phj2059rmkq2utxtwnnu6ox 0 12858 2396744 2356841 2026-03-30T13:32:52Z EMcFarland-WMF 48104 /* Guidelines */ Clarify that communicating in #wikimedia-operations on LiberaChat requires having your nick registered with NickServ 2396744 wikitext text/x-wiki q1013qepz878abh00psmyarjt1yz9ue 2396748 2396744 2026-03-30T13:34:19Z EMcFarland-WMF 48104 /* How to submit a patch for backport */ Add a reminder to check that IRC nick is registered with NickServ 2396748 wikitext text/x-wiki 4c5d8bil8xzole06qzplnis6uwwcea1 0 15652 2397072 121409 2026-03-30T21:54:18Z LSandergreen-WMF 46345 /* Deploying a CiviCRM upgrade */ 2397072 wikitext text/x-wiki 9lvl89pcl7pd0tbu0ar485hl42bs50w 2397087 2397072 2026-03-30T22:08:37Z LSandergreen-WMF 46345 /* Creating a CiviCRM upgrade patch */ 2397087 wikitext text/x-wiki 281vojt7e4500541itbp4b7lu9t9pmz 2397093 2397087 2026-03-30T22:25:12Z LSandergreen-WMF 46345 2397093 wikitext text/x-wiki cg0eeoiuxcjqkdvfvfzczzk20bra6hf 2397097 2397093 2026-03-30T22:25:33Z LSandergreen-WMF 46345 2397097 wikitext text/x-wiki 1odcpma3v9ei1jntd9eegm90t60vh72 0 17290 2396731 2396707 2026-03-30T13:10:55Z Stashbot 7414 hashar: gerrit: abandon mediawiki/core changes that are 2+years old and are attached to a task (`Bug: Txxxx`) 2396731 wikitext text/x-wiki t55vpn24m2pdl27fbtuyl6hjg4o7qzy 2397024 2396731 2026-03-30T20:29:23Z Stashbot 7414 bd808: Unblock 73.162.0.0/16 (T421549) 2397024 wikitext text/x-wiki qhb9f9t47cdmzwvwjai5wdnetm0h1i3 2397049 2397024 2026-03-30T20:49:18Z Stashbot 7414 bd808: Unblock 95.89.191.0/24 (T421774) 2397049 wikitext text/x-wiki 0tsen4ew67o9mvqgq81zkdjvk0o3p5h 2397059 2397049 2026-03-30T21:04:35Z Stashbot 7414 bd808: Unblock 95.88.0.0/15 (T421774) 2397059 wikitext text/x-wiki 3v3q8lmh47mtbnuupj74kxjagmy4m3v 2397082 2397059 2026-03-30T22:01:04Z Stashbot 7414 bd808: Unblock 78.20.0.0/14 (T421586) 2397082 wikitext text/x-wiki ep0i0m5f8cgov5wnvap3u2imgh3oxu8 498 30942 2396760 2396711 2026-03-30T13:42:48Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment codfw1dev 2396760 wikitext text/x-wiki kdn1e6d8hnc4ovymo87rt6v53ez6svz 2396763 2396760 2026-03-30T13:45:52Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=0) on deployment codfw1dev 2396763 wikitext text/x-wiki oi2ybc5ahrpp9s71qroixm6ld4lrtlx 2396767 2396763 2026-03-30T13:48:11Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services 2396767 wikitext text/x-wiki 2quzgeon74aqk80tjz6vjvqmlii9qn2 2396781 2396767 2026-03-30T13:52:19Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services 2396781 wikitext text/x-wiki sxz4ufx9far5afpj01a6toj2daylioq 2397164 2396781 2026-03-31T07:12:38Z Stashbot 7414 godog: start maint on neutron - T421054 2397164 wikitext text/x-wiki mqmwprjf3x2g3hsu4nmukyj74oymrwi 2397180 2397164 2026-03-31T07:27:00Z Stashbot 7414 godog: neutron maint done - T421054 2397180 wikitext text/x-wiki 1jg6urwskhyt02zua1sqqjpo9nw0ja7 0 33280 2397292 2361559 2026-03-31T09:52:48Z Novem Linguae 23682 add 2397292 wikitext text/x-wiki 3pwsrc73ai81nmwpg4et3z52rqzj5ok 2397294 2397292 2026-03-31T09:53:05Z Novem Linguae 23682 ce 2397294 wikitext text/x-wiki ojioh09orbp4rk1ftg5nm8oibrr6b2w 0 440819 2397203 2396493 2026-03-31T07:53:22Z Meno25 1446 update redirected link 2397203 wikitext text/x-wiki tu3bavx12555w20l1i1dmpc1xe7q5y3 498 443947 2396724 2394886 2026-03-30T12:54:44Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 7b4b75736e (l10n updates: ja, ru) 2396724 wikitext text/x-wiki 2u2w5h3bjwahzj5xrg7a1ibm5u89trl 117 447050 2396824 2396030 2026-03-30T14:38:28Z Addshore 138 /* Any logs? */ Reply 2396824 wikitext text/x-wiki aswejdm29wuzuvv0dytc5pmeumafh11 0 447260 2396883 2394973 2026-03-30T15:52:16Z LToscano (WMF) 7191 /* Renewing a existing intermediate */ 2396883 wikitext text/x-wiki l2puee53zbn8ixrgfmj4f5gj3vtuy6y 2397086 2396883 2026-03-30T22:04:25Z Taavi 13997 /* Renewing a existing intermediate */ LISTGAP 2397086 wikitext text/x-wiki oi985nh3p7onuj8bwl2j96em4drz2z2 498 448391 2397242 2199943 2026-03-31T08:42:33Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console 2397242 wikitext text/x-wiki alytj9lvrhtes6z1o818u6gt76ya0mv 2397246 2397242 2026-03-31T08:45:06Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) 2397246 wikitext text/x-wiki ozxygr2i3vgii8g1mkd4cft0rvgks98 0 448814 2396876 2396691 2026-03-30T15:49:14Z FCeratto-WMF 39104 Add monitoring 2396876 wikitext text/x-wiki rug6ortml2rb8clmrfc9t5143ig8nwy 0 449160 2396874 2396618 2026-03-30T15:44:40Z Dexbot 30554 Bot: Updating the report 2396874 wikitext text/x-wiki d3oql29tc5ndcgw8xr65bigxe9bnx0g 2397116 2396874 2026-03-31T00:01:22Z Dexbot 30554 Bot: Updating the report 2397116 wikitext text/x-wiki prc710njw3twcv4voawhlsa36dsdops 2397148 2397116 2026-03-31T06:38:19Z Dexbot 30554 Bot: Updating the report 2397148 wikitext text/x-wiki jo6kt22ujxepnwtv2w7xbg3e6vmn31e 2397154 2397148 2026-03-31T06:53:22Z Dexbot 30554 Bot: Updating the report 2397154 wikitext text/x-wiki ryu4grmsplryd6zgpggmt0krklx8nhi 2397314 2397154 2026-03-31T10:24:09Z Dexbot 30554 Bot: Updating the report 2397314 wikitext text/x-wiki juqmdmo2fcn1q2w0qwtnbsgzx6whv6q 2397359 2397314 2026-03-31T11:36:20Z Dexbot 30554 Bot: Updating the report 2397359 wikitext text/x-wiki 3atbgufe7fu3v66ksf66dvps8ksj144 0 449219 2397071 1954131 2026-03-30T21:50:02Z LSandergreen-WMF 46345 /* Deploying to staging */ 2397071 wikitext text/x-wiki cvrmwa7rjwjc5f7zt958lwmrh2v0xla 0 450435 2397160 2310566 2026-03-31T07:01:30Z JMeybohm (WMF) 16709 /* Add in Service Mesh */ Link to existing documentation instead of duplicating it 2397160 wikitext text/x-wiki sa6u4fxzrlz9vs8sbun9u8ik3p8f48u 498 452890 2397124 2396582 2026-03-31T01:21:44Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23775778726 (https://github.com/cluebotng/component-configs/commits/47f8e20c39e29b952f6dbbd04917970802ce1a0b) 2397124 wikitext text/x-wiki fj78pjzekrv38vlyr9rouc88x609l37 2397141 2397124 2026-03-31T06:07:07Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23782957628 (https://github.com/cluebotng/component-configs/commits/7888bbd75773dc064d78ad2ee8949f1540eab0fd) 2397141 wikitext text/x-wiki 3f53kwfzqdt7ieheox4r16v4gdv2ulv 0 453048 2396855 2252247 2026-03-30T15:11:40Z Nemoralis 21869 typo 2396855 wikitext text/x-wiki ggf6xln0s8n1k2eyxruofj05f1vxy9z 116 453906 2396861 2396616 2026-03-30T15:36:25Z Gitlabaccountapprovalbot 37332 @atsuko was approved. 2396861 wikitext text/x-wiki hep239zpuqgdwhzccpi8qd4c2s0sbwt 2397337 2396861 2026-03-31T11:00:24Z Gitlabaccountapprovalbot 37332 krrishsehgal was rejected. 2397337 wikitext text/x-wiki fmlzecqewv8gdarqc5wj6u6tuihp5n0 0 459115 2397135 2396595 2026-03-31T03:45:32Z AndrewBogott (WMF) 51047 /* Neutron */ 2397135 wikitext text/x-wiki hqdh8lihyl8be2003kum4vvbyix369w 0 459138 2397068 2396681 2026-03-30T21:32:45Z BryanDavis 1604 more detail on the hCaptcha SaaS product 2397068 wikitext text/x-wiki 9wlmxx2l760xleddyquncbi6h13g2fp 0 459695 2396926 2393409 2026-03-30T17:23:37Z APaskulin (WMF) 14880 /* Core API */ Clarify deprecation 2396926 wikitext text/x-wiki 0lc3snbfrn2fmyfdozymwsypc2jutzd 2396942 2396926 2026-03-30T18:10:20Z APaskulin (WMF) 14880 /* Feed API */ Link to new landing page 2396942 wikitext text/x-wiki mqrn7uj68jc0ace365094r67tuayx4w 2397099 2396942 2026-03-30T22:40:45Z APaskulin (WMF) 14880 /* Rate limits */ New rate limits apply 2397099 wikitext text/x-wiki jr3rybz4cfg7tpx48chjyc2m6r9asyw 0 459697 2397206 2396651 2026-03-31T07:55:49Z GModena (WMF) 20194 2397206 wikitext text/x-wiki 6odlrsy0z6hfoelywuyevbaxm71w2fk 0 459699 2396833 2396602 2026-03-30T14:51:36Z Stashbot 7414 ejegg: standalone (IPN listener) SmashPig upgraded from 545f0b10 to abf8682a 2396833 wikitext text/x-wiki rcxqtdc1qr6kr5xmsh8782j4uu8ofjy 2397039 2396833 2026-03-30T20:40:44Z Stashbot 7414 eileen: civicrm upgraded from 58426b1e to 3724cc2d 2397039 wikitext text/x-wiki cep5cxo1xyzmk93eg29v6h5mrquzvxa 2397062 2397039 2026-03-30T21:09:25Z Stashbot 7414 eileen: civicrm upgraded from 3724cc2d to 7d299b48 2397062 wikitext text/x-wiki mhnppl7fr30bne7hmx7uc4fpvijobbj 2397115 2397062 2026-03-30T23:19:26Z Stashbot 7414 eileen: civicrm upgraded from 7d299b48 to cf871dd3 2397115 wikitext text/x-wiki 6chamffd04lqloa9ni2ze1l5yzz39qv 2397123 2397115 2026-03-31T01:00:20Z Stashbot 7414 eileen: civicrm upgraded from cf871dd3 to 591bef29 2397123 wikitext text/x-wiki 10xg5kg5mgfa3n8u4tupp67o6qf87fg 2397132 2397123 2026-03-31T02:46:04Z Stashbot 7414 eileen: civicrm upgraded from 591bef29 to c3cc3562 2397132 wikitext text/x-wiki ki0ucd7s5wwe0tphd1ilnkig3m5lhr5 0 459731 2397243 2379690 2026-03-31T08:44:36Z DPogorzelski-WMF 48508 2397243 wikitext text/x-wiki a8jmhmvtsieh857dbw11m5d74h2x9nb 2397247 2397243 2026-03-31T08:45:25Z DPogorzelski-WMF 48508 2397247 wikitext text/x-wiki 0l0kolvvl57uhu7h6pgap4s1d4cm74e 2397264 2397247 2026-03-31T09:08:18Z DPogorzelski-WMF 48508 2397264 wikitext text/x-wiki gzqwyhql7t5s2ft8vpcrbdrg3hke5u5 2397265 2397264 2026-03-31T09:09:06Z DPogorzelski-WMF 48508 2397265 wikitext text/x-wiki 5bys9oqathm59kdzpehdxjdzmqvujyd 0 459993 2396903 2395298 2026-03-30T16:41:02Z APaskulin (WMF) 14880 /* Change your code to use Lift Wing API */ Split sentence for readability 2396903 wikitext text/x-wiki r7bk7wxh2fpzkvv050ls6dv3nqlellj 2 460015 2397125 2026-03-31T01:23:46Z Krinkle 44 Created page with "In Feb 2026 we learned that '''new articles often take weeks or months to surface in Google Search'''. The current investigation started after a discussion on the English Wikipedia VPT about new pages that get patrolled but not indexed in Google. == Background == When unregistered or inexperienced users create an article on English Wikipedia, that page is marked "noindex" by default. This means the web page contains <code><meta name="robots" content="noindex,nofollow"><..." 2397125 wikitext text/x-wiki pp6ezoercwrnn0l8ytb4f9c0rhjuza7