Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.46.0-wmf.26 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 0 4108 2407414 2407329 2026-04-29T12:57:12Z ScheduleDeploymentBot 37566 Add [[gerrit:1279330]] to Wednesday, April 29 UTC afternoon backport window 2407414 wikitext text/x-wiki 14kq8lgwyr07fywaephlsiv8oqyh492 2407415 2407414 2026-04-29T12:57:19Z ScheduleDeploymentBot 37566 Add [[gerrit:1279331]] to Wednesday, April 29 UTC afternoon backport window 2407415 wikitext text/x-wiki mfooq5uf1he2dxzt8l1fa5nbkszkr0x 2407430 2407415 2026-04-29T13:18:04Z ScheduleDeploymentBot 37566 Add [[gerrit:1279335]] to Wednesday, April 29 UTC afternoon backport window 2407430 wikitext text/x-wiki 4k3iv5yewypisnwb9wq0j4omipyiymg 2407652 2407430 2026-04-29T17:30:19Z ScheduleDeploymentBot 37566 Add [[gerrit:1279431]] to Wednesday, April 29 UTC late backport window 2407652 wikitext text/x-wiki fq18g2gfvlrq9k5yrj7fv8c0xdoyp6y 2407666 2407652 2026-04-29T18:39:33Z ScheduleDeploymentBot 37566 Add [[gerrit:1277569]] to Wednesday, April 29 UTC late backport window 2407666 wikitext text/x-wiki 9e388u94ww2r5g57ugf9w6d33gwo4c7 2407667 2407666 2026-04-29T18:42:40Z ScheduleDeploymentBot 37566 Add [[gerrit:1279448]] to Wednesday, April 29 UTC late backport window 2407667 wikitext text/x-wiki 9uir7dpwwj4qlfcsc1awj3xl6ovvtxw 2407671 2407667 2026-04-29T18:50:30Z ScheduleDeploymentBot 37566 Add [[gerrit:1279406]] to Wednesday, April 29 UTC late backport window 2407671 wikitext text/x-wiki gbmvuatie7u3b2jzut8lta3rlnj83lp 2407672 2407671 2026-04-29T18:50:54Z ScheduleDeploymentBot 37566 Add [[gerrit:1279422]] to Wednesday, April 29 UTC late backport window 2407672 wikitext text/x-wiki o4c21dbim399ixgmqiryh968oi4jlnk 2407673 2407672 2026-04-29T18:51:03Z ScheduleDeploymentBot 37566 Add [[gerrit:1279404]] to Wednesday, April 29 UTC late backport window 2407673 wikitext text/x-wiki mfl9chlxfeo6qu7q30q9cpdfn0q9ab2 2407677 2407673 2026-04-29T19:12:53Z ScheduleDeploymentBot 37566 Add [[gerrit:1277770]] to Wednesday, April 29 UTC late backport window 2407677 wikitext text/x-wiki i8upddccb7frr5n96hipkw7zn8n342j 2407678 2407677 2026-04-29T19:13:07Z ScheduleDeploymentBot 37566 Add [[gerrit:1279450]] to Wednesday, April 29 UTC late backport window 2407678 wikitext text/x-wiki mryenco4tjj3sod8efcmo7vf6f5fj33 2407679 2407678 2026-04-29T19:13:41Z ScheduleDeploymentBot 37566 Add [[gerrit:1279452]] to Wednesday, April 29 UTC late backport window 2407679 wikitext text/x-wiki 9zth95auogbr6juhhp1md29ptodj8t1 2407682 2407679 2026-04-29T19:17:01Z ScheduleDeploymentBot 37566 Add [[gerrit:1279430]] to Wednesday, April 29 UTC late backport window 2407682 wikitext text/x-wiki jdat3au0r78ym25hketkxw231rfr5ac 2407690 2407682 2026-04-29T19:28:49Z ScheduleDeploymentBot 37566 Add [[gerrit:1278363]] to Wednesday, April 29 UTC late backport window 2407690 wikitext text/x-wiki b8sbkglgm008l8uuoxvos73vyuel831 2407692 2407690 2026-04-29T19:40:19Z ScheduleDeploymentBot 37566 Add [[gerrit:1265959]] to Wednesday, April 29 UTC late backport window 2407692 wikitext text/x-wiki dn1rkpzrygndnaq47fx6uqrkz8h1n42 2407693 2407692 2026-04-29T19:42:35Z ZhaoFJx 34635 /* Week of May 04 */ rv 2407693 wikitext text/x-wiki pnp8lwsffw02xwlvlogp9gzcxtmzm9d 2407699 2407693 2026-04-29T20:13:28Z ScheduleDeploymentBot 37566 Add [[gerrit:1279476]] to Thursday, April 30 UTC morning backport window 2407699 wikitext text/x-wiki 7j728prawwqxuolmtmgh93hh984ph13 2407807 2407699 2026-04-30T00:39:19Z ScheduleDeploymentBot 37566 Add [[gerrit:1279453]] to Thursday, April 30 UTC afternoon backport window 2407807 wikitext text/x-wiki mfnski0s3eu845w78pzk2m2eqyrrg5p 2407808 2407807 2026-04-30T00:39:50Z ScheduleDeploymentBot 37566 Add [[gerrit:1279454]] to Thursday, April 30 UTC late backport window 2407808 wikitext text/x-wiki 323uu7tfifwvharilcyf2aphs5aeek1 2407821 2407808 2026-04-30T03:43:20Z ScheduleDeploymentBot 37566 Add [[gerrit:1279477]] to Monday, May 04 UTC morning backport window 2407821 wikitext text/x-wiki k34j4hct9c1ysr0bpv5q0x6zkauoprm 2407822 2407821 2026-04-30T04:39:30Z ScheduleDeploymentBot 37566 Add [[gerrit:1274928]] to Thursday, April 30 UTC late backport window 2407822 wikitext text/x-wiki aohv3qof9uuoih4bu7do25huy2b3ifo 2407846 2407822 2026-04-30T05:47:46Z ScheduleDeploymentBot 37566 Add [[gerrit:1279079]] to Thursday, April 30 UTC morning backport window 2407846 wikitext text/x-wiki 53erushtfhg8maaqxxtag3xz4zr7phd 2407923 2407846 2026-04-30T09:06:01Z ScheduleDeploymentBot 37566 Add [[gerrit:1280210]] to Thursday, April 30 UTC afternoon backport window 2407923 wikitext text/x-wiki 0slwbjsaj7j6u91s04ybdngnzqjghld 2407929 2407923 2026-04-30T09:22:04Z ScheduleDeploymentBot 37566 Add [[gerrit:1280226]] to Thursday, April 30 UTC afternoon backport window 2407929 wikitext text/x-wiki iygarcjjzh3unx6kkkdk6ib60bdzv10 0 7919 2407359 2407358 2026-04-29T12:00:48Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1223.eqiad.wmnet with reason: Reimage to Trixie 2407359 wikitext text/x-wiki mkdv2w4of7x12awqnjmbp54d51i6mpu 2407360 2407359 2026-04-29T12:00:53Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1223: Reimage to Trixie 2407360 wikitext text/x-wiki 03p1k44e061cmmag76s7qkypnpv84at 2407361 2407360 2026-04-29T12:00:57Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2177.codfw.wmnet with reason: Reimage to Trixie 2407361 wikitext text/x-wiki p3sfphd9piqo58mt9qy9jn32z2rerzo 2407362 2407361 2026-04-29T12:01:03Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2177: Reimage to Trixie 2407362 wikitext text/x-wiki htfnbphfuzke2gm60npcbwtz1cxqgv0 2407363 2407362 2026-04-29T12:01:21Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1223: Reimage to Trixie 2407363 wikitext text/x-wiki jyzhmqt1ullomb8xkr4mp3ove2roc2o 2407364 2407363 2026-04-29T12:01:23Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2177: Reimage to Trixie 2407364 wikitext text/x-wiki 37gi2tpuuwpm0bxhemvw5jmy30ysu62 2407365 2407364 2026-04-29T12:03:05Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1223.eqiad.wmnet with OS trixie 2407365 wikitext text/x-wiki 34zt2byejge5hrtumf8xwjwmfztchds 2407366 2407365 2026-04-29T12:03:17Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2177.codfw.wmnet with OS trixie 2407366 wikitext text/x-wiki r0ezzda83z1epepofjrwrk3kmge23l3 2407367 2407366 2026-04-29T12:03:51Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1197.eqiad.wmnet with reason: Reimage to Trixie 2407367 wikitext text/x-wiki 8qcj2cdbjit3qn8hamy5uj6oulnt1xe 2407368 2407367 2026-04-29T12:03:56Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1197: Reimage to Trixie 2407368 wikitext text/x-wiki 97yvuswl4ers1gsc94itor9k2gt4ha8 2407369 2407368 2026-04-29T12:04:05Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2204.codfw.wmnet with reason: Reimage to Trixie 2407369 wikitext text/x-wiki hg5soo58bg06hwlvdb2iu9zok0x9byh 2407370 2407369 2026-04-29T12:04:11Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2204: Reimage to Trixie 2407370 wikitext text/x-wiki n44j090z386l2nhk73nhdhtvz5k145f 2407371 2407370 2026-04-29T12:04:31Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2204: Reimage to Trixie 2407371 wikitext text/x-wiki 1v4272kq4a6b4k4tyiehfjiksyos08p 2407372 2407371 2026-04-29T12:04:34Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1197: Reimage to Trixie 2407372 wikitext text/x-wiki e28kq2hgg5zgth6cnuxgr1b2gga37kg 2407373 2407372 2026-04-29T12:05:44Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1197.eqiad.wmnet with OS trixie 2407373 wikitext text/x-wiki pena3s3yhin44ph7r71615yraoz905h 2407374 2407373 2026-04-29T12:05:59Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2204.codfw.wmnet with OS trixie 2407374 wikitext text/x-wiki gohsrlvkftzq4687fryeifpwc5clbok 2407375 2407374 2026-04-29T12:08:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T419961)', diff saved to https://phabricator.wikimedia.org/P91911 and previous config saved to /var/cache/conftool/dbconfig/20260429-120837-fceratto.json 2407375 wikitext text/x-wiki 211vzs8r1qrodujda3wjgrmdo8agnrb 2407376 2407375 2026-04-29T12:09:02Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance 2407376 wikitext text/x-wiki 9knfzxprz9t1wqa793x8fxng3doahtx 2407377 2407376 2026-04-29T12:09:10Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1231 (T419961)', diff saved to https://phabricator.wikimedia.org/P91912 and previous config saved to /var/cache/conftool/dbconfig/20260429-120907-fceratto.json 2407377 wikitext text/x-wiki ov0x5y6m5jcz0445chkxfu55x0dd70t 2407378 2407377 2026-04-29T12:11:41Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2407378 wikitext text/x-wiki 2aklrjx7pkv6al2uv8191981cnx02f4 2407379 2407378 2026-04-29T12:14:16Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2407379 wikitext text/x-wiki 9sj418bh6t89yn84ncbhgj476nctl45 2407380 2407379 2026-04-29T12:14:37Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2407380 wikitext text/x-wiki n36czm783vvy2janydvhyzl0itm18nu 2407381 2407380 2026-04-29T12:16:29Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2407381 wikitext text/x-wiki nl5j7n14lzdhfgy5e3vdflb3d1jmy51 2407382 2407381 2026-04-29T12:16:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T419961)', diff saved to https://phabricator.wikimedia.org/P91913 and previous config saved to /var/cache/conftool/dbconfig/20260429-121633-fceratto.json 2407382 wikitext text/x-wiki e2xewjt5pciabpqfqwyved7x9pm37dj 2407383 2407382 2026-04-29T12:18:36Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1223.eqiad.wmnet with reason: host reimage 2407383 wikitext text/x-wiki rc52hga9yzvwu4bl34d1r56m0j4ghws 2407384 2407383 2026-04-29T12:19:52Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1197.eqiad.wmnet with reason: host reimage 2407384 wikitext text/x-wiki 2tjuklx9sj6hgbaa5oyqkk8fnm7u6zi 2407385 2407384 2026-04-29T12:21:08Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet 2407385 wikitext text/x-wiki c3kp49qe2sbtfu6n33preidybddtdso 2407386 2407385 2026-04-29T12:21:18Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2407386 wikitext text/x-wiki 3d7o7m4nqbjwt48b7kq7r9dmujs0kgt 2407387 2407386 2026-04-29T12:21:24Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2177.codfw.wmnet with reason: host reimage 2407387 wikitext text/x-wiki qq99qe4grumc5cl6i9hjy7f0mhz3bkg 2407388 2407387 2026-04-29T12:21:40Z Stashbot 7414 Emperor: TLS key rollover for ms-fe T424674 2407388 wikitext text/x-wiki 8dyt79ggqndueb053xlcdnack3funju 2407389 2407388 2026-04-29T12:22:46Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2204.codfw.wmnet with reason: host reimage 2407389 wikitext text/x-wiki hzni7bnvlp8d8f0pwe4qc0ci4pvpwgb 2407390 2407389 2026-04-29T12:22:55Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2407390 wikitext text/x-wiki k2s19zgq0mehkzy66h4klldgml49le4 2407391 2407390 2026-04-29T12:24:17Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1223.eqiad.wmnet with reason: host reimage 2407391 wikitext text/x-wiki r0rrcxev2jq3qqcnengvlol51t8ecl2 2407392 2407391 2026-04-29T12:26:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P91914 and previous config saved to /var/cache/conftool/dbconfig/20260429-122641-fceratto.json 2407392 wikitext text/x-wiki p2q3cxc9fq1c7a0ao83u152a1fpnqrg 2407394 2407392 2026-04-29T12:28:32Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2177.codfw.wmnet with reason: host reimage 2407394 wikitext text/x-wiki m1qfzsbwvx5ty58k3dl89z9s9po4ols 2407395 2407394 2026-04-29T12:29:58Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2407395 wikitext text/x-wiki aa2xz6fnjz58lbfivilx7i60ikfgqsk 2407396 2407395 2026-04-29T12:31:03Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet 2407396 wikitext text/x-wiki fztvs3jh12qlhkkymkw3dmu8kbtlkkd 2407397 2407396 2026-04-29T12:32:23Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1197.eqiad.wmnet with reason: host reimage 2407397 wikitext text/x-wiki 1ub6yi39jw1blsgm0yewi9mh3rlbshh 2407398 2407397 2026-04-29T12:34:20Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5005.eqsin.wmnet to cluster eqsin02 and group 01 2407398 wikitext text/x-wiki opd7eoz1raed0jbube08hvu8nfy1n77 2407399 2407398 2026-04-29T12:35:48Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti5005.eqsin.wmnet to cluster eqsin02 and group 01 2407399 wikitext text/x-wiki bnx0625533ihg7hhq9ml3jrnjbvzo3t 2407400 2407399 2026-04-29T12:36:09Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2204.codfw.wmnet with reason: host reimage 2407400 wikitext text/x-wiki hge86979rffvj1qtbc6hc1v03fdxqmk 2407401 2407400 2026-04-29T12:36:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P91915 and previous config saved to /var/cache/conftool/dbconfig/20260429-123648-fceratto.json 2407401 wikitext text/x-wiki 3fipiaxwfkq60e766t49ow9ooiz78ce 2407402 2407401 2026-04-29T12:40:58Z Stashbot 7414 tappof: migrate prometheus5002 to prometheus5003 T424024 2407402 wikitext text/x-wiki ll4ywghjhh5yeebl2nq2ohy2dox9mu3 2407403 2407402 2026-04-29T12:46:41Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1223.eqiad.wmnet with OS trixie 2407403 wikitext text/x-wiki s8ciqg3czkk0somsx4cpkbt806oz271 2407404 2407403 2026-04-29T12:46:59Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T419961)', diff saved to https://phabricator.wikimedia.org/P91916 and previous config saved to /var/cache/conftool/dbconfig/20260429-124656-fceratto.json 2407404 wikitext text/x-wiki bjhkmkovub6npe7taugj27v9g3me5tq 2407405 2407404 2026-04-29T12:47:19Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1236.eqiad.wmnet with reason: Maintenance 2407405 wikitext text/x-wiki i0jlha3qzebcjv4p6gpkmw86bd1s9m4 2407406 2407405 2026-04-29T12:47:28Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1236 (T419961)', diff saved to https://phabricator.wikimedia.org/P91917 and previous config saved to /var/cache/conftool/dbconfig/20260429-124725-fceratto.json 2407406 wikitext text/x-wiki b0njcqtviu7gpzljlyccodis0nz2kiq 2407407 2407406 2026-04-29T12:50:40Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2177.codfw.wmnet with OS trixie 2407407 wikitext text/x-wiki 74gip0cal8rvethlh4v765ylh1tdt0n 2407408 2407407 2026-04-29T12:51:09Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1223: after reimage to trixie 2407408 wikitext text/x-wiki 7aqdab5nx41k8my8f77jh92ll54u2ug 2407409 2407408 2026-04-29T12:53:56Z Stashbot 7414 tappof@dns1004: START - running authdns-update 2407409 wikitext text/x-wiki s93r7ys0suotjfloacz909vduhrlze5 2407410 2407409 2026-04-29T12:54:32Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1197.eqiad.wmnet with OS trixie 2407410 wikitext text/x-wiki i1dzomfjciv226l1e6a0p91udzlr89o 2407411 2407410 2026-04-29T12:55:34Z Stashbot 7414 tappof@dns1004: END - running authdns-update 2407411 wikitext text/x-wiki 28ya44gu87wks2kbx0685pvifjrkem8 2407412 2407411 2026-04-29T12:55:38Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . 2407412 wikitext text/x-wiki go7eyg4d81hanvctbzxb74nt3yfjkuz 2407413 2407412 2026-04-29T12:56:03Z Stashbot 7414 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply 2407413 wikitext text/x-wiki 3p4iquc8a0eg8e6v3hdz0vl2c6zg1os 2407416 2407413 2026-04-29T12:57:29Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2204.codfw.wmnet with OS trixie 2407416 wikitext text/x-wiki 9b1g63zkn6zmgobr5uu9lg5jghqbrfv 2407417 2407416 2026-04-29T12:57:42Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1197: after reimage to trixie 2407417 wikitext text/x-wiki 5lf9quk0b293uvpa7zskfj88zzhra72 2407418 2407417 2026-04-29T12:57:48Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=GrayStorm --performer=GrayStorm --as-job # T418194 2407418 wikitext text/x-wiki nh1hx7n2ozpk0hzsc3s4vjmgzgdq0vl 2407419 2407418 2026-04-29T12:58:13Z Stashbot 7414 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply 2407419 wikitext text/x-wiki oz0x8wmjlnk6ubj6egzkqjp9q5yd9nq 2407420 2407419 2026-04-29T12:58:33Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2177: after reimage to trixie 2407420 wikitext text/x-wiki tu8c35wjaaxkmdgbikahjacia5gny9v 2407421 2407420 2026-04-29T13:00:00Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2204: after reimage to trixie 2407421 wikitext text/x-wiki 3faatk7r1b6nxat4ccagqen7nvf4fzy 2407422 2407421 2026-04-29T13:02:33Z Stashbot 7414 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply 2407422 wikitext text/x-wiki av2ultinv5vsi6qotu1imb45uxt5b3m 2407423 2407422 2026-04-29T13:03:53Z Stashbot 7414 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply 2407423 wikitext text/x-wiki 63ig69pos7grq8qdcodohnkn4mdsl2s 2407424 2407423 2026-04-29T13:10:15Z Stashbot 7414 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply 2407424 wikitext text/x-wiki oz2an52dyuj514jc623l4lzur2dfflz 2407425 2407424 2026-04-29T13:13:39Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2407425 wikitext text/x-wiki szduqsfp1x15d4c91zwplpn3fldzzch 2407426 2407425 2026-04-29T13:15:29Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2407426 wikitext text/x-wiki ibn77jrvy347fvxse29blnl3kidpkem 2407427 2407426 2026-04-29T13:16:06Z Stashbot 7414 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1271215|lbwiki: Limit ContentTranslation extension to autoconfirmed and confirmed users (T423100)]] 2407427 wikitext text/x-wiki m6qpl23ahkwbd1btfkn18qkc56ht3vh 2407428 2407427 2026-04-29T13:16:14Z Stashbot 7414 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply 2407428 wikitext text/x-wiki 6c3k68yqlhhuwmgh5jkyp17844w3fw2 2407429 2407428 2026-04-29T13:17:59Z Stashbot 7414 sbisson@deploy1003: sbisson, codenamenoreste: Backport for [[gerrit:1271215|lbwiki: Limit ContentTranslation extension to autoconfirmed and confirmed users (T423100)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2407429 wikitext text/x-wiki 64envwi24117yekezfw5vpjfrsblgqa 2407431 2407429 2026-04-29T13:21:39Z Stashbot 7414 moritzm: installing tiff security updates 2407431 wikitext text/x-wiki 6p427vyup6e9fic10zx7hvo8mv40q4c 2407432 2407431 2026-04-29T13:22:46Z Stashbot 7414 sbisson@deploy1003: sbisson, codenamenoreste: Continuing with deployment 2407432 wikitext text/x-wiki q0e7f4lu0ml8zm6o59shq0sp0ukpejr 2407434 2407432 2026-04-29T13:26:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1236 (T419961)', diff saved to https://phabricator.wikimedia.org/P91928 and previous config saved to /var/cache/conftool/dbconfig/20260429-132635-fceratto.json 2407434 wikitext text/x-wiki 1z3kpw4u1lut0o75juip7yw42amdh6o 2407435 2407434 2026-04-29T13:26:41Z Stashbot 7414 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1271215|lbwiki: Limit ContentTranslation extension to autoconfirmed and confirmed users (T423100)]] (duration: 10m 33s) 2407435 wikitext text/x-wiki rmwn4lwojwdlmtuhj20y7k21siqg1ub 2407436 2407435 2026-04-29T13:28:28Z Stashbot 7414 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1278584|testwiki: Article Guidance experiment config (T417200)]] 2407436 wikitext text/x-wiki soexjz8rt114zjlanmh7ynacu8gjq4q 2407437 2407436 2026-04-29T13:30:19Z Stashbot 7414 sbisson@deploy1003: sbisson: Backport for [[gerrit:1278584|testwiki: Article Guidance experiment config (T417200)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2407437 wikitext text/x-wiki h4i3lgaguydwk9aedmdyy5bycu4dn4d 2407438 2407437 2026-04-29T13:32:53Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reimage for host cloudelastic1007.eqiad.wmnet with OS trixie 2407438 wikitext text/x-wiki 9kp0u50qzb264itrej88a4t61mzfka4 2407439 2407438 2026-04-29T13:33:31Z Stashbot 7414 sbisson@deploy1003: sbisson: Continuing with deployment 2407439 wikitext text/x-wiki 2lqftsmd99oyw1g2h28ui0engpgal17 2407440 2407439 2026-04-29T13:34:15Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.dns.netbox 2407440 wikitext text/x-wiki ca6l6wjm8bokokm84uwtnalg007y5p0 2407441 2407440 2026-04-29T13:36:33Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1223: after reimage to trixie 2407441 wikitext text/x-wiki 708pzmgtqdm9sxygxk4lc3ww1ykmd4r 2407442 2407441 2026-04-29T13:36:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P91933 and previous config saved to /var/cache/conftool/dbconfig/20260429-133643-fceratto.json 2407442 wikitext text/x-wiki 60n39zj6xikhaipoaem9h5arvp817l2 2407443 2407442 2026-04-29T13:37:20Z Stashbot 7414 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1278584|testwiki: Article Guidance experiment config (T417200)]] (duration: 08m 51s) 2407443 wikitext text/x-wiki pyimxfx2jctuqysjfupdw3cltoxl7z3 2407444 2407443 2026-04-29T13:38:50Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) 2407444 wikitext text/x-wiki cbr3i2uior8u0r3yddr8i0sgffcspv0 2407445 2407444 2026-04-29T13:40:16Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2407445 wikitext text/x-wiki mtopzf2cdc2exejy42vj52zqgyk5xd6 2407446 2407445 2026-04-29T13:42:07Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2407446 wikitext text/x-wiki qx33cttkbvd12hs6nbrvdycuorgcvfr 2407447 2407446 2026-04-29T13:43:08Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1197: after reimage to trixie 2407447 wikitext text/x-wiki ojmtcvdi4qinkmwiv3xd1moc9c0u9cz 2407448 2407447 2026-04-29T13:43:57Z Stashbot 7414 stran@deploy1003: Started scap sync-world: Backport for [[gerrit:1278380|Update action parameter for bulk blocking instrumented events (T420517)]], [[gerrit:1279279|Support staggered rollout via Test Kitchen (T424220)]], [[gerrit:1279280|Update IRS instrumentation (T424075)]], [[gerrit:1279335|Instrument link clicks on success pages per spec (T424075)]], [[gerrit:1279269|Enable staggered rollout for IRS on testwiki (T 2407448 wikitext text/x-wiki 495xfbvpyqbs04bhtqlivshrwqc9ng4 2407449 2407448 2026-04-29T13:44:01Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2177: after reimage to trixie 2407449 wikitext text/x-wiki rs8ewd79eqa42wtro37jftoboc7flw4 2407450 2407449 2026-04-29T13:45:26Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2204: after reimage to trixie 2407450 wikitext text/x-wiki atshvoa1vip1ryrcdvstxr01syawq18 2407451 2407450 2026-04-29T13:45:47Z Stashbot 7414 stran@deploy1003: stran: Backport for [[gerrit:1278380|Update action parameter for bulk blocking instrumented events (T420517)]], [[gerrit:1279279|Support staggered rollout via Test Kitchen (T424220)]], [[gerrit:1279280|Update IRS instrumentation (T424075)]], [[gerrit:1279335|Instrument link clicks on success pages per spec (T424075)]], [[gerrit:1279269|Enable staggered rollout for IRS on testwiki (T424075)]] synced t 2407451 wikitext text/x-wiki m6j2ydwv8pjg8015o1k4cognxj469g1 2407453 2407451 2026-04-29T13:46:54Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P91937 and previous config saved to /var/cache/conftool/dbconfig/20260429-134651-fceratto.json 2407453 wikitext text/x-wiki j3twvsxt9e90jfclm1skj9dckm5yqae 2407454 2407453 2026-04-29T13:47:27Z Stashbot 7414 stran@deploy1003: stran: Continuing with deployment 2407454 wikitext text/x-wiki 8t4kvee81589bkw5vpirhh45lak0g17 2407455 2407454 2026-04-29T13:47:51Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1157.eqiad.wmnet with reason: Reimage to Trixie 2407455 wikitext text/x-wiki c26p4qor1um861fpiq0tgjyryl1vr2a 2407456 2407455 2026-04-29T13:47:54Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2156.codfw.wmnet with reason: Reimage to Trixie 2407456 wikitext text/x-wiki 4j9bs9oovasa0xy0s1046w26g72w1o7 2407457 2407456 2026-04-29T13:47:56Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1157: Reimage to Trixie 2407457 wikitext text/x-wiki 4997e78f50agssw5q5r0hzt1ixdiut5 2407458 2407457 2026-04-29T13:47:59Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2156: Reimage to Trixie 2407458 wikitext text/x-wiki swof8ams3dd2bwd44pknkae3fjnznob 2407459 2407458 2026-04-29T13:48:15Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2156: Reimage to Trixie 2407459 wikitext text/x-wiki 2tne019pp1s6qx7g4u1gbc2zo3b4eo7 2407460 2407459 2026-04-29T13:48:23Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1157: Reimage to Trixie 2407460 wikitext text/x-wiki 6udpe5773n1n8owofinbxfqsvo5l7k7 2407462 2407460 2026-04-29T13:49:56Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2156.codfw.wmnet with OS trixie 2407462 wikitext text/x-wiki l2xkne3v4baztsje8ybf244pupggjo2 2407464 2407462 2026-04-29T13:50:11Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1157.eqiad.wmnet with OS trixie 2407464 wikitext text/x-wiki n4dk4svho16r65gaybyw4vctymm6wjq 2407466 2407464 2026-04-29T13:51:27Z Stashbot 7414 stran@deploy1003: Finished scap sync-world: Backport for [[gerrit:1278380|Update action parameter for bulk blocking instrumented events (T420517)]], [[gerrit:1279279|Support staggered rollout via Test Kitchen (T424220)]], [[gerrit:1279280|Update IRS instrumentation (T424075)]], [[gerrit:1279335|Instrument link clicks on success pages per spec (T424075)]], [[gerrit:1279269|Enable staggered rollout for IRS on testwiki ( 2407466 wikitext text/x-wiki rgd0ft1f2pymh65vj7smtt08vhu28s4 2407467 2407466 2026-04-29T13:52:24Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudelastic1007.eqiad.wmnet with reason: host reimage 2407467 wikitext text/x-wiki 513cce69588n30yevz92t1hxw57o5xg 2407468 2407467 2026-04-29T13:53:46Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.hosts.decommission for hosts prometheus5002.eqsin.wmnet 2407468 wikitext text/x-wiki 64g28royvozh6cvtniowj4h1671j7wj 2407469 2407468 2026-04-29T13:54:26Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.dns.netbox 2407469 wikitext text/x-wiki irj3rt4nyf535div74f32nrlaij6b3o 2407471 2407469 2026-04-29T13:55:40Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2407471 wikitext text/x-wiki j63kj123vrh9mglkh5du4hahe6m78ni 2407472 2407471 2026-04-29T13:56:27Z Stashbot 7414 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1279330|CentralAuthTokenSessionProvider: Add security context to "centralauthtoken is invalid" logs]], [[gerrit:1279331|CentralAuthTokenSessionProvider: Add security context to "centralauthtoken is invalid" logs]] 2407472 wikitext text/x-wiki 3a7ieo336tozi26d10bhs9pyhjbs9ga 2407473 2407472 2026-04-29T13:56:47Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) 2407473 wikitext text/x-wiki 8o3o9em052vf3f2ajfng0kpz217tqqu 2407474 2407473 2026-04-29T13:57:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1236 (T419961)', diff saved to https://phabricator.wikimedia.org/P91940 and previous config saved to /var/cache/conftool/dbconfig/20260429-135659-fceratto.json 2407474 wikitext text/x-wiki 9fji4mmdnff5xmq2p0wvcztuh9rsvgn 2407475 2407474 2026-04-29T13:57:05Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2407475 wikitext text/x-wiki s3qe92sp3vhakjbv54b90vjiczen0me 2407476 2407475 2026-04-29T13:58:15Z Stashbot 7414 sbisson@deploy1003: matmarex, sbisson: Backport for [[gerrit:1279330|CentralAuthTokenSessionProvider: Add security context to "centralauthtoken is invalid" logs]], [[gerrit:1279331|CentralAuthTokenSessionProvider: Add security context to "centralauthtoken is invalid" logs]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2407476 wikitext text/x-wiki 6feqxaub53gv1h8chsczzlbjpaz3l6v 2407477 2407476 2026-04-29T13:58:39Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.dns.netbox 2407477 wikitext text/x-wiki j51ceic20n9mo9o5pvv0gjrj1an7a5g 2407478 2407477 2026-04-29T13:59:27Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudelastic1007.eqiad.wmnet with reason: host reimage 2407478 wikitext text/x-wiki 0ae3fgqt07hioae73ps0psrb8686edx 2407479 2407478 2026-04-29T14:00:44Z Stashbot 7414 sbisson@deploy1003: matmarex, sbisson: Continuing with deployment 2407479 wikitext text/x-wiki 6l4a7gato00pdjfqe9m8uw2c15ptmyf 2407480 2407479 2026-04-29T14:03:20Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1157.eqiad.wmnet with reason: host reimage 2407480 wikitext text/x-wiki 7twgg4fy9ammeacee2emif21kk2t8e3 2407481 2407480 2026-04-29T14:03:36Z Stashbot 7414 tappof@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: prometheus5002.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1003" 2407481 wikitext text/x-wiki rnuziubbbhl4s8c4nv2wg28evrwezvk 2407482 2407481 2026-04-29T14:04:10Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.dns.netbox 2407482 wikitext text/x-wiki 6gxmlin6fvjgtqqzac4vory8cl7jufd 2407483 2407482 2026-04-29T14:04:35Z Stashbot 7414 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1279330|CentralAuthTokenSessionProvider: Add security context to "centralauthtoken is invalid" logs]], [[gerrit:1279331|CentralAuthTokenSessionProvider: Add security context to "centralauthtoken is invalid" logs]] (duration: 08m 08s) 2407483 wikitext text/x-wiki kaosla1ojho4dtt5pit7257v9wpc4ie 2407484 2407483 2026-04-29T14:06:12Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2407484 wikitext text/x-wiki ccq2ye0tecjwwekfe0k75uum4zve0f9 2407485 2407484 2026-04-29T14:06:17Z Stashbot 7414 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2407485 wikitext text/x-wiki rj2janvrsx577xl6yadobizfyzsuuie 2407486 2407485 2026-04-29T14:06:46Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2407486 wikitext text/x-wiki 382u7xexo2chsyjtusql47eb8ejt65u 2407487 2407486 2026-04-29T14:08:41Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1157.eqiad.wmnet with reason: host reimage 2407487 wikitext text/x-wiki 6atebnraan24h9jekqavpybgxuev7gx 2407488 2407487 2026-04-29T14:09:11Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2156.codfw.wmnet with reason: host reimage 2407488 wikitext text/x-wiki 0zheomxi6vr95h2g80d6um1ywfpejt1 2407489 2407488 2026-04-29T14:13:11Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2156.codfw.wmnet with reason: host reimage 2407489 wikitext text/x-wiki crenbghkft4fz16frvxw0vdciq7ga6b 2407490 2407489 2026-04-29T14:16:58Z Stashbot 7414 elukey: uploaded spicerack_12.5.0 to apt.wikimedia.org bookworm-wikimedia 2407490 wikitext text/x-wiki jj7vq63nurlqxg2po1luewep7r4iw0w 2407491 2407490 2026-04-29T14:18:44Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1375 2407491 wikitext text/x-wiki se29so6qs84dy0emw5v2vln8rybzqrq 2407492 2407491 2026-04-29T14:19:26Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2407492 wikitext text/x-wiki 2j1h45c9rlui5m3mxhofltqma45c7co 2407493 2407492 2026-04-29T14:19:33Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2407493 wikitext text/x-wiki huz2u9tqdn75rb4q850boi6556t5qkl 2407494 2407493 2026-04-29T14:19:57Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1375 2407494 wikitext text/x-wiki 1w8nfk1vpui7hiy7b28m3dbbwljl4ys 2407495 2407494 2026-04-29T14:20:59Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2152.codfw.wmnet with reason: Maintenance 2407495 wikitext text/x-wiki 4uiubqo0odh6y0jbczg7asw2smqk5ca 2407496 2407495 2026-04-29T14:21:06Z Stashbot 7414 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2407496 wikitext text/x-wiki 8qt6tse239u289tpkmvq968k3lm5lyk 2407497 2407496 2026-04-29T14:21:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2152 (T419961)', diff saved to https://phabricator.wikimedia.org/P91941 and previous config saved to /var/cache/conftool/dbconfig/20260429-142105-fceratto.json 2407497 wikitext text/x-wiki q9jwobfxhpuaccf8d3n1l5lkjbnebmm 2407498 2407497 2026-04-29T14:21:12Z Stashbot 7414 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2407498 wikitext text/x-wiki hnkjd7gqhc8mhf5hrhh33c4q6gsn0rw 2407499 2407498 2026-04-29T14:21:23Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: prometheus5002.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1003" 2407499 wikitext text/x-wiki 9k5c8f90dnzxdbqlas00ychrxmctmsd 2407500 2407499 2026-04-29T14:21:26Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2407500 wikitext text/x-wiki 2wu6lf8gwaunvup8o3x6gnshmk6vxkt 2407501 2407500 2026-04-29T14:21:29Z Stashbot 7414 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts prometheus5002.eqsin.wmnet 2407501 wikitext text/x-wiki p6gn23d1l184li7uedydfm8y6l41dao 2407502 2407501 2026-04-29T14:21:47Z Stashbot 7414 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2407502 wikitext text/x-wiki 8r7cz8fi9uuci23hxc0jke194ad5ow9 2407504 2407502 2026-04-29T14:23:09Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudelastic1007.eqiad.wmnet with OS trixie 2407504 wikitext text/x-wiki 1rowpqn8huzpkowntije8cwgh6ti351 2407505 2407504 2026-04-29T14:23:20Z Stashbot 7414 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2407505 wikitext text/x-wiki lqzuvv96mk9osw2241oqxy6q5yfzxhl 2407506 2407505 2026-04-29T14:24:46Z Stashbot 7414 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2407506 wikitext text/x-wiki kguyqchm69namch7d313vhf4b7hqopu 2407507 2407506 2026-04-29T14:25:27Z Stashbot 7414 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2407507 wikitext text/x-wiki r1rzfrii5aoswtvnltmm4ydgv19fpyb 2407508 2407507 2026-04-29T14:25:55Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . 2407508 wikitext text/x-wiki b0nh0e8d4x7z0xnjwfcxn7zpte23it7 2407509 2407508 2026-04-29T14:29:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T419961)', diff saved to https://phabricator.wikimedia.org/P91942 and previous config saved to /var/cache/conftool/dbconfig/20260429-142916-fceratto.json 2407509 wikitext text/x-wiki gmn13hm3kjbvhrk9gxs0g33gy3t46gj 2407510 2407509 2026-04-29T14:29:59Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1157.eqiad.wmnet with OS trixie 2407510 wikitext text/x-wiki 077jlith2bt4ot5431i1x7vdp7oj0o2 2407511 2407510 2026-04-29T14:32:09Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2407511 wikitext text/x-wiki 699n5wts6g7lit8g4s3eccwgzkzr6sf 2407512 2407511 2026-04-29T14:33:58Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2407512 wikitext text/x-wiki 4yr2tj2pshipi6s8xvkg6wetpclm1l4 2407513 2407512 2026-04-29T14:34:43Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1157: after reimage to trixie 2407513 wikitext text/x-wiki du359uxp7zijlkzalytmykjyt0biic6 2407514 2407513 2026-04-29T14:34:45Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2407514 wikitext text/x-wiki pjaf2501va0dvn8opw36vmtt8kt1ucq 2407515 2407514 2026-04-29T14:35:23Z Stashbot 7414 moritzm: installing zsh updates from Trixie point release 2407515 wikitext text/x-wiki 9qnd6m8wftpnzmjh0ywixjh7ezauw6g 2407516 2407515 2026-04-29T14:35:45Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2407516 wikitext text/x-wiki o4yraxu1c4lpcye8fe7tprkyr9qu242 2407517 2407516 2026-04-29T14:36:10Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2407517 wikitext text/x-wiki i8l2nzdvrr87x1a2sdwrhjs65o87nq0 2407518 2407517 2026-04-29T14:36:31Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2407518 wikitext text/x-wiki ejs7y8mafuywpk7zh5tb4v2wjeobfdw 2407519 2407518 2026-04-29T14:36:34Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2407519 wikitext text/x-wiki hleiir9cqnhq2p9hf0gb8mwf2mwy9pa 2407520 2407519 2026-04-29T14:37:02Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2407520 wikitext text/x-wiki tip5dzy5ta15wztgztg7cnsy4jemkt7 2407521 2407520 2026-04-29T14:37:08Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2156.codfw.wmnet with OS trixie 2407521 wikitext text/x-wiki 5wuw33df2ufb58g5le0ts2k5fq75in9 2407522 2407521 2026-04-29T14:37:12Z Stashbot 7414 inflatador: bking@cloudelastic1010 run smartctl against all physical disks T424852 2407522 wikitext text/x-wiki 7gge55fionv9w4slhnxnl2vwmrclkgz 2407523 2407522 2026-04-29T14:37:22Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2407523 wikitext text/x-wiki 9xlt7ov0i6cw5cxlw21bgkkela60g5a 2407524 2407523 2026-04-29T14:37:51Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2407524 wikitext text/x-wiki kp8290v86aw98qp8ht8mmlo99o1j28z 2407525 2407524 2026-04-29T14:39:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P91944 and previous config saved to /var/cache/conftool/dbconfig/20260429-143924-fceratto.json 2407525 wikitext text/x-wiki cluesx98wgu3zox2siy2tncl9lf8sem 2407526 2407525 2026-04-29T14:40:12Z Stashbot 7414 bking@cumin2002: conftool action : set/weight=10; selector: name=cloudelastic1007. 2407526 wikitext text/x-wiki 2ll6mrxble0p3tn7q1sxm167k6w6nnq 2407527 2407526 2026-04-29T14:40:14Z Stashbot 7414 mstyles@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply 2407527 wikitext text/x-wiki 0hgswqsuwvwf7vvprtnnzwimfq2hvvd 2407528 2407527 2026-04-29T14:40:22Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2407528 wikitext text/x-wiki q0orybp6s1tvxag27c0gfiae6p7g2wz 2407529 2407528 2026-04-29T14:40:30Z Stashbot 7414 mstyles@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply 2407529 wikitext text/x-wiki lo6yyoa9mzahuj5dozqz166ghzdcv2l 2407530 2407529 2026-04-29T14:40:38Z Stashbot 7414 bking@cumin2002: conftool action : set/pooled/yes; selector: dc=eqiad,cluster=cloudelastic,name=cloudelastic1007. 2407530 wikitext text/x-wiki bbnue6s5ugldebgcc0vqkx8fvxnknic 2407531 2407530 2026-04-29T14:40:46Z Stashbot 7414 mstyles@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply 2407531 wikitext text/x-wiki kunr06q5zi46frg3mtcbhlg0mzle3e3 2407532 2407531 2026-04-29T14:40:55Z Stashbot 7414 mstyles@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply 2407532 wikitext text/x-wiki 185l0q9skc44vvnp4wyizm14ke41y7h 2407533 2407532 2026-04-29T14:41:06Z Stashbot 7414 mstyles@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply 2407533 wikitext text/x-wiki i0p823m255l0njhrpzlw8cuvcyn58jk 2407534 2407533 2026-04-29T14:41:12Z Stashbot 7414 mstyles@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply 2407534 wikitext text/x-wiki 7ey49loqo2p3osgosyqc9tad6pe7uy8 2407535 2407534 2026-04-29T14:41:24Z Stashbot 7414 mstyles@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply 2407535 wikitext text/x-wiki nu36idpinfjuaqz2hsvmign7gl9q5n7 2407536 2407535 2026-04-29T14:41:31Z Stashbot 7414 mstyles@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply 2407536 wikitext text/x-wiki 2taulpftyx59gcfay2o34v684aolp1x 2407537 2407536 2026-04-29T14:41:35Z Stashbot 7414 mstyles@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply 2407537 wikitext text/x-wiki gzkv40ajxzoilhdwgb23wdpcf4r1q60 2407538 2407537 2026-04-29T14:41:39Z Stashbot 7414 mstyles@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply 2407538 wikitext text/x-wiki 15p7q7338lh36epxgr9a6prdm2hc03v 2407539 2407538 2026-04-29T14:41:55Z Stashbot 7414 bking@cumin2002: conftool action : set/pooled=yes; selector: name=cloudelastic1007.eqiad.wmnet 2407539 wikitext text/x-wiki 0v04zvc1tkfiwbg516vqyupczk88q5i 2407540 2407539 2026-04-29T14:42:09Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: sync 2407540 wikitext text/x-wiki e9cam2vzko4dlgk7u801cw64ilyuhip 2407541 2407540 2026-04-29T14:42:32Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: sync 2407541 wikitext text/x-wiki eas888xs76ae2bt94odm6f7wdygvse3 2407542 2407541 2026-04-29T14:43:39Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2156: after reimage to trixie 2407542 wikitext text/x-wiki t3g2qntvh025jio6ireispa26t9ux09 2407543 2407542 2026-04-29T14:46:45Z Stashbot 7414 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. 2407543 wikitext text/x-wiki 6ccv7vwga3gi0flsccpsd25u64ufmtf 2407544 2407543 2026-04-29T14:46:52Z Stashbot 7414 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. 2407544 wikitext text/x-wiki 1ps6ljeu9dajyhb5bztyqogkbnh1ln0 2407545 2407544 2026-04-29T14:47:03Z Stashbot 7414 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. 2407545 wikitext text/x-wiki ir51s34rv2ey209tdiwsmn51k98yxav 2407546 2407545 2026-04-29T14:47:19Z Stashbot 7414 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. 2407546 wikitext text/x-wiki qk0k17td7qrwyurqh2xfa86f4nmyrmn 2407547 2407546 2026-04-29T14:49:34Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P91947 and previous config saved to /var/cache/conftool/dbconfig/20260429-144932-fceratto.json 2407547 wikitext text/x-wiki 6l60ruap3mjhsyf60guatze9n38o7kk 2407548 2407547 2026-04-29T14:50:06Z Stashbot 7414 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. 2407548 wikitext text/x-wiki 9og5t5x2wp5tcg4h5hsh28tarsc83mz 2407549 2407548 2026-04-29T14:50:11Z Stashbot 7414 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. 2407549 wikitext text/x-wiki nukse8v8c74fy2qbglg7wyyzndt4i2v 2407550 2407549 2026-04-29T14:50:45Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2407550 wikitext text/x-wiki 8ouv0b87g9hgox7czgh22rqz1oig1oo 2407551 2407550 2026-04-29T14:50:48Z Stashbot 7414 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. 2407551 wikitext text/x-wiki 4v5rq3av234yecigf2cmchnry5qn8si 2407552 2407551 2026-04-29T14:51:00Z Stashbot 7414 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. 2407552 wikitext text/x-wiki 5sax3rctjt8cjkfeefybqdjiymw8pw9 2407553 2407552 2026-04-29T14:51:03Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2407553 wikitext text/x-wiki jc9cshq3sdxdub0vk65yhfei7fshiwe 2407554 2407553 2026-04-29T14:52:07Z Stashbot 7414 mstyles@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply 2407554 wikitext text/x-wiki 5avvqwszgseadte7tvezqytgkg3v2ro 2407555 2407554 2026-04-29T14:52:30Z Stashbot 7414 mstyles@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply 2407555 wikitext text/x-wiki 1rwhuj12w4vleje9xmvkjix2s92xst9 2407556 2407555 2026-04-29T14:52:35Z Stashbot 7414 mstyles@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply 2407556 wikitext text/x-wiki qo9a6816kmgoewq2on9rrevsgngq47c 2407557 2407556 2026-04-29T14:52:54Z Stashbot 7414 mstyles@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply 2407557 wikitext text/x-wiki q841ce70hk18s1tcf1jt7egy9y3p4bl 2407558 2407557 2026-04-29T14:52:59Z Stashbot 7414 mstyles@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply 2407558 wikitext text/x-wiki d74mkqm57cucgv8iwjl7yyvzkkvipwu 2407559 2407558 2026-04-29T14:53:21Z Stashbot 7414 mstyles@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply 2407559 wikitext text/x-wiki tf4nk0zv86k2evkhv01ni6ut7tsspuc 2407560 2407559 2026-04-29T14:53:26Z Stashbot 7414 mstyles@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply 2407560 wikitext text/x-wiki t0trfujb7y3ql7mjr8vsad90xxldvnn 2407561 2407560 2026-04-29T14:53:34Z Stashbot 7414 mstyles@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply 2407561 wikitext text/x-wiki eaan0i5mvd8ayhhxud12vw2tovngcb6 2407562 2407561 2026-04-29T14:53:43Z Stashbot 7414 mstyles@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply 2407562 wikitext text/x-wiki 53xcd3o4u4u34q32smvad00itgjt5o2 2407563 2407562 2026-04-29T14:53:47Z Stashbot 7414 mstyles@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply 2407563 wikitext text/x-wiki kfuozim08man54ayki0msr84rx4wdex 2407564 2407563 2026-04-29T14:55:13Z Stashbot 7414 elukey@dns1004: START - running authdns-update 2407564 wikitext text/x-wiki tupes8upnacfnaqa7rlwok15xvxv6h2 2407565 2407564 2026-04-29T14:56:51Z Stashbot 7414 elukey@dns1004: END - running authdns-update 2407565 wikitext text/x-wiki 6yu8lm0s8n6wiwgozcsa4wl6410hpo4 2407566 2407565 2026-04-29T14:59:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T419961)', diff saved to https://phabricator.wikimedia.org/P91950 and previous config saved to /var/cache/conftool/dbconfig/20260429-145940-fceratto.json 2407566 wikitext text/x-wiki 8lpybza7yw7ghdqpaxgcxsc70li1i2x 2407567 2407566 2026-04-29T15:00:04Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance 2407567 wikitext text/x-wiki mpghiskucjf74zdn3kziq5ub1khvhb7 2407568 2407567 2026-04-29T15:00:12Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2154 (T419961)', diff saved to https://phabricator.wikimedia.org/P91951 and previous config saved to /var/cache/conftool/dbconfig/20260429-150010-fceratto.json 2407568 wikitext text/x-wiki 92utv8pet5bgymqwlofaq0632z37cyw 2407570 2407568 2026-04-29T15:00:57Z Stashbot 7414 elukey@dns1004: START - running authdns-update 2407570 wikitext text/x-wiki f4lvd7lr2rsooiqdreoxs6fn3jzzfra 2407571 2407570 2026-04-29T15:02:28Z Stashbot 7414 elukey@dns1004: END - running authdns-update 2407571 wikitext text/x-wiki 3ovbymjq8dckegtr2m9gzv64gytatuc 2407572 2407571 2026-04-29T15:05:33Z Stashbot 7414 eevans@cumin1003: DONE (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for cassandra-dev2001.codfw.wmnet: Renew puppet certificate - eevans@cumin1003 2407572 wikitext text/x-wiki f5vvnt0v76vrcu6n75w474lopgoc8p2 2407573 2407572 2026-04-29T15:07:12Z Stashbot 7414 eevans@cumin1003: DONE (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for cassandra-dev2001.codfw.wmnet: Renew puppet certificate - eevans@cumin1003 2407573 wikitext text/x-wiki oqx4n98mjsdswm0v0wfq16nocpeaoqd 2407574 2407573 2026-04-29T15:07:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T419961)', diff saved to https://phabricator.wikimedia.org/P91953 and previous config saved to /var/cache/conftool/dbconfig/20260429-150719-fceratto.json 2407574 wikitext text/x-wiki b4e32v0aeasoqzarjd49vlf3h3w7ugq 2407575 2407574 2026-04-29T15:09:38Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache wikifunctions-javascript-evaluator.discovery.wmnet on all recursors 2407575 wikitext text/x-wiki dq7o71yo4ryd32tg3i1vnua6nqj13gn 2407576 2407575 2026-04-29T15:09:42Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikifunctions-javascript-evaluator.discovery.wmnet on all recursors 2407576 wikitext text/x-wiki sp40odfk93baowkjsci6w8qxqd68v10 2407577 2407576 2026-04-29T15:09:50Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.dns.wipe-cache wikifunctions-python-evaluator.discovery.wmnet on all recursors 2407577 wikitext text/x-wiki g4o5zf2ydyvh4c36rir96y4z3g6as9y 2407578 2407577 2026-04-29T15:09:54Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikifunctions-python-evaluator.discovery.wmnet on all recursors 2407578 wikitext text/x-wiki axxsunerouf0cayr2figcb8sj1wn2dn 2407580 2407578 2026-04-29T15:11:48Z Stashbot 7414 eevans@cumin1003: DONE (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for cassandra-dev2001.codfw.wmnet: Renew puppet certificate - eevans@cumin1003 2407580 wikitext text/x-wiki dsucwoyrcb7a8yvp40jx2mv2jaw0tew 2407581 2407580 2026-04-29T15:13:27Z Stashbot 7414 eevans@cumin1003: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for cassandra-dev2001.codfw.wmnet: Renew puppet certificate - eevans@cumin1003 2407581 wikitext text/x-wiki nrfulmjdyrcx48u2au8tg5fn8d3dsr6 2407583 2407581 2026-04-29T15:15:33Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2407583 wikitext text/x-wiki ol7zrnbmfw5u19dffifvs79pu5yog57 2407584 2407583 2026-04-29T15:15:47Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2407584 wikitext text/x-wiki r9zu4rfbn2uwshqxkor0se2uapl23ox 2407585 2407584 2026-04-29T15:17:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P91955 and previous config saved to /var/cache/conftool/dbconfig/20260429-151727-fceratto.json 2407585 wikitext text/x-wiki 0z7fx3sjbk837px3vv1lxpe351egf9v 2407587 2407585 2026-04-29T15:20:06Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1157: after reimage to trixie 2407587 wikitext text/x-wiki 9v2d07bnfmhlu3zk1p0gixixb7x9bao 2407588 2407587 2026-04-29T15:24:56Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be2066 2407588 wikitext text/x-wiki hs2yx2guqlggs2injemn9qvmv31u1af 2407589 2407588 2026-04-29T15:27:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P91957 and previous config saved to /var/cache/conftool/dbconfig/20260429-152735-fceratto.json 2407589 wikitext text/x-wiki ploziqy6z3nffacdl29fjwlvylgy6mi 2407591 2407589 2026-04-29T15:29:05Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2156: after reimage to trixie 2407591 wikitext text/x-wiki 9m1w1gx6e0nznpwx6vep6ei8cgv5ooi 2407593 2407591 2026-04-29T15:37:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T419961)', diff saved to https://phabricator.wikimedia.org/P91960 and previous config saved to /var/cache/conftool/dbconfig/20260429-153743-fceratto.json 2407593 wikitext text/x-wiki rzv1yvldadzqnjx4tj5ot8sqe3jmakv 2407594 2407593 2026-04-29T15:38:08Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2161.codfw.wmnet with reason: Maintenance 2407594 wikitext text/x-wiki bc63eqiag6usnxj8k5o1qrjyrt96ijr 2407595 2407594 2026-04-29T15:38:16Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2161 (T419961)', diff saved to https://phabricator.wikimedia.org/P91961 and previous config saved to /var/cache/conftool/dbconfig/20260429-153814-fceratto.json 2407595 wikitext text/x-wiki kadtmfuurhgm8c41ynpgeih1v446otb 2407596 2407595 2026-04-29T15:40:41Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be2067 2407596 wikitext text/x-wiki 6ejavq1xl6htv5ciwtbptws5eg2mp1e 2407597 2407596 2026-04-29T15:45:27Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2161 (T419961)', diff saved to https://phabricator.wikimedia.org/P91962 and previous config saved to /var/cache/conftool/dbconfig/20260429-154525-fceratto.json 2407597 wikitext text/x-wiki 25xxjl5a6dmg77d53pljc2rook9xmu7 2407600 2407597 2026-04-29T15:55:35Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P91963 and previous config saved to /var/cache/conftool/dbconfig/20260429-155533-fceratto.json 2407600 wikitext text/x-wiki dvm66lo6pyfkw7x45cgp9uchz6kkpg1 2407602 2407600 2026-04-29T16:05:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P91964 and previous config saved to /var/cache/conftool/dbconfig/20260429-160541-fceratto.json 2407602 wikitext text/x-wiki 7h78vyfnpp4q6niv6ednnmyrjhby78o 2407605 2407602 2026-04-29T16:15:21Z Stashbot 7414 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be2066 2407605 wikitext text/x-wiki 8sw9xbt412mohz444qkqzz6ljuo3y77 2407606 2407605 2026-04-29T16:15:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2161 (T419961)', diff saved to https://phabricator.wikimedia.org/P91965 and previous config saved to /var/cache/conftool/dbconfig/20260429-161549-fceratto.json 2407606 wikitext text/x-wiki cqv4czcbafwhm40ifmymuverx0i6n3b 2407607 2407606 2026-04-29T16:15:57Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2066.codfw.wmnet with OS bullseye 2407607 wikitext text/x-wiki ijavbaq3bicnpl0z1h5rqjlbteteom2 2407608 2407607 2026-04-29T16:16:13Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance 2407608 wikitext text/x-wiki o28rlogsucume89nh0r435978d9bbz6 2407609 2407608 2026-04-29T16:16:17Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2066 2407609 wikitext text/x-wiki g5htmg2hf5h3nh2cnxljce4e688dkj9 2407610 2407609 2026-04-29T16:16:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2163 (T419961)', diff saved to https://phabricator.wikimedia.org/P91966 and previous config saved to /var/cache/conftool/dbconfig/20260429-161619-fceratto.json 2407610 wikitext text/x-wiki 3u2h9jqc916vtzsu6djonnq2iph6bxb 2407611 2407610 2026-04-29T16:16:26Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.netbox 2407611 wikitext text/x-wiki ob841p2m6kg52fdq36hv92vetujqnz5 2407613 2407611 2026-04-29T16:21:36Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2066 - mvernon@cumin2002" 2407613 wikitext text/x-wiki cmrc40e5rajqkpma4n89px684hnjrkb 2407615 2407613 2026-04-29T16:21:42Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2066 - mvernon@cumin2002" 2407615 wikitext text/x-wiki bu5qb8xr0li5qi866z24wjrzyico55w 2407616 2407615 2026-04-29T16:21:44Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2407616 wikitext text/x-wiki 11dhdd216ttfa880dku696xxs7jpm6g 2407617 2407616 2026-04-29T16:21:47Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2066.codfw.wmnet 209.0.192.10.in-addr.arpa 9.0.2.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2407617 wikitext text/x-wiki fspwlnnpefgas1ruw846z6gmurqa55t 2407618 2407617 2026-04-29T16:21:49Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2066.codfw.wmnet 209.0.192.10.in-addr.arpa 9.0.2.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2407618 wikitext text/x-wiki h712lh4qj0lgcirobxvtmcar82ye4rr 2407619 2407618 2026-04-29T16:21:52Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2066 2407619 wikitext text/x-wiki t8gpu0nton1s47c85udv5esgpd08hdz 2407620 2407619 2026-04-29T16:22:00Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2066 2407620 wikitext text/x-wiki 7m197w6iw7ok5913gaq4dzzoduqa52t 2407621 2407620 2026-04-29T16:22:03Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2066 2407621 wikitext text/x-wiki 7qv7wwk63v03vd6muo292lktt6cw2z8 2407622 2407621 2026-04-29T16:23:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T419961)', diff saved to https://phabricator.wikimedia.org/P91967 and previous config saved to /var/cache/conftool/dbconfig/20260429-162337-fceratto.json 2407622 wikitext text/x-wiki ifksvjm3q5icl86ue33fchlgtg09sb6 2407623 2407622 2026-04-29T16:33:47Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P91968 and previous config saved to /var/cache/conftool/dbconfig/20260429-163345-fceratto.json 2407623 wikitext text/x-wiki rg5w81twbwibvo44m1lbxw026syyk0e 2407625 2407623 2026-04-29T16:38:08Z Stashbot 7414 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be2067 2407625 wikitext text/x-wiki leqgtq6leth6rz6i0o0rw04wby2kuhh 2407626 2407625 2026-04-29T16:38:33Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2067.codfw.wmnet with OS bullseye 2407626 wikitext text/x-wiki jfusxtskwf6ypg1vamr831ocepevu4c 2407627 2407626 2026-04-29T16:38:58Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2067 2407627 wikitext text/x-wiki 5ahcssptaunv72qt86ml4aiwbky1s9d 2407628 2407627 2026-04-29T16:39:12Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.netbox 2407628 wikitext text/x-wiki 7sgsuvhw99q58lsvm9v4caxngj7i1fw 2407629 2407628 2026-04-29T16:40:16Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2066.codfw.wmnet with reason: host reimage 2407629 wikitext text/x-wiki aj0uwh89n6ty1axygg5uxbainfqk1xr 2407631 2407629 2026-04-29T16:43:09Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2067 - mvernon@cumin2002" 2407631 wikitext text/x-wiki berlzob310xuf2354c3opycvtz29s6b 2407632 2407631 2026-04-29T16:43:14Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2067 - mvernon@cumin2002" 2407632 wikitext text/x-wiki qe32v8uagy4z2zbknj4i7l3syqztp8p 2407633 2407632 2026-04-29T16:43:17Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2407633 wikitext text/x-wiki 09bglzen6c2krogpudv8oj7euzufv9p 2407634 2407633 2026-04-29T16:43:19Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2067.codfw.wmnet 160.16.192.10.in-addr.arpa 0.6.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2407634 wikitext text/x-wiki sl2zf0m109ijwnz9fc6nsvycxqew6um 2407635 2407634 2026-04-29T16:43:22Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2067.codfw.wmnet 160.16.192.10.in-addr.arpa 0.6.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2407635 wikitext text/x-wiki nc87f7to887hfm3e4lspcjlcp3rmh41 2407636 2407635 2026-04-29T16:43:24Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2067 2407636 wikitext text/x-wiki 0puxajlzp4ie2xmmcp4d93qy25obdbe 2407637 2407636 2026-04-29T16:43:42Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2067 2407637 wikitext text/x-wiki juqyq78rxderfwaaiy5r5rdi4depaxj 2407638 2407637 2026-04-29T16:43:44Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2067 2407638 wikitext text/x-wiki agrjehvdv7r3cfystk1tx8fu5wnycn5 2407639 2407638 2026-04-29T16:43:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P91969 and previous config saved to /var/cache/conftool/dbconfig/20260429-164353-fceratto.json 2407639 wikitext text/x-wiki 97744wuio1vv9nmg7t3kef113arjk4o 2407641 2407639 2026-04-29T16:44:30Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2066.codfw.wmnet with reason: host reimage 2407641 wikitext text/x-wiki h6l9u2hfxvikv73213oq9ue8kpi2x83 2407642 2407641 2026-04-29T16:54:04Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T419961)', diff saved to https://phabricator.wikimedia.org/P91970 and previous config saved to /var/cache/conftool/dbconfig/20260429-165401-fceratto.json 2407642 wikitext text/x-wiki r7klglzy6qofupoa1v6yinyb8ia7po1 2407643 2407642 2026-04-29T16:54:25Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance 2407643 wikitext text/x-wiki bboy8i2bpsocwxlkdcq1iv8698fgkjk 2407644 2407643 2026-04-29T16:54:34Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2164 (T419961)', diff saved to https://phabricator.wikimedia.org/P91971 and previous config saved to /var/cache/conftool/dbconfig/20260429-165431-fceratto.json 2407644 wikitext text/x-wiki 4k3r0r298v05baa7eywnltytred53gf 2407645 2407644 2026-04-29T17:01:54Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2067.codfw.wmnet with reason: host reimage 2407645 wikitext text/x-wiki lsf79r9baozpekf1n2yb4bjsmw0tp08 2407646 2407645 2026-04-29T17:01:59Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T419961)', diff saved to https://phabricator.wikimedia.org/P91972 and previous config saved to /var/cache/conftool/dbconfig/20260429-170157-fceratto.json 2407646 wikitext text/x-wiki sv0k2jhr21ua72qfzdbrr82kfo27kta 2407647 2407646 2026-04-29T17:03:47Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2066.codfw.wmnet with OS bullseye 2407647 wikitext text/x-wiki 7i69i0umvifipti0dngd3ogsqw2ip4a 2407648 2407647 2026-04-29T17:09:00Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2067.codfw.wmnet with reason: host reimage 2407648 wikitext text/x-wiki 3vbsvz4byfk5yd738nsheurpup3rr29 2407649 2407648 2026-04-29T17:12:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P91973 and previous config saved to /var/cache/conftool/dbconfig/20260429-171205-fceratto.json 2407649 wikitext text/x-wiki eb8xs3l8myjhoerhkmt5z4ofyok44m5 2407650 2407649 2026-04-29T17:22:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P91975 and previous config saved to /var/cache/conftool/dbconfig/20260429-172214-fceratto.json 2407650 wikitext text/x-wiki ni63cxexpdaq4h3olvyavp2mt4ztskq 2407651 2407650 2026-04-29T17:27:14Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2067.codfw.wmnet with OS bullseye 2407651 wikitext text/x-wiki l6i4zxcfvhsw2te46uunzl3kyzi00r4 2407653 2407651 2026-04-29T17:32:25Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T419961)', diff saved to https://phabricator.wikimedia.org/P91976 and previous config saved to /var/cache/conftool/dbconfig/20260429-173222-fceratto.json 2407653 wikitext text/x-wiki 5mk0lh9vh3yexpos93wjlkfy5sxb5st 2407654 2407653 2026-04-29T17:32:47Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance 2407654 wikitext text/x-wiki arkmr28otqdj9g7avcnq0ro8stk9f80 2407655 2407654 2026-04-29T17:32:56Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2166 (T419961)', diff saved to https://phabricator.wikimedia.org/P91977 and previous config saved to /var/cache/conftool/dbconfig/20260429-173253-fceratto.json 2407655 wikitext text/x-wiki tflppv32n4su5934wc8u9sznzv3u5c8 2407656 2407655 2026-04-29T17:40:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T419961)', diff saved to https://phabricator.wikimedia.org/P91978 and previous config saved to /var/cache/conftool/dbconfig/20260429-174016-fceratto.json 2407656 wikitext text/x-wiki gmw3sgox3xwzxzkume9eqbzfg1hxtid 2407657 2407656 2026-04-29T17:50:27Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P91979 and previous config saved to /var/cache/conftool/dbconfig/20260429-175024-fceratto.json 2407657 wikitext text/x-wiki ewhljep3mj7a8ddbt7avwbi6rfd5nf2 2407658 2407657 2026-04-29T18:00:35Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P91980 and previous config saved to /var/cache/conftool/dbconfig/20260429-180032-fceratto.json 2407658 wikitext text/x-wiki 8o2pz2ulompwu6tfami4bgr9phh2ft0 2407659 2407658 2026-04-29T18:10:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T419961)', diff saved to https://phabricator.wikimedia.org/P91981 and previous config saved to /var/cache/conftool/dbconfig/20260429-181041-fceratto.json 2407659 wikitext text/x-wiki 7rc8f9r552969g5yup3t8h7dwmdxt94 2407660 2407659 2026-04-29T18:11:05Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance 2407660 wikitext text/x-wiki 4rukqzi7to58i4j35xebtwo7fm2uahl 2407661 2407660 2026-04-29T18:11:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2167 (T419961)', diff saved to https://phabricator.wikimedia.org/P91982 and previous config saved to /var/cache/conftool/dbconfig/20260429-181111-fceratto.json 2407661 wikitext text/x-wiki o6kke7jwvt7kg71ol2kv7iynsbdgcpo 2407662 2407661 2026-04-29T18:16:06Z Stashbot 7414 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.46.0-wmf.26 refs T423877 2407662 wikitext text/x-wiki t612y7usb52vzwb6pzpwhx36sf45d0w 2407663 2407662 2026-04-29T18:18:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2167 (T419961)', diff saved to https://phabricator.wikimedia.org/P91983 and previous config saved to /var/cache/conftool/dbconfig/20260429-181829-fceratto.json 2407663 wikitext text/x-wiki kcnv4argpf58jjkr8vgwwood4e1xtch 2407664 2407663 2026-04-29T18:28:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P91984 and previous config saved to /var/cache/conftool/dbconfig/20260429-182837-fceratto.json 2407664 wikitext text/x-wiki iquggd98r3jzb2al07px7omi9780fnf 2407665 2407664 2026-04-29T18:38:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P91985 and previous config saved to /var/cache/conftool/dbconfig/20260429-183845-fceratto.json 2407665 wikitext text/x-wiki 5rs7kjv4nik2114kfyveftbvsw4lvlv 2407668 2407665 2026-04-29T18:48:57Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2167 (T419961)', diff saved to https://phabricator.wikimedia.org/P91986 and previous config saved to /var/cache/conftool/dbconfig/20260429-184854-fceratto.json 2407668 wikitext text/x-wiki em4j03zy32ot0vrsomeyzgl8mzaq0p2 2407669 2407668 2026-04-29T18:49:19Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance 2407669 wikitext text/x-wiki hjqtpvjm4swx1e81e06wgvk3u2di4bk 2407670 2407669 2026-04-29T18:49:27Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2181 (T419961)', diff saved to https://phabricator.wikimedia.org/P91987 and previous config saved to /var/cache/conftool/dbconfig/20260429-184925-fceratto.json 2407670 wikitext text/x-wiki cmh6f6dxbd3vsxyldkk00r37o0u8rsg 2407674 2407670 2026-04-29T18:56:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T419961)', diff saved to https://phabricator.wikimedia.org/P91988 and previous config saved to /var/cache/conftool/dbconfig/20260429-185634-fceratto.json 2407674 wikitext text/x-wiki 7bbdkd5376c0anmpb3ryacjb9k6fppu 2407675 2407674 2026-04-29T19:06:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P91989 and previous config saved to /var/cache/conftool/dbconfig/20260429-190641-fceratto.json 2407675 wikitext text/x-wiki eirmvghv95ebb7tp0iq0t6ujhogeare 2407680 2407675 2026-04-29T19:16:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P91990 and previous config saved to /var/cache/conftool/dbconfig/20260429-191650-fceratto.json 2407680 wikitext text/x-wiki oz0iz3rbjvs4t2org9i3pk0781dz2e0 2407683 2407680 2026-04-29T19:20:28Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.dns.netbox 2407683 wikitext text/x-wiki ooj0o99uj3zrlnnsew6h99ua0li1ij5 2407684 2407683 2026-04-29T19:24:37Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns infor for asw1-23-ulsfo - pt1979@cumin2002" 2407684 wikitext text/x-wiki 77nhz9r35lukfnfcuuyscfmdviohmiv 2407685 2407684 2026-04-29T19:25:05Z Stashbot 7414 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns infor for asw1-23-ulsfo - pt1979@cumin2002" 2407685 wikitext text/x-wiki isiy1ylro5ochtjh5xcgcarmlddoue8 2407686 2407685 2026-04-29T19:25:08Z Stashbot 7414 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2407686 wikitext text/x-wiki bedywr3vv65z8cvkumiwrbvrm6qe4d4 2407687 2407686 2026-04-29T19:27:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T419961)', diff saved to https://phabricator.wikimedia.org/P91992 and previous config saved to /var/cache/conftool/dbconfig/20260429-192658-fceratto.json 2407687 wikitext text/x-wiki 13mo74ab4ob4d0pqh7d20whwbi8vdf3 2407688 2407687 2026-04-29T19:27:23Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance 2407688 wikitext text/x-wiki pgobmmoi3ggvi6vvs8qcpsyxhlhd1tr 2407689 2407688 2026-04-29T19:27:32Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2195 (T419961)', diff saved to https://phabricator.wikimedia.org/P91993 and previous config saved to /var/cache/conftool/dbconfig/20260429-192729-fceratto.json 2407689 wikitext text/x-wiki imvb7ynjjuft8sxqlfkbd1bt4un5fsb 2407691 2407689 2026-04-29T19:34:35Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2195 (T419961)', diff saved to https://phabricator.wikimedia.org/P91994 and previous config saved to /var/cache/conftool/dbconfig/20260429-193431-fceratto.json 2407691 wikitext text/x-wiki 8npv61gaqcugjln37ggdct4rme4d4gw 2407694 2407691 2026-04-29T19:44:41Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P91995 and previous config saved to /var/cache/conftool/dbconfig/20260429-194439-fceratto.json 2407694 wikitext text/x-wiki gi7je5ahl6ecw787zztv65xt3pt3f4n 2407695 2407694 2026-04-29T19:54:49Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P91996 and previous config saved to /var/cache/conftool/dbconfig/20260429-195447-fceratto.json 2407695 wikitext text/x-wiki hf15yvg19ybdllunwewe3n5c9nlkqab 2407696 2407695 2026-04-29T20:04:57Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2195 (T419961)', diff saved to https://phabricator.wikimedia.org/P91997 and previous config saved to /var/cache/conftool/dbconfig/20260429-200455-fceratto.json 2407696 wikitext text/x-wiki 8pv3gpuw2g0gosp00r51h1spmzt3zum 2407697 2407696 2026-04-29T20:10:38Z Stashbot 7414 dancy@deploy1003: Started scap sync-world: Backport for [[gerrit:1279406|cswiki: lift IP cap for edithathon (T424843)]], [[gerrit:1279422|arbcom_enwiki: update logo, icon (T424555)]], [[gerrit:1279404|enwikiquote: enable UseSandboxLink (T424863)]] 2407697 wikitext text/x-wiki dgsvianim1b5rfjjonqv61trpupjrnb 2407698 2407697 2026-04-29T20:12:31Z Stashbot 7414 dancy@deploy1003: anzx, dancy: Backport for [[gerrit:1279406|cswiki: lift IP cap for edithathon (T424843)]], [[gerrit:1279422|arbcom_enwiki: update logo, icon (T424555)]], [[gerrit:1279404|enwikiquote: enable UseSandboxLink (T424863)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2407698 wikitext text/x-wiki h5s4yg647ek89fqen2dlidc2jxq5dor 2407700 2407698 2026-04-29T20:13:42Z Stashbot 7414 dancy@deploy1003: anzx, dancy: Continuing with deployment 2407700 wikitext text/x-wiki jd3r2rusv2aenda47itbw1gqka5y57f 2407702 2407700 2026-04-29T20:17:38Z Stashbot 7414 dancy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1279406|cswiki: lift IP cap for edithathon (T424843)]], [[gerrit:1279422|arbcom_enwiki: update logo, icon (T424555)]], [[gerrit:1279404|enwikiquote: enable UseSandboxLink (T424863)]] (duration: 06m 59s) 2407702 wikitext text/x-wiki dt5hannhnn2jcybqxsmw5rthsjqlmif 2407703 2407702 2026-04-29T20:20:01Z Stashbot 7414 dancy@deploy1003: Started scap sync-world: Backport for [[gerrit:1265959|arbcom_zhwiki: Enable SecurePoll without PII rights (T419309)]] 2407703 wikitext text/x-wiki 68bpexi92wvg9n7pin6t34et0n1d7r4 2407704 2407703 2026-04-29T20:21:54Z Stashbot 7414 dancy@deploy1003: 1f616emo, dancy: Backport for [[gerrit:1265959|arbcom_zhwiki: Enable SecurePoll without PII rights (T419309)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2407704 wikitext text/x-wiki t61piima63zd1h32czpuycwu2wubpwt 2407705 2407704 2026-04-29T20:23:31Z Stashbot 7414 dancy@deploy1003: 1f616emo, dancy: Continuing with deployment 2407705 wikitext text/x-wiki i1wuz55y7jkgx06ume7vzqguqxhmv23 2407706 2407705 2026-04-29T20:27:17Z Stashbot 7414 dancy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1265959|arbcom_zhwiki: Enable SecurePoll without PII rights (T419309)]] (duration: 07m 16s) 2407706 wikitext text/x-wiki so2a34mstvn5lbivm5ycs07qmb5cscd 2407707 2407706 2026-04-29T20:32:56Z Stashbot 7414 dancy@deploy1003: Started scap sync-world: Backport for [[gerrit:1277569|Enable mobile editor abandonment survey on enwiki (T423923)]], [[gerrit:1279448|Abandon editor survey: UI updates (T422931)]] 2407707 wikitext text/x-wiki 6fadbeysnqsoat7m8wfup1ayxf4gbrk 2407708 2407707 2026-04-29T20:34:53Z Stashbot 7414 dancy@deploy1003: dancy, caro, kemayo: Backport for [[gerrit:1277569|Enable mobile editor abandonment survey on enwiki (T423923)]], [[gerrit:1279448|Abandon editor survey: UI updates (T422931)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2407708 wikitext text/x-wiki 8n7nqw7dkxjw42hznl9x0cnwdvwybr6 2407713 2407708 2026-04-29T20:40:56Z Stashbot 7414 dancy@deploy1003: dancy, caro, kemayo: Continuing with deployment 2407713 wikitext text/x-wiki owif2yhe6oand8y9foyfq63qrjgxsfu 2407714 2407713 2026-04-29T20:44:37Z Stashbot 7414 dancy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1277569|Enable mobile editor abandonment survey on enwiki (T423923)]], [[gerrit:1279448|Abandon editor survey: UI updates (T422931)]] (duration: 11m 43s) 2407714 wikitext text/x-wiki fokcpqnj1cps9h1eqmlk3jytv6jykfr 2407715 2407714 2026-04-29T20:48:46Z Stashbot 7414 phuedx@deploy1003: Started scap sync-world: Backport for [[gerrit:1279431|Load TestKitchen earlier (T424876)]] 2407715 wikitext text/x-wiki 1dhoka6zlvhhlft2r5c27bxg7akc817 2407716 2407715 2026-04-29T20:50:40Z Stashbot 7414 phuedx@deploy1003: phuedx, sbisson: Backport for [[gerrit:1279431|Load TestKitchen earlier (T424876)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2407716 wikitext text/x-wiki rhsjbz5crgls0clk51ig3p04w4disn9 2407717 2407716 2026-04-29T20:56:27Z Stashbot 7414 phuedx@deploy1003: phuedx, sbisson: Continuing with deployment 2407717 wikitext text/x-wiki 8cf84bwq31u75x9pjxa69uc4vy7k582 2407718 2407717 2026-04-29T21:00:19Z Stashbot 7414 phuedx@deploy1003: Finished scap sync-world: Backport for [[gerrit:1279431|Load TestKitchen earlier (T424876)]] (duration: 11m 33s) 2407718 wikitext text/x-wiki 9og6h8d5pi5r5jr8ayejn8qoql9cbj9 2407720 2407718 2026-04-29T21:04:45Z Stashbot 7414 dancy@deploy1003: Started scap sync-world: Backport for [[gerrit:1279430|enwikiversity: Enable the abuse filter block action on English Wikiversity (T424053)]], [[gerrit:1278363|enwikiversity: Add some user rights to the curator user group on English Wikiversity (T424445)]] 2407720 wikitext text/x-wiki rttqrxm2k9omxhtvq33ncvd7zjoqv1r 2407721 2407720 2026-04-29T21:06:36Z Stashbot 7414 dancy@deploy1003: vadymts1, dancy: Backport for [[gerrit:1279430|enwikiversity: Enable the abuse filter block action on English Wikiversity (T424053)]], [[gerrit:1278363|enwikiversity: Add some user rights to the curator user group on English Wikiversity (T424445)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2407721 wikitext text/x-wiki 5po9f56bhh87wcp6br3pl0iowb13bot 2407729 2407721 2026-04-29T22:03:16Z Stashbot 7414 dancy@deploy1003: vadymts1, dancy: Continuing with deployment 2407729 wikitext text/x-wiki 6xeap306xyym7ty8udzsqrh48yi9ogg 2407730 2407729 2026-04-29T22:07:10Z Stashbot 7414 dancy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1279430|enwikiversity: Enable the abuse filter block action on English Wikiversity (T424053)]], [[gerrit:1278363|enwikiversity: Add some user rights to the curator user group on English Wikiversity (T424445)]] (duration: 62m 26s) 2407730 wikitext text/x-wiki 5ju8444dae0efn6va5al7bgaehhvjqj 2407731 2407730 2026-04-29T22:18:53Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1277770|Deploy PRV to 12 wikis (T424590)]], [[gerrit:1279450|Deploy Parsoid Read Views to 6 language converter wikis (T423785)]] 2407731 wikitext text/x-wiki kjwxcvlc4pfl6rqo2wsq1n3bmk7ki25 2407732 2407731 2026-04-29T22:20:47Z Stashbot 7414 cscott@deploy1003: arlolra, cscott: Backport for [[gerrit:1277770|Deploy PRV to 12 wikis (T424590)]], [[gerrit:1279450|Deploy Parsoid Read Views to 6 language converter wikis (T423785)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2407732 wikitext text/x-wiki di7k4wfnk22pockaj616gc8z3912esv 2407734 2407732 2026-04-29T22:35:05Z Stashbot 7414 cscott@deploy1003: arlolra, cscott: Continuing with deployment 2407734 wikitext text/x-wiki hylufstvi0hffaz6066a7hcc3o3k50i 2407735 2407734 2026-04-29T22:39:56Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1277770|Deploy PRV to 12 wikis (T424590)]], [[gerrit:1279450|Deploy Parsoid Read Views to 6 language converter wikis (T423785)]] (duration: 21m 03s) 2407735 wikitext text/x-wiki ikjul1qfjo7d0kve22dyrncq6adu4w5 2407736 2407735 2026-04-29T22:50:30Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1279452|Enable Parsoid Read Views for 20% of enwiki mobile web traffic (T424880)]] 2407736 wikitext text/x-wiki 6f523rfhk38zxdz0wcjy06ezsz571mn 2407737 2407736 2026-04-29T22:52:25Z Stashbot 7414 cscott@deploy1003: cscott: Backport for [[gerrit:1279452|Enable Parsoid Read Views for 20% of enwiki mobile web traffic (T424880)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2407737 wikitext text/x-wiki lc42syral903zlh176ghx09vapu5cta 2407738 2407737 2026-04-29T22:59:56Z Stashbot 7414 cscott@deploy1003: cscott: Continuing with deployment 2407738 wikitext text/x-wiki ofg0upcdzja5iwphnvsl8eyfuh1i89s 2407739 2407738 2026-04-29T23:04:33Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1279452|Enable Parsoid Read Views for 20% of enwiki mobile web traffic (T424880)]] (duration: 14m 03s) 2407739 wikitext text/x-wiki ev8lomcxmrmp4da2d1o881vngsr6qmf 2407742 2407739 2026-04-29T23:18:11Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407742 wikitext text/x-wiki ghei3oosjta7lp2kqvr4nai1769f04v 2407743 2407742 2026-04-29T23:18:26Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407743 wikitext text/x-wiki qznoqor3w7t11qesx0jng87hfsfrn5v 2407744 2407743 2026-04-29T23:18:28Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407744 wikitext text/x-wiki m5wdpyfqq5ps6iqz3sd48jb3i6zwots 2407745 2407744 2026-04-29T23:18:30Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407745 wikitext text/x-wiki 8qpn94bstrjj13zsugdkajrt31mt68k 2407746 2407745 2026-04-29T23:18:33Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-ctrl1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407746 wikitext text/x-wiki gbyikiid9w4dz0u1qevntj70vuagtiu 2407747 2407746 2026-04-29T23:18:36Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407747 wikitext text/x-wiki d527es316hl4d3l5a0bvmu2w7wiqi4q 2407748 2407747 2026-04-29T23:19:50Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host tools-k8s-worker1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407748 wikitext text/x-wiki m59vsb26fqf8j5qg6ps69nrscc1zute 2407749 2407748 2026-04-29T23:19:53Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host tools-k8s-worker1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407749 wikitext text/x-wiki irod9rw7ea90fjfoqtijywqvhmfcoas 2407750 2407749 2026-04-29T23:19:56Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host tools-k8s-ctrl1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407750 wikitext text/x-wiki mbak876bmh5grebrej4wr7ur3ge0x80 2407751 2407750 2026-04-29T23:19:58Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host tools-k8s-worker1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407751 wikitext text/x-wiki pydeyutfjucgg0836l58ou3re97tvev 2407752 2407751 2026-04-29T23:24:13Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.dns.netbox 2407752 wikitext text/x-wiki j1nlp2xgsegb18w1fmvssxkri26zt8d 2407753 2407752 2026-04-29T23:26:01Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host tools-k8s-worker1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407753 wikitext text/x-wiki tscb1iclmooc0c049un0f9xhd3x3isp 2407754 2407753 2026-04-29T23:27:06Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host tools-k8s-worker1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407754 wikitext text/x-wiki bno26yklwodn8ujnp6bprb71m7uov4a 2407755 2407754 2026-04-29T23:27:40Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1375.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407755 wikitext text/x-wiki m9tet24dqappf4pl7z7m0wlz1ubuy84 2407756 2407755 2026-04-29T23:28:02Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker1375 to eqiad - jclark@cumin1003" 2407756 wikitext text/x-wiki ckfavh8uco96b6dsvsn30ujbus0u7ug 2407757 2407756 2026-04-29T23:28:08Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker1375 to eqiad - jclark@cumin1003" 2407757 wikitext text/x-wiki 5ih6remrj5ekbapbcgxsotkqv0f5lfm 2407758 2407757 2026-04-29T23:28:11Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2407758 wikitext text/x-wiki 5jipde9mlyjlt1ftfrpq1j5nmfoozbx 2407759 2407758 2026-04-29T23:28:14Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1376.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407759 wikitext text/x-wiki sofhkao1pr5m18xm11ffai8j4fibc93 2407760 2407759 2026-04-29T23:28:23Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1377.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407760 wikitext text/x-wiki 10xa9ue3gdvgoioeizeg1mnicload1f 2407761 2407760 2026-04-29T23:28:29Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1375.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407761 wikitext text/x-wiki qvvbkohgek475lspwssyf36gd7eangt 2407762 2407761 2026-04-29T23:28:45Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1377.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407762 wikitext text/x-wiki bm39yzoncrm2p0igcngnr2oaav9a3by 2407763 2407762 2026-04-29T23:29:02Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1378.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407763 wikitext text/x-wiki hufqhxw01xxk1zo34vsrm8xdmgzh1fm 2407764 2407763 2026-04-29T23:29:12Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1378.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407764 wikitext text/x-wiki dwfflmllkp4p8fjygab2287n6crerbz 2407765 2407764 2026-04-29T23:29:36Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1379.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407765 wikitext text/x-wiki f25ppfxoy1ym9zvq482irl8dt33cyd1 2407766 2407765 2026-04-29T23:30:05Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407766 wikitext text/x-wiki 6bcz0lmhk8gp0acctu6nxkezwga9mjz 2407767 2407766 2026-04-29T23:30:26Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407767 wikitext text/x-wiki 1n5dytxp0zp0jo6sxsom0l8z91dnmzj 2407770 2407767 2026-04-29T23:54:45Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1376.eqiad.wmnet with reason: host reimage 2407770 wikitext text/x-wiki kjyhi84t7ig4m1a58cq06xtagfuboh8 2407771 2407770 2026-04-29T23:54:51Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1375.eqiad.wmnet with reason: host reimage 2407771 wikitext text/x-wiki 2kkasds6gxus8e9yfq8kzgouc1iqo9r 2407772 2407771 2026-04-29T23:55:30Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1379.eqiad.wmnet with reason: host reimage 2407772 wikitext text/x-wiki 4sbk9ijffrbk5vldiq73ezdwfkydi0t 2407773 2407772 2026-04-29T23:56:05Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1381.eqiad.wmnet with reason: host reimage 2407773 wikitext text/x-wiki h7xhsr7yrzrdchhik46byjts1t8obrs 2407774 2407773 2026-04-29T23:56:10Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1380.eqiad.wmnet with reason: host reimage 2407774 wikitext text/x-wiki 3d3e6a76xf114vpvte34v3vf77torpq 2407775 2407774 2026-04-29T23:57:15Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1382.eqiad.wmnet with reason: host reimage 2407775 wikitext text/x-wiki 49e4grcoky1eadvsupvngm1g71e7o5l 2407776 2407775 2026-04-29T23:58:51Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1376.eqiad.wmnet with reason: host reimage 2407776 wikitext text/x-wiki 9g4ukp1i04oniat974tj62g8ohag7u8 2407778 2407776 2026-04-30T00:02:51Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1382.eqiad.wmnet with reason: host reimage 2407778 wikitext text/x-wiki hhezzbhefyzzqbdv1fera1j4di419re 2407779 2407778 2026-04-30T00:05:44Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1381.eqiad.wmnet with reason: host reimage 2407779 wikitext text/x-wiki 1n7jskjase0syi2omrhxn1mxwlv7iro 2407780 2407779 2026-04-30T00:09:16Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1379.eqiad.wmnet with reason: host reimage 2407780 wikitext text/x-wiki islrwydyjo1sl4duvwfg12th5w7t6u9 2407781 2407780 2026-04-30T00:12:04Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wikikube-worker1380.eqiad.wmnet with reason: host reimage 2407781 wikitext text/x-wiki 1qkwtxeu03y57byrcqf8atu20s2v0gr 2407782 2407781 2026-04-30T00:12:07Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1375.eqiad.wmnet with reason: host reimage 2407782 wikitext text/x-wiki aqb0krumenhdbllmidv8jc3jfqe8js8 2407783 2407782 2026-04-30T00:14:38Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2407783 wikitext text/x-wiki 68oez03o80mz645b5fe6ketwtr4f0yn 2407784 2407783 2026-04-30T00:15:09Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2407784 wikitext text/x-wiki 0te2b4e0re7qivbml1axkl28j7xjx63 2407785 2407784 2026-04-30T00:15:12Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1376.eqiad.wmnet with OS trixie 2407785 wikitext text/x-wiki 3tvgu0o07pvlk9rfyqkuna3mxouj9pz 2407786 2407785 2026-04-30T00:16:10Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1383.eqiad.wmnet with OS trixie 2407786 wikitext text/x-wiki icbwi1u85ivfm18nhpmwh2hmtwltdjq 2407787 2407786 2026-04-30T00:18:57Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2407787 wikitext text/x-wiki bd4eqq8ulq848nc5j5ngu3gjo49hnkt 2407788 2407787 2026-04-30T00:19:16Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2407788 wikitext text/x-wiki 6rubsedr1vzg2zmi0n8hksy9ggrfqg1 2407789 2407788 2026-04-30T00:19:19Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1382.eqiad.wmnet with OS trixie 2407789 wikitext text/x-wiki baqn7fmyo0dweroh91k07ri6dngi0m1 2407790 2407789 2026-04-30T00:19:53Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1384.eqiad.wmnet with OS trixie 2407790 wikitext text/x-wiki 6mnlo9ik9gbnerwxzs3v8wge4rx5dck 2407791 2407790 2026-04-30T00:21:45Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1380.eqiad.wmnet with OS trixie 2407791 wikitext text/x-wiki 1zk9ax3hd2h487r83gwkm0e2h54ero8 2407792 2407791 2026-04-30T00:22:18Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2407792 wikitext text/x-wiki 07fdxkizkx2b3avmsgu13vsb6wdj2aw 2407793 2407792 2026-04-30T00:22:38Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2407793 wikitext text/x-wiki t39freqeqwgqyc6msghqhluze9tru0b 2407794 2407793 2026-04-30T00:22:41Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1381.eqiad.wmnet with OS trixie 2407794 wikitext text/x-wiki doygahdqmmp8z4c6l1lkk6n5vrby9j1 2407795 2407794 2026-04-30T00:24:19Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1380.eqiad.wmnet with OS trixie 2407795 wikitext text/x-wiki 28odwnl4fk7ckjd088fy9ia62rbkbnz 2407796 2407795 2026-04-30T00:25:28Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2407796 wikitext text/x-wiki bit0537pajkll9piugi223bmzeiajim 2407797 2407796 2026-04-30T00:26:12Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2407797 wikitext text/x-wiki 802zbonacx7av412a5w474cjxw9xgqt 2407798 2407797 2026-04-30T00:26:15Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1379.eqiad.wmnet with OS trixie 2407798 wikitext text/x-wiki 9w067orkio65diq85a1x6sze00evw9z 2407799 2407798 2026-04-30T00:28:19Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1383.eqiad.wmnet with reason: host reimage 2407799 wikitext text/x-wiki n4e8opw198rs7obpnntvl9sxkioltqb 2407800 2407799 2026-04-30T00:29:39Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2407800 wikitext text/x-wiki 6f6phwjwbod79xzpza42sslmjicu6fl 2407801 2407800 2026-04-30T00:30:02Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2407801 wikitext text/x-wiki c4wjyjpiw80fwbeizjh1mqu6yqxizp1 2407802 2407801 2026-04-30T00:30:05Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1375.eqiad.wmnet with OS trixie 2407802 wikitext text/x-wiki 9qmpksagy79q4vpuqr3jj6tljy4fo8y 2407803 2407802 2026-04-30T00:31:40Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1384.eqiad.wmnet with reason: host reimage 2407803 wikitext text/x-wiki 3zrpzcy5kig6vpoajy59bj9l1j920xv 2407804 2407803 2026-04-30T00:33:46Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1383.eqiad.wmnet with reason: host reimage 2407804 wikitext text/x-wiki g82n8wnd9cojx75mwb58l568ve420mt 2407805 2407804 2026-04-30T00:36:45Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1380.eqiad.wmnet with reason: host reimage 2407805 wikitext text/x-wiki giw5h51vc1wmkc3e09hx3q3ol0gatc2 2407806 2407805 2026-04-30T00:37:55Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1384.eqiad.wmnet with reason: host reimage 2407806 wikitext text/x-wiki q6j2fnhbvrzxrpldilopft4vsbf4lq7 2407809 2407806 2026-04-30T00:41:38Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1380.eqiad.wmnet with reason: host reimage 2407809 wikitext text/x-wiki 2xgbbnwwxttwgof5hqhd205hnjisxvh 2407810 2407809 2026-04-30T00:49:51Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2407810 wikitext text/x-wiki rqjo7s5ws06j5b69p3w8fjf0jj4enei 2407811 2407810 2026-04-30T00:50:09Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2407811 wikitext text/x-wiki 8kmyowfl5o05ev9jsd7cc0c2oei3605 2407812 2407811 2026-04-30T00:50:12Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1383.eqiad.wmnet with OS trixie 2407812 wikitext text/x-wiki 0s3scquf2t92g588noz3d1e9f4jw0c3 2407813 2407812 2026-04-30T00:53:48Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2407813 wikitext text/x-wiki cpock5hzyhqdwu7rb6qucvlbnz40bar 2407814 2407813 2026-04-30T00:57:03Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2407814 wikitext text/x-wiki sx1hi7o7u7bgxtn3m5rny7smuhotlol 2407815 2407814 2026-04-30T00:57:05Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1384.eqiad.wmnet with OS trixie 2407815 wikitext text/x-wiki o8gbvd5decu3wfn3eju1qq86e4mtdol 2407816 2407815 2026-04-30T00:57:19Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2407816 wikitext text/x-wiki 2o5gtldel0vo56jqrbdnr188qtw991z 2407817 2407816 2026-04-30T00:58:16Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2407817 wikitext text/x-wiki 1nvvattsteb1r3eqdsdi5r8ku5dhdkh 2407818 2407817 2026-04-30T00:58:18Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1380.eqiad.wmnet with OS trixie 2407818 wikitext text/x-wiki adzk6plvjmixdsocfob8okoqkmcjn03 2407819 2407818 2026-04-30T02:01:44Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2407819 wikitext text/x-wiki hixop3zdweqylt5giqrk6x55t672hoi 2407820 2407819 2026-04-30T02:08:10Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 26s) 2407820 wikitext text/x-wiki pn9yw1pd8su4kinffgqo0zx96p49q8h 2407823 2407820 2026-04-30T04:53:03Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2149.codfw.wmnet with reason: Reimage to Trixie 2407823 wikitext text/x-wiki muuyfm2fxsr3x7g8ht39kmadqomyyw1 2407824 2407823 2026-04-30T04:53:08Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2149: Reimage to Trixie 2407824 wikitext text/x-wiki 718ivmbrdtybq0rbgqk41u4ggytkyl3 2407825 2407824 2026-04-30T04:53:27Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2149: Reimage to Trixie 2407825 wikitext text/x-wiki 5kktq7rs07snxv1hqn3kx7dzaseu9hn 2407826 2407825 2026-04-30T04:55:49Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2149.codfw.wmnet with OS trixie 2407826 wikitext text/x-wiki k82c7i038lwz9cq27k5abl7vibhq5x5 2407828 2407826 2026-04-30T04:59:48Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts pc2012.codfw.wmnet 2407828 wikitext text/x-wiki beankd4cocoug8tdunarr8ylnfjwkkn 2407829 2407828 2026-04-30T05:04:29Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.dns.netbox 2407829 wikitext text/x-wiki cnma95no0l7tafimj4tifoak2u5753w 2407830 2407829 2026-04-30T05:09:35Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc2012.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2407830 wikitext text/x-wiki 6lamqid1du6zm12uamv90bmuk006ga6 2407831 2407830 2026-04-30T05:09:52Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pc2012.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2407831 wikitext text/x-wiki 3nm660twg988nwwrnrn6dpa98talwlo 2407832 2407831 2026-04-30T05:09:55Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2407832 wikitext text/x-wiki 9j0n9i5vvmrtre1mxtea5pqyw6byxx2 2407833 2407832 2026-04-30T05:09:57Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc2012.codfw.wmnet 2407833 wikitext text/x-wiki dplovmt1d2p6r0qfq3i7ntts99mvxct 2407834 2407833 2026-04-30T05:14:25Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2149.codfw.wmnet with reason: host reimage 2407834 wikitext text/x-wiki 6u4s1unu2v2gp5vi5az9oad8vi3k5wn 2407835 2407834 2026-04-30T05:18:43Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2149.codfw.wmnet with reason: host reimage 2407835 wikitext text/x-wiki snvrdlndr9rxrcfel2bgjkvnw49sjsj 2407836 2407835 2026-04-30T05:20:24Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts db2146.codfw.wmnet 2407836 wikitext text/x-wiki 5ppta7hbtkhz2efkblmx1c7lnigt3id 2407837 2407836 2026-04-30T05:27:20Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.dns.netbox 2407837 wikitext text/x-wiki 7w0w2fojnvvb2xigvcp7tqfs22htzn0 2407838 2407837 2026-04-30T05:31:23Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2146.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2407838 wikitext text/x-wiki 96k8wapytv356cccaq39xkn1glaeeff 2407839 2407838 2026-04-30T05:33:24Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2146.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2407839 wikitext text/x-wiki 8s3iuvftrl5m1284e0e9bwiewedgpzq 2407840 2407839 2026-04-30T05:33:26Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2407840 wikitext text/x-wiki 62opgbyt1gmbsah9wfhmao8o534xcbe 2407841 2407840 2026-04-30T05:33:30Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2146.codfw.wmnet 2407841 wikitext text/x-wiki 2k8kmnqkuvkjkwfpyxvtyjclgz4x5f6 2407842 2407841 2026-04-30T05:37:16Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Remove db2147 from dbctl T424226', diff saved to https://phabricator.wikimedia.org/P92000 and previous config saved to /var/cache/conftool/dbconfig/20260430-053712-marostegui.json 2407842 wikitext text/x-wiki tcgb2w34mamjixegg6xk20aph9xu8ad 2407843 2407842 2026-04-30T05:40:57Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2149.codfw.wmnet with OS trixie 2407843 wikitext text/x-wiki 7dznwfv3o9cal22e4dpza4kwppy73wd 2407845 2407843 2026-04-30T05:47:08Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2149: after reimage to trixie 2407845 wikitext text/x-wiki 1pjnn2m52fu1jedi6yr8z3iwn7cml7e 2407850 2407845 2026-04-30T06:32:34Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2149: after reimage to trixie 2407850 wikitext text/x-wiki dum6yivqqidd2gypgrnaw0aqam3chzu 2407852 2407850 2026-04-30T07:03:15Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host hcaptcha-proxy5003.wikimedia.org 2407852 wikitext text/x-wiki 140m1ke31hsora5yb4jh25m4zhjwtms 2407853 2407852 2026-04-30T07:03:17Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2407853 wikitext text/x-wiki 6qqa7xuqpsethqflop707lheewvs8uh 2407854 2407853 2026-04-30T07:15:43Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM hcaptcha-proxy5003.wikimedia.org - jmm@cumin2002" 2407854 wikitext text/x-wiki junu5n6fion5wdwepa7c3j6z8i77i1y 2407855 2407854 2026-04-30T07:15:48Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM hcaptcha-proxy5003.wikimedia.org - jmm@cumin2002" 2407855 wikitext text/x-wiki 8n99yykzycsoz9yrw7nfhrw763ql4au 2407856 2407855 2026-04-30T07:15:51Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2407856 wikitext text/x-wiki 3xazd4kovepe0z5lr8qc2gv6921nd9x 2407857 2407856 2026-04-30T07:15:53Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache hcaptcha-proxy5003.wikimedia.org on all recursors 2407857 wikitext text/x-wiki 5raj1rzfasr57d61n87sdtpnapgqr8i 2407858 2407857 2026-04-30T07:15:56Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) hcaptcha-proxy5003.wikimedia.org on all recursors 2407858 wikitext text/x-wiki 67e86cju1n9i4v8n1x2mtv15y6svrfl 2407859 2407858 2026-04-30T07:16:28Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM hcaptcha-proxy5003.wikimedia.org - jmm@cumin2002" 2407859 wikitext text/x-wiki jxvi7e2f2ypjk4bn345u8n41a0faqy9 2407860 2407859 2026-04-30T07:16:33Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM hcaptcha-proxy5003.wikimedia.org - jmm@cumin2002" 2407860 wikitext text/x-wiki dkyz65hwvllhebobjmur8ybrpzzqp8b 2407861 2407860 2026-04-30T07:16:36Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4006.ulsfo.wmnet 2407861 wikitext text/x-wiki q76gqexir6iycuyigqemzqj4t3hn2jd 2407862 2407861 2026-04-30T07:17:39Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4006.ulsfo.wmnet 2407862 wikitext text/x-wiki 8humy95je4llpwv2c97vwu2gpgjuf2i 2407863 2407862 2026-04-30T07:25:12Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4006.ulsfo.wmnet 2407863 wikitext text/x-wiki 4l9ngfauldus430qhinv0fja7dbvtdf 2407864 2407863 2026-04-30T07:33:27Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance 2407864 wikitext text/x-wiki 6c9mn67p1nrctr64bosnl3xh2133ko5 2407865 2407864 2026-04-30T07:36:00Z Stashbot 7414 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1279079|Don't load general modules as style modules (T424618)]] 2407865 wikitext text/x-wiki cz38zkffdjemf8xiobjyta335x273d0 2407866 2407865 2026-04-30T07:37:45Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host hcaptcha-proxy5003.wikimedia.org with OS bookworm 2407866 wikitext text/x-wiki 21woec0g8pvwn2ti71k8ou01gaa2cze 2407867 2407866 2026-04-30T07:38:00Z Stashbot 7414 kartik@deploy1003: kartik, abi: Backport for [[gerrit:1279079|Don't load general modules as style modules (T424618)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2407867 wikitext text/x-wiki 5kxfwir06y4n97yjnaik43j29igt254 2407869 2407867 2026-04-30T07:41:37Z Stashbot 7414 kartik@deploy1003: kartik, abi: Continuing with deployment 2407869 wikitext text/x-wiki 1g9rymzstfbxox6s254hk0tmljrx11i 2407870 2407869 2026-04-30T07:45:29Z Stashbot 7414 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1279079|Don't load general modules as style modules (T424618)]] (duration: 09m 29s) 2407870 wikitext text/x-wiki lwktgaov5ae8n7o43dj8rutlhlt8zhv 2407871 2407870 2026-04-30T07:47:11Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance 2407871 wikitext text/x-wiki j2wwks6ans2x722zvgpyyk76nc2xzvh 2407872 2407871 2026-04-30T07:47:20Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2153 (T419961)', diff saved to https://phabricator.wikimedia.org/P92006 and previous config saved to /var/cache/conftool/dbconfig/20260430-074717-fceratto.json 2407872 wikitext text/x-wiki secvtwp63uaua4rkyy6ost80zv0b9es 2407874 2407872 2026-04-30T07:54:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T419961)', diff saved to https://phabricator.wikimedia.org/P92007 and previous config saved to /var/cache/conftool/dbconfig/20260430-075436-fceratto.json 2407874 wikitext text/x-wiki 9rtvw777pydnn7ot4u7g1runf09r96k 2407875 2407874 2026-04-30T07:55:02Z Stashbot 7414 phuedx@deploy1003: Started scap sync-world: Backport for [[gerrit:1279476|JS SDK: Remove compat deprecation warnings]] 2407875 wikitext text/x-wiki eq96cfcp52sy6obiob87vnth4ce8epj 2407876 2407875 2026-04-30T07:56:53Z Stashbot 7414 phuedx@deploy1003: phuedx: Backport for [[gerrit:1279476|JS SDK: Remove compat deprecation warnings]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2407876 wikitext text/x-wiki r3cmh9k5losd2hv6i0x89mjn1ayw0vq 2407877 2407876 2026-04-30T08:01:18Z Stashbot 7414 phuedx@deploy1003: phuedx: Continuing with deployment 2407877 wikitext text/x-wiki m29a03r8dtipraqe670z722t5hhgrq7 2407878 2407877 2026-04-30T08:04:47Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P92008 and previous config saved to /var/cache/conftool/dbconfig/20260430-080444-fceratto.json 2407878 wikitext text/x-wiki qnbqm40pxv6gtufx2r2jwh15qf6yuqo 2407879 2407878 2026-04-30T08:05:16Z Stashbot 7414 phuedx@deploy1003: Finished scap sync-world: Backport for [[gerrit:1279476|JS SDK: Remove compat deprecation warnings]] (duration: 10m 13s) 2407879 wikitext text/x-wiki 1y30hdgjrt7o53388x48l3vp774txg3 2407880 2407879 2026-04-30T08:08:19Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2407880 wikitext text/x-wiki mty4083j87s35zo0b6yearsggoz3anj 2407881 2407880 2026-04-30T08:09:00Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2407881 wikitext text/x-wiki 7x0ys18svt3jrljyh1pti401ctyb30z 2407882 2407881 2026-04-30T08:09:23Z Stashbot 7414 phuedx: UTC morning backport window finished 2407882 wikitext text/x-wiki h0v25103ra0482981tbvi3bcztccaz5 2407883 2407882 2026-04-30T08:09:26Z Stashbot 7414 moritzm: installing rsync security updates 2407883 wikitext text/x-wiki j0uy8rl1kt3af0uchunvkjdr3pl94is 2407884 2407883 2026-04-30T08:09:30Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2407884 wikitext text/x-wiki 4iqwzj0njtpd9dfpyclyzeqrxqxedhc 2407885 2407884 2026-04-30T08:10:02Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2407885 wikitext text/x-wiki tcju754xuj5wfpfgdjc3vp7celjighr 2407886 2407885 2026-04-30T08:14:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P92009 and previous config saved to /var/cache/conftool/dbconfig/20260430-081452-fceratto.json 2407886 wikitext text/x-wiki mg42wypnryfm5ucfedrwa1ve5foygh0 2407887 2407886 2026-04-30T08:18:33Z Stashbot 7414 moritzm: installing nginx security updates 2407887 wikitext text/x-wiki 1eu6snzroqb22ntpxs9mhdcx401t007 2407888 2407887 2026-04-30T08:25:03Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T419961)', diff saved to https://phabricator.wikimedia.org/P92010 and previous config saved to /var/cache/conftool/dbconfig/20260430-082501-fceratto.json 2407888 wikitext text/x-wiki 22fqevqyzupu3puz7hzgbydarc31b46 2407889 2407888 2026-04-30T08:25:24Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance 2407889 wikitext text/x-wiki oz2jijx2yo06b4yusb4vz2qqjhlcdr9 2407890 2407889 2026-04-30T08:25:32Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2170 (T419961)', diff saved to https://phabricator.wikimedia.org/P92011 and previous config saved to /var/cache/conftool/dbconfig/20260430-082530-fceratto.json 2407890 wikitext text/x-wiki q60qbhpc63mdce6b9u0i5ihv70ddehm 2407891 2407890 2026-04-30T08:25:46Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on hcaptcha-proxy5003.wikimedia.org with reason: host reimage 2407891 wikitext text/x-wiki g6lc0pknbwn6r8oecz97ei1seicu6n2 2407892 2407891 2026-04-30T08:27:36Z Stashbot 7414 hashar@deploy1003: Started deploy [gerrit/gerrit@83b886a]: wm-checks-api: add tag for PostgreSQL jobs 2407892 wikitext text/x-wiki 06rkd62rgi3exwo3dngetfqpa3q0f8l 2407893 2407892 2026-04-30T08:27:50Z Stashbot 7414 hashar@deploy1003: Finished deploy [gerrit/gerrit@83b886a]: wm-checks-api: add tag for PostgreSQL jobs (duration: 00m 14s) 2407893 wikitext text/x-wiki 0n0ca76zmw5k38psxs6wifvabcvtg7b 2407894 2407893 2026-04-30T08:29:38Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on hcaptcha-proxy5003.wikimedia.org with reason: host reimage 2407894 wikitext text/x-wiki gzj082yxrrttgdcmyxmdhhql21sla8e 2407895 2407894 2026-04-30T08:33:15Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T419961)', diff saved to https://phabricator.wikimedia.org/P92012 and previous config saved to /var/cache/conftool/dbconfig/20260430-083313-fceratto.json 2407895 wikitext text/x-wiki 560521i7jk0n52lhuewezk6pb36nizi 2407896 2407895 2026-04-30T08:43:23Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P92013 and previous config saved to /var/cache/conftool/dbconfig/20260430-084321-fceratto.json 2407896 wikitext text/x-wiki ka68023u9adga0isj5zaf8myx72r32c 2407897 2407896 2026-04-30T08:48:43Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host hcaptcha-proxy5003.wikimedia.org with OS bookworm 2407897 wikitext text/x-wiki sijw4w20n5yy2or0x5swf4lq43bwt7v 2407898 2407897 2026-04-30T08:48:47Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host hcaptcha-proxy5003.wikimedia.org 2407898 wikitext text/x-wiki mk1m4j4eq0801i31ssur99mgkg4wwsu 2407899 2407898 2026-04-30T08:49:17Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2407899 wikitext text/x-wiki lza9w9eqx3ybyy0n47eaczkjswo98wz 2407900 2407899 2026-04-30T08:50:36Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host hcaptcha-proxy5004.wikimedia.org 2407900 wikitext text/x-wiki hh3e3ujnzqesugiggdy4z87274psqah 2407901 2407900 2026-04-30T08:50:42Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host hcaptcha-proxy5004.wikimedia.org 2407901 wikitext text/x-wiki gwu8lmgcs6etx7wfzr1udwdfmx3h46x 2407902 2407901 2026-04-30T08:51:03Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host hcaptcha-proxy5004.wikimedia.org 2407902 wikitext text/x-wiki 0t82hxekuqrf97lysg9fnsq6zgie1jh 2407903 2407902 2026-04-30T08:51:06Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2407903 wikitext text/x-wiki pb3425mmn49p4b2jtgoj6ihvhpp0t0z 2407904 2407903 2026-04-30T08:53:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P92014 and previous config saved to /var/cache/conftool/dbconfig/20260430-085329-fceratto.json 2407904 wikitext text/x-wiki jw4t19so3jm5qa39usllpr9tfdeadwo 2407905 2407904 2026-04-30T08:54:16Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2407905 wikitext text/x-wiki q9peknzgeokonndd68e4dxycu1cnpz5 2407906 2407905 2026-04-30T08:54:26Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2407906 wikitext text/x-wiki 7on0jatdomi4ap5z9s3rvpxe76x8oph 2407907 2407906 2026-04-30T08:54:37Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2407907 wikitext text/x-wiki jfre4kcoxaddqne1idx22upps0b36fo 2407908 2407907 2026-04-30T08:54:50Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM hcaptcha-proxy5004.wikimedia.org - jmm@cumin2002" 2407908 wikitext text/x-wiki rfhv17ut57f5imcmft6cum0i5gdu90e 2407909 2407908 2026-04-30T08:55:05Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM hcaptcha-proxy5004.wikimedia.org - jmm@cumin2002" 2407909 wikitext text/x-wiki fewu4out979ifvnxas7ue1i306l3ldx 2407910 2407909 2026-04-30T08:55:08Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2407910 wikitext text/x-wiki pupj3wa0d1eelxl8ezrhjmzbmuf6oc0 2407911 2407910 2026-04-30T08:55:10Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache hcaptcha-proxy5004.wikimedia.org on all recursors 2407911 wikitext text/x-wiki eaejd609xxopglw6k6migiu3js93g8m 2407912 2407911 2026-04-30T08:55:13Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) hcaptcha-proxy5004.wikimedia.org on all recursors 2407912 wikitext text/x-wiki odsp80hktp0juf468w2ocybu74nwod0 2407913 2407912 2026-04-30T08:55:45Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM hcaptcha-proxy5004.wikimedia.org - jmm@cumin2002" 2407913 wikitext text/x-wiki 1sowe6e547kac8nbmti7e8vqoukra22 2407914 2407913 2026-04-30T08:55:51Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM hcaptcha-proxy5004.wikimedia.org - jmm@cumin2002" 2407914 wikitext text/x-wiki ggbsa9g8s5a08m8a97vm4dl5v1ser34 2407915 2407914 2026-04-30T08:56:45Z Stashbot 7414 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply 2407915 wikitext text/x-wiki 7y5hjsjftxjv64z2bftfv4isoqcg7tp 2407916 2407915 2026-04-30T08:56:59Z Stashbot 7414 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply 2407916 wikitext text/x-wiki f5oc5pqtlu2qcjl7m6n2c3zoxw2xsx3 2407917 2407916 2026-04-30T08:57:39Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host hcaptcha-proxy5004.wikimedia.org with OS bookworm 2407917 wikitext text/x-wiki ano30jnrjo83jllybwt2n2d1q857jut 2407918 2407917 2026-04-30T09:01:22Z Stashbot 7414 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply 2407918 wikitext text/x-wiki lmt8p6dqth31j8271bcv2eruu6lphdw 2407919 2407918 2026-04-30T09:01:28Z Stashbot 7414 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply 2407919 wikitext text/x-wiki fhr46tahldivjno7fs9wqv4t1makczf 2407920 2407919 2026-04-30T09:03:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T419961)', diff saved to https://phabricator.wikimedia.org/P92015 and previous config saved to /var/cache/conftool/dbconfig/20260430-090337-fceratto.json 2407920 wikitext text/x-wiki e0fxu9c34k8yqlt6m6tlj48qhv5pnwp 2407921 2407920 2026-04-30T09:04:02Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance 2407921 wikitext text/x-wiki 6v66cs4gr8qk3ky0fmggxiuiyj6wn52 2407922 2407921 2026-04-30T09:04:10Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2173 (T419961)', diff saved to https://phabricator.wikimedia.org/P92016 and previous config saved to /var/cache/conftool/dbconfig/20260430-090408-fceratto.json 2407922 wikitext text/x-wiki 7368ixvfxp317ji2pdc1zfupra2pj1d 2407924 2407922 2026-04-30T09:11:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T419961)', diff saved to https://phabricator.wikimedia.org/P92017 and previous config saved to /var/cache/conftool/dbconfig/20260430-091147-fceratto.json 2407924 wikitext text/x-wiki 0pfbk0hmsk5e53izxfecp1tbwteh21x 2407925 2407924 2026-04-30T09:12:58Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-jumbo-eqiad 2407925 wikitext text/x-wiki ovey8cdgeu5ouu4a8aiwtcqok9tk0mm 2407926 2407925 2026-04-30T09:15:21Z Stashbot 7414 brouberol@cumin1003: END (ERROR) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=97) rolling restart_daemons on A:kafka-jumbo-eqiad 2407926 wikitext text/x-wiki 8dz0nstydgp0r4z5e7pc7seczk8ue96 2407927 2407926 2026-04-30T09:17:10Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4006.ulsfo.wmnet 2407927 wikitext text/x-wiki i8o2no18uazw6n5ib8468dsd0k7vrjq 2407928 2407927 2026-04-30T09:21:57Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P92018 and previous config saved to /var/cache/conftool/dbconfig/20260430-092154-fceratto.json 2407928 wikitext text/x-wiki nziv89n8srfotaye7fn0yl51z6utitu 2407930 2407928 2026-04-30T09:24:31Z Stashbot 7414 moritzm: temporarily remove ganeti4006 from the ulsfo02 Ganeti cluster in preparation of forthcoming switch maintenance in ulsfo T424686 2407930 wikitext text/x-wiki m2s8hknifc9m8jy5afd2xy9foneds5w 2407931 2407930 2026-04-30T09:27:24Z Stashbot 7414 moritzm: failover Ganeti master in ulsfo02 to ganeti4005 in preparation of forthcoming switch maintenance in ulsfo T424686 2407931 wikitext text/x-wiki bc2vvi7e3k5x4vem19jrxvxrti049t4 2407932 2407931 2026-04-30T09:32:05Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P92019 and previous config saved to /var/cache/conftool/dbconfig/20260430-093202-fceratto.json 2407932 wikitext text/x-wiki g5oywt12z1x14uz07xql1d0m1119bsc 2407933 2407932 2026-04-30T09:42:12Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T419961)', diff saved to https://phabricator.wikimedia.org/P92020 and previous config saved to /var/cache/conftool/dbconfig/20260430-094210-fceratto.json 2407933 wikitext text/x-wiki tt2xz5btcullycanz5vfjrl53meag38 2407934 2407933 2026-04-30T09:42:34Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance 2407934 wikitext text/x-wiki fcf6nh857zxb8i899d6ihdmg0embe9v 2407935 2407934 2026-04-30T09:42:42Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2174 (T419961)', diff saved to https://phabricator.wikimedia.org/P92021 and previous config saved to /var/cache/conftool/dbconfig/20260430-094239-fceratto.json 2407935 wikitext text/x-wiki dxdkn80s4g5tyehlcbs8oazsvd491vi 2407936 2407935 2026-04-30T09:50:04Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T419961)', diff saved to https://phabricator.wikimedia.org/P92022 and previous config saved to /var/cache/conftool/dbconfig/20260430-095000-fceratto.json 2407936 wikitext text/x-wiki ow8iquy4764kvc4pb2xl01gctl9o7dz 2407937 2407936 2026-04-30T09:50:23Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on hcaptcha-proxy5004.wikimedia.org with reason: host reimage 2407937 wikitext text/x-wiki ag6sj3wma2ftobrcfd6t6taztm6rukn 2407938 2407937 2026-04-30T09:54:34Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on hcaptcha-proxy5004.wikimedia.org with reason: host reimage 2407938 wikitext text/x-wiki e4fdo1sktyps05hfx5a5fcmbu58xrdr 2407939 2407938 2026-04-30T10:00:12Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P92023 and previous config saved to /var/cache/conftool/dbconfig/20260430-100009-fceratto.json 2407939 wikitext text/x-wiki 0iaxokxoyy0vl1evx7jvv3nrwvgxdjx 2407940 2407939 2026-04-30T10:10:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P92024 and previous config saved to /var/cache/conftool/dbconfig/20260430-101017-fceratto.json 2407940 wikitext text/x-wiki r8fk529xjl34tgwhzfd26qg4omqivci 2407941 2407940 2026-04-30T10:14:40Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host hcaptcha-proxy5004.wikimedia.org with OS bookworm 2407941 wikitext text/x-wiki 36275s1y6avie9897n8718rj44zc9xu 2407942 2407941 2026-04-30T10:14:43Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host hcaptcha-proxy5004.wikimedia.org 2407942 wikitext text/x-wiki 5ugn1i4i69rbna9qkw90tadn8r6pwuo 2407943 2407942 2026-04-30T10:20:28Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T419961)', diff saved to https://phabricator.wikimedia.org/P92025 and previous config saved to /var/cache/conftool/dbconfig/20260430-102026-fceratto.json 2407943 wikitext text/x-wiki fun6zxbdhebprbzvia0l8nddq5u2ufm 2407944 2407943 2026-04-30T10:20:50Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2176.codfw.wmnet with reason: Maintenance 2407944 wikitext text/x-wiki q4txbqpngi4qmf34im4jic9eclt0a4h 2407945 2407944 2026-04-30T10:20:57Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2176 (T419961)', diff saved to https://phabricator.wikimedia.org/P92026 and previous config saved to /var/cache/conftool/dbconfig/20260430-102055-fceratto.json 2407945 wikitext text/x-wiki lrhqd5lccdkizhkzkk924hh462k1z9m 2407946 2407945 2026-04-30T10:28:38Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T419961)', diff saved to https://phabricator.wikimedia.org/P92027 and previous config saved to /var/cache/conftool/dbconfig/20260430-102830-fceratto.json 2407946 wikitext text/x-wiki 52g07wf696z9a8cda5y18rki9miusfj 2407947 2407946 2026-04-30T10:35:02Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4008.ulsfo.wmnet 2407947 wikitext text/x-wiki 94uu1q2k7waeh3xqcxl64kfe8i6n7o7 2407948 2407947 2026-04-30T10:36:03Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4008.ulsfo.wmnet 2407948 wikitext text/x-wiki 261hdeh7tj1ke0gxyvw6t9tisfmhbqi 2407949 2407948 2026-04-30T10:36:27Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host bast5005.wikimedia.org 2407949 wikitext text/x-wiki 8g5pqhk1xse0xgrmhytpfvybutedv83 2407950 2407949 2026-04-30T10:36:29Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2407950 wikitext text/x-wiki f3cjhcbz2xqhz5f6tt8an3tgfg39poh 2407951 2407950 2026-04-30T10:38:41Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P92028 and previous config saved to /var/cache/conftool/dbconfig/20260430-103838-fceratto.json 2407951 wikitext text/x-wiki r86y9au86lpsm9acgu5f5sw8enlfbvl 2407952 2407951 2026-04-30T10:39:29Z Stashbot 7414 phuedx: Clearing stuck Test Kitchen experiment configs value from codfw local cluster cache 2407952 wikitext text/x-wiki oqswhypmkjkfynraqgqt3qobcru78bi 2407953 2407952 2026-04-30T10:40:26Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM bast5005.wikimedia.org - jmm@cumin2002" 2407953 wikitext text/x-wiki emevxoftyejjaavr9tgm64ra4vrd1jk 2407954 2407953 2026-04-30T10:40:44Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM bast5005.wikimedia.org - jmm@cumin2002" 2407954 wikitext text/x-wiki c3xiibosqk064o2xgoq5zaxlb41pdvf 2407955 2407954 2026-04-30T10:40:47Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2407955 wikitext text/x-wiki l7vy6mzl3dn22i5jyqjyy5ehalwvq0g 2407956 2407955 2026-04-30T10:40:49Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache bast5005.wikimedia.org on all recursors 2407956 wikitext text/x-wiki av947m64b0jyvawlz7g7p0m28kmn7yf 2407957 2407956 2026-04-30T10:40:52Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) bast5005.wikimedia.org on all recursors 2407957 wikitext text/x-wiki 4v9reb2z6mqn2soa6ft122648dfu8te 2407958 2407957 2026-04-30T10:41:25Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM bast5005.wikimedia.org - jmm@cumin2002" 2407958 wikitext text/x-wiki mbdckl7iuw30evotj1cboeeod934n3n 2407959 2407958 2026-04-30T10:41:30Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM bast5005.wikimedia.org - jmm@cumin2002" 2407959 wikitext text/x-wiki mza4ad30fgn5r10rm9gbtcffamk8gk7 2407960 2407959 2026-04-30T10:41:48Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host bast5005.wikimedia.org with OS trixie 2407960 wikitext text/x-wiki lao683g53gjcvlngutcxnig27e7x3d9 2407961 2407960 2026-04-30T10:41:55Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4008.ulsfo.wmnet 2407961 wikitext text/x-wiki ka9h610bng3h2u6yjunkoccbpzuybsz 2407962 2407961 2026-04-30T10:42:02Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4008.ulsfo.wmnet 2407962 wikitext text/x-wiki 6gpfwvu86wlzlcz7pdy9bymxo0qnm0y 2407964 2407962 2026-04-30T10:46:56Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2205.codfw.wmnet with reason: Reimage to Trixie 2407964 wikitext text/x-wiki jzliuvsjfau7w866iif4ifnn635rl06 2407965 2407964 2026-04-30T10:47:02Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2205: Reimage to Trixie 2407965 wikitext text/x-wiki ij096yhwc34ydkqfjb4h0uq11gmee5m 2407966 2407965 2026-04-30T10:47:20Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2205: Reimage to Trixie 2407966 wikitext text/x-wiki jwgt3dm7jh8jw3v1gg18nujim5mad19 2407967 2407966 2026-04-30T10:48:49Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P92030 and previous config saved to /var/cache/conftool/dbconfig/20260430-104846-fceratto.json 2407967 wikitext text/x-wiki 6qrk6q0pedquf94jpwf8ez3lbz1tv3w 2407968 2407967 2026-04-30T10:48:56Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2205.codfw.wmnet with OS trixie 2407968 wikitext text/x-wiki abblwjtsx8bawezczv1pwk2deisulgu 2407969 2407968 2026-04-30T10:58:57Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T419961)', diff saved to https://phabricator.wikimedia.org/P92031 and previous config saved to /var/cache/conftool/dbconfig/20260430-105854-fceratto.json 2407969 wikitext text/x-wiki 0t5pm3ep2rodphq48ilafltzt5k2z5w 2407970 2407969 2026-04-30T10:59:18Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2188.codfw.wmnet with reason: Maintenance 2407970 wikitext text/x-wiki a7gej00ho97mkbjrbj4q9hvu1vqr8ih 2407971 2407970 2026-04-30T10:59:27Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2188 (T419961)', diff saved to https://phabricator.wikimedia.org/P92032 and previous config saved to /var/cache/conftool/dbconfig/20260430-105924-fceratto.json 2407971 wikitext text/x-wiki mx6t6hlf1lk87pg6mh6yv002vb8imi1 2407972 2407971 2026-04-30T10:59:57Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2407972 wikitext text/x-wiki tup4p350l47mpi7vb0xbiwve2vdhas4 2407973 2407972 2026-04-30T11:00:52Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2407973 wikitext text/x-wiki jgjfn3kis2x38mojm7og63opppbsvmb 2407974 2407973 2026-04-30T11:01:18Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2407974 wikitext text/x-wiki hel226ge06yfaxfh6pxokpzunsclwlh 2407975 2407974 2026-04-30T11:02:12Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2407975 wikitext text/x-wiki 6y70um4s1llsgthjgdd5b23eobqjp2j 2407976 2407975 2026-04-30T11:06:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T419961)', diff saved to https://phabricator.wikimedia.org/P92033 and previous config saved to /var/cache/conftool/dbconfig/20260430-110640-fceratto.json 2407976 wikitext text/x-wiki pp87yf21nv77lwyq25bkw4kw2kqtvdr 2407977 2407976 2026-04-30T11:06:59Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2205.codfw.wmnet with reason: host reimage 2407977 wikitext text/x-wiki ksnwape9wdjxtmvgbuuy08z7q04ph09 2407978 2407977 2026-04-30T11:10:29Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2205.codfw.wmnet with reason: host reimage 2407978 wikitext text/x-wiki howfihj96nsgm8d447ewshoq3igbqir 2407980 2407978 2026-04-30T11:16:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P92034 and previous config saved to /var/cache/conftool/dbconfig/20260430-111648-fceratto.json 2407980 wikitext text/x-wiki cg8l8gurmrlpyoa0saplq5wthh8gxsm 2407981 2407980 2026-04-30T11:19:24Z Stashbot 7414 elukey: upgrade spicerack on cumin hosts to 12.5.0 2407981 wikitext text/x-wiki 1v2ns3esgtt5dd0p2h6tpu1o5u25dzd 2407982 2407981 2026-04-30T11:20:54Z Stashbot 7414 moritzm: installing policykit-1 security updates 2407982 wikitext text/x-wiki tg3qqqzfn3k2c4y12o25pzk3axe5d25 2407985 2407982 2026-04-30T11:26:58Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P92035 and previous config saved to /var/cache/conftool/dbconfig/20260430-112656-fceratto.json 2407985 wikitext text/x-wiki gyksi1qqd9g5tbbnprbfquqmhw72ypj 2407986 2407985 2026-04-30T11:27:09Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1039.eqiad.wmnet 2407986 wikitext text/x-wiki 1f2cm05nrsv6uhrleqtxsit249dt4mx 2407987 2407986 2026-04-30T11:27:12Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1039.eqiad.wmnet 2407987 wikitext text/x-wiki 1s6xjkkpwc4s35hcqeerurjfc7fzs6s 2407988 2407987 2026-04-30T11:27:32Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.remove-downtime for wikikube-worker1039.eqiad.wmnet 2407988 wikitext text/x-wiki 9yvtjs1v213yy8xxkk4u2h02rtac68z 2407989 2407988 2026-04-30T11:27:35Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker1039.eqiad.wmnet 2407989 wikitext text/x-wiki 5u6morynrmfjwi5x020q4p4fogbr6uy 2407990 2407989 2026-04-30T11:27:49Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407990 wikitext text/x-wiki 7ucnkef9w1qm4zv5ct196dr28m44zh0 2407991 2407990 2026-04-30T11:28:54Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407991 wikitext text/x-wiki 98d9zrsdd5o320ty1bihssbntmq9cyc 2407992 2407991 2026-04-30T11:33:30Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2205.codfw.wmnet with OS trixie 2407992 wikitext text/x-wiki ly0kq0y1z9fnjsw8a4ju4pkcc4hkg0m 2407993 2407992 2026-04-30T11:35:31Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1377.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407993 wikitext text/x-wiki 50lvivba1wo1zqxryb2e2zrfykxfo9g 2407994 2407993 2026-04-30T11:36:00Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407994 wikitext text/x-wiki 4xf46yx5aytj8xw57t2y80mxevxcs19 2407995 2407994 2026-04-30T11:36:16Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407995 wikitext text/x-wiki ebao1jkoznc5npebsr5nn1rhcl9wkzb 2407996 2407995 2026-04-30T11:37:07Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T419961)', diff saved to https://phabricator.wikimedia.org/P92036 and previous config saved to /var/cache/conftool/dbconfig/20260430-113704-fceratto.json 2407996 wikitext text/x-wiki iq3kozrychjyk8c37m9d8wkcag82622 2407997 2407996 2026-04-30T11:38:29Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1378.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2407997 wikitext text/x-wiki 4vfqr3tivk9kcgu83ill2q5vcq93k9x 2407998 2407997 2026-04-30T11:39:05Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1159.eqiad.wmnet with reason: Maintenance 2407998 wikitext text/x-wiki ra44q0i0vp2jc7fms568rnc784r0c8f 2407999 2407998 2026-04-30T11:39:08Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2205: after reimage to trixie 2407999 wikitext text/x-wiki ri2cbfvsaull2oxc6l2q5sbs159a3az 2408000 2407999 2026-04-30T11:39:14Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1159 (T419635)', diff saved to https://phabricator.wikimedia.org/P92038 and previous config saved to /var/cache/conftool/dbconfig/20260430-113910-fceratto.json 2408000 wikitext text/x-wiki 4ro0ijcbydz4g0j0uioli1mwt1zh1oz 2408001 2408000 2026-04-30T11:39:43Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2203.codfw.wmnet with reason: Maintenance 2408001 wikitext text/x-wiki 6e1d9o6ahkz14tqid60fwuwx57zpj4v 2408002 2408001 2026-04-30T11:39:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2203 (T419961)', diff saved to https://phabricator.wikimedia.org/P92039 and previous config saved to /var/cache/conftool/dbconfig/20260430-113948-fceratto.json 2408002 wikitext text/x-wiki ilsfgpmgmclu9tknsngy0a22lg3er26 2408003 2408002 2026-04-30T11:40:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T419635)', diff saved to https://phabricator.wikimedia.org/P92040 and previous config saved to /var/cache/conftool/dbconfig/20260430-114020-fceratto.json 2408003 wikitext text/x-wiki jlc3kh29oz8x75ogb2zablw457g38dr 2408004 2408003 2026-04-30T11:40:39Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on bast5005.wikimedia.org with reason: host reimage 2408004 wikitext text/x-wiki ar0f1qf6jgx7jl37c896k28mdrh3ie3 2408006 2408004 2026-04-30T11:42:16Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1377.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2408006 wikitext text/x-wiki 515ktxqzl6fzc2i4q09r5hfgbozsc8t 2408007 2408006 2026-04-30T11:44:52Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1377.eqiad.wmnet with OS trixie 2408007 wikitext text/x-wiki 1vlmgq5q8vhkb7i4u9wr5kuxo47k3kr 2408008 2408007 2026-04-30T11:45:01Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on bast5005.wikimedia.org with reason: host reimage 2408008 wikitext text/x-wiki n8nw991t3dhsje5cy9eaczti66u273r 2408009 2408008 2026-04-30T11:46:05Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1378.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2408009 wikitext text/x-wiki 3f3r5b54yk81hnljffirq1ev8ed932s 2408010 2408009 2026-04-30T11:46:27Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1378.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2408010 wikitext text/x-wiki 6ymwi1znv8obbkzjqmmsdf0q04vvdlo 2408011 2408010 2026-04-30T11:47:07Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2203 (T419961)', diff saved to https://phabricator.wikimedia.org/P92041 and previous config saved to /var/cache/conftool/dbconfig/20260430-114703-fceratto.json 2408011 wikitext text/x-wiki 038pe7bg81vr7ybtukg2irhktg5sbwj 2408012 2408011 2026-04-30T11:47:12Z Stashbot 7414 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@fb711fc] (releasing): Update backup releases Jenkins 2408012 wikitext text/x-wiki mlz3kk8x526impj4efjoyuwi95saws7 2408013 2408012 2026-04-30T11:47:35Z Stashbot 7414 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@fb711fc] (releasing): Update backup releases Jenkins (duration: 00m 33s) 2408013 wikitext text/x-wiki orgykay7e1hfat0ldoxijblx16n4e2l 2408014 2408013 2026-04-30T11:49:35Z Stashbot 7414 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@fb711fc] (releasing): Update production releases Jenkins 2408014 wikitext text/x-wiki 5dzqmiqvcf6tlsq4gzi5katt8ygv0y7 2408015 2408014 2026-04-30T11:50:24Z Stashbot 7414 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@fb711fc] (releasing): Update production releases Jenkins (duration: 01m 04s) 2408015 wikitext text/x-wiki j7xrroua6vv08452u0vqrcsw987xgu6 2408016 2408015 2026-04-30T11:50:30Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P92042 and previous config saved to /var/cache/conftool/dbconfig/20260430-115028-fceratto.json 2408016 wikitext text/x-wiki pmbfpy77f5jjknkdj0997j7awewyvlv 2408019 2408016 2026-04-30T11:56:45Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1377.eqiad.wmnet with reason: host reimage 2408019 wikitext text/x-wiki 10n1ayqxyxqlq82c9uy3w5mpdp6xgfu 2408020 2408019 2026-04-30T11:57:14Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2203', diff saved to https://phabricator.wikimedia.org/P92044 and previous config saved to /var/cache/conftool/dbconfig/20260430-115712-fceratto.json 2408020 wikitext text/x-wiki nijk3ehyzzab6jjbay1hkkkblawpq8p 498 14256 2407979 2394147 2026-04-30T11:12:09Z Stashbot 7414 wikibugs: Updated channels.yaml to: 75b8ede37a2b00f4a6d1b230320b983b89391f64f3f30d87d57519c5d314daf0 2407979 wikitext text/x-wiki aqijub7uueitufg8tfngdk9bx73jyux 0 17290 2407590 2407024 2026-04-29T15:28:48Z Stashbot 7414 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1279392, https://gerrit.wikimedia.org/r/1279397 2407590 wikitext text/x-wiki n5ncyz0peo9bp237641hp8d7xmxhvt6 2407592 2407590 2026-04-29T15:36:53Z Stashbot 7414 James_F: Zuul: Drop experimental node22 jobs, never used in practice 2407592 wikitext text/x-wiki m5mgxfxj6luzldcg7dtgt2d47m5coeq 2407598 2407592 2026-04-29T15:49:21Z Stashbot 7414 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Add ConfirmEdit dependency, for T424597 2407598 wikitext text/x-wiki tr9nzx66mlgv36hmobudy4bznv0veqc 498 30942 2407603 2406933 2026-04-29T16:05:52Z Stashbot 7414 andrewbogott: cleaning up stray broken osbpo references on VMs, e.g. /etc/apt/sources.list.d/openstack-dalmatian-bookworm.sources 2407603 wikitext text/x-wiki 2b2f9ru2fhro56cbntesiup37w7cnkb 0 442598 2407727 2401124 2026-04-29T21:48:38Z LErickson-WMF 51715 /* Categories update lag */ 2407727 wikitext text/x-wiki b9l8iu4lwttzdv3mfnku29dexzh62mp 2407728 2407727 2026-04-29T21:49:05Z LErickson-WMF 51715 /* Free allocators decrease rapidly */ 2407728 wikitext text/x-wiki nmjf8enplc085oysnuobbfmatu90ewl 13 448634 2407599 2390730 2026-04-29T15:49:42Z Dw31415 50147 /* Customer script vs library */ new section 2407599 wikitext text/x-wiki gmxkkcvelr2tmktl7o3gc9jc0vh8kut 0 449160 2407503 2407227 2026-04-29T14:22:53Z Dexbot 30554 Bot: Updating the report 2407503 wikitext text/x-wiki 1ru9oxnolnpsyu6qrzrkqq6znjnuoii 2407777 2407503 2026-04-30T00:00:26Z Dexbot 30554 Bot: Updating the report 2407777 wikitext text/x-wiki qfd4b6wblufr232r22ff3wd83341pkw 2407873 2407777 2026-04-30T07:49:38Z Dexbot 30554 Bot: Updating the report 2407873 wikitext text/x-wiki 9u9293bwfzqsyrpxpooqbqfhd9s7o5e 2408005 2407873 2026-04-30T11:41:16Z Dexbot 30554 Bot: Updating the report 2408005 wikitext text/x-wiki a8clhxke14xumcao4xwai0uq8paizhz 0 451850 2407740 2365205 2026-04-29T23:07:44Z Abhipoojarybannur 55155 Undid revision [[Special:Diff/2201839|2201839]] by [[Special:Contributions/TBurmeister (WMF)|TBurmeister (WMF)]] ([[User talk:TBurmeister (WMF)|talk]]) 2407740 wikitext text/x-wiki 3a0ko82s3powoamcwode87h1khg4407 2407827 2407740 2026-04-30T04:59:10Z JJMC89 7474 Reverted edit by [[Special:Contributions/Abhipoojarybannur|Abhipoojarybannur]] ([[User talk:Abhipoojarybannur|talk]]) to last revision by [[User:JJMC89|JJMC89]] 2201977 wikitext text/x-wiki bnrgac5bevuojq7g6mbwn3ikjot93yo 0 452746 2407582 2349961 2026-04-29T15:14:18Z JMonton-WMF 47887 link to the html enrichment pipeline 2407582 wikitext text/x-wiki 9l5u9zd6qrg5kj4erbt5q8f96devdo6 116 453426 2407709 2405016 2026-04-29T20:37:08Z Phabbanbot 37210 Hr574380 was disabled by JJMC89 2407709 wikitext text/x-wiki eab6thn2jy76bpj9wmd6zf0j3o4ikdj 2407733 2407709 2026-04-29T22:31:42Z Phabbanbot 37210 jhsoby-WMNO was disabled by Zabe 2407733 wikitext text/x-wiki mrvakqnwelifwi75ejtoqu2r7p9s97z 116 453906 2407851 2407083 2026-04-30T06:36:35Z Gitlabaccountapprovalbot 37332 @leo321 was approved. 2407851 wikitext text/x-wiki a3k3qpgmf6r9nnb8lyakkce6s261o3b 2407868 2407851 2026-04-30T07:39:26Z Gitlabaccountapprovalbot 37332 @mahveotm was approved. 2407868 wikitext text/x-wiki qvumlpp529v2m2mqvln8aaelzq10ynb 0 455928 2408017 2386297 2026-04-30T11:52:04Z SFaci-WMF 35637 /* Experiment#setStream */ 2408017 wikitext text/x-wiki i8naan2sl4xnezqodyqh58zos2jk7sy 0 455930 2408018 2396479 2026-04-30T11:53:01Z SFaci-WMF 35637 /* Experiment#setStream */ 2408018 wikitext text/x-wiki 1jghdf7aza6kjo35kolfq8couopom87 498 458787 2407768 2405098 2026-04-29T23:50:08Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25139750319 (https://github.com/cluebotng/component-configs/commits/9a101172ba64f47f96b75a6a6d77f65ee589ab4e) 2407768 wikitext text/x-wiki 6aw3nhh4vlsxnxym40mjmdfhurp65kz 0 459044 2407569 2360983 2026-04-29T15:00:21Z BKing (WMF) 29017 /* Operations */ 2407569 wikitext text/x-wiki bnpza8e430a3nh6a79ztdhzd7docoqx 0 459290 2407452 2407074 2026-04-29T13:46:01Z APizzata-WMF 47941 deployment done 2407452 wikitext text/x-wiki 1a8bcoipti4b3w3iftk10fwpvxxl1fe 2407470 2407452 2026-04-29T13:55:12Z XCollazo-WMF 31846 /* NEXT TRAIN */ 2407470 wikitext text/x-wiki mjsgwi479gmpchig9js1w7s1cxqdv8h 2407601 2407470 2026-04-29T16:02:24Z XCollazo-WMF 31846 /* NEXT TRAIN */ 2407601 wikitext text/x-wiki ht5pm4si5ere4w1vfmpcdndxxapsn23 2407719 2407601 2026-04-29T21:01:09Z XCollazo-WMF 31846 /* NEXT TRAIN */ 2407719 wikitext text/x-wiki 8qo9ckyefye6j6tu3ly7ewwewhzswp3 0 459342 2407393 2404877 2026-04-29T12:28:20Z GLederrey (WMF) 6860 Clarify SLO section 2407393 wikitext text/x-wiki a9k12etg5mipb4mr9wfdtjptqkrbri0 2407624 2407393 2026-04-29T16:37:26Z BTullis (WMF) 25295 Update the SLI/SLO sections 2407624 wikitext text/x-wiki nhr2o3symcgkugmyxnv8z4mpghwf6ow 0 459699 2407463 2407057 2026-04-29T13:49:58Z Stashbot 7414 jgleeson: tools upgraded from f52a5dcf to afbd0f67 2407463 wikitext text/x-wiki fj09kdpvt5kr08gx02rmha710nq6cel 2407465 2407463 2026-04-29T13:51:09Z Stashbot 7414 jgleeson: payments-wiki upgraded from 2e2eb8a2 to 4e0c944b 2407465 wikitext text/x-wiki a2a7ymaj1ocm5602hoqb5xk6ts7rw0w 0 459719 2407612 2406996 2026-04-29T16:21:26Z BTullis (WMF) 25295 /* Realistic targets */ 2407612 wikitext text/x-wiki 73t95gqxko3d7s5kwpa0ow4d1npos5e 2407614 2407612 2026-04-29T16:21:38Z BTullis (WMF) 25295 /* Ideal targets */ 2407614 wikitext text/x-wiki 60pc74x8jpixdijrqqfzee7xrujurdn 0 459743 2407701 2404367 2026-04-29T20:16:29Z LErickson-WMF 51715 /* Runbooks */ 2407701 wikitext text/x-wiki h1qllxmwnko9gptvz47thidyepm3atp 2407710 2407701 2026-04-29T20:38:13Z LErickson-WMF 51715 /* Runbooks */ 2407710 wikitext text/x-wiki i4eonm6z6e9b1pse4nyruckpa1iz9ah 2407711 2407710 2026-04-29T20:38:44Z LErickson-WMF 51715 /* Runbooks */ 2407711 wikitext text/x-wiki 0y7qvabfbv33j882iagis5vty6e4hey 2407712 2407711 2026-04-29T20:39:22Z LErickson-WMF 51715 /* Runbooks */ 2407712 wikitext text/x-wiki j2bkzamorn5klx638ll4rw42aii83c6 0 459777 2407630 2385045 2026-04-29T16:42:18Z GGoncalves-WMF 42848 Add a simplified diagram. 2407630 wikitext text/x-wiki ojurwuh6lc1g8tygh7ggw3brsgepj73 2407640 2407630 2026-04-29T16:44:01Z GGoncalves-WMF 42848 Replace nutshell template with an actual section. 2407640 wikitext text/x-wiki tsii08d30p2tm2euzcqsk4hu8jdla89 0 459783 2407963 2400011 2026-04-30T10:44:36Z Duesentrieb 52342 Link internal documents 2407963 wikitext text/x-wiki 3rukph64i99niynvvz3hz0dbkjpmo0l 0 460018 2407461 2401125 2026-04-29T13:49:25Z LErickson-WMF 51715 2407461 wikitext text/x-wiki ar9vs3tq1j1937n0tikke6lfd3clsoi 0 460088 2407579 2405632 2026-04-29T15:11:24Z JMonton-WMF 47887 fix alert name 2407579 wikitext text/x-wiki m1p97pxcsh0gw6052v1hufegt5llhco 2407604 2407579 2026-04-29T16:10:52Z JMonton-WMF 47887 info about backfilling 2407604 wikitext text/x-wiki 9z2vpaaqlweu7zjjzxfot4mlleaj6kq 0 460089 2407722 2405893 2026-04-29T21:28:11Z EEvans (WMF) 5035 an attempt at better intro text (more work needed though) 2407722 wikitext text/x-wiki g239z6oklvvkeoy1gft7zw60ou9s1ub 2407724 2407722 2026-04-29T21:30:27Z EEvans (WMF) 5035 fixup links 2407724 wikitext text/x-wiki siaj1lza9ifgkpvybv8l4ngeg20y42a 2407725 2407724 2026-04-29T21:31:28Z EEvans (WMF) 5035 nit 2407725 wikitext text/x-wiki 2oc129nmpqjrw1xvklt28t7rz0ou5ig 2407726 2407725 2026-04-29T21:34:42Z EEvans (WMF) 5035 reorganize intro/background text 2407726 wikitext text/x-wiki 6yy2806x978z522llibghsrrj8mk5yv 2 460108 2407844 2407142 2026-04-30T05:46:28Z SSapaty (WMF) 28955 /* Steps */ 2407844 wikitext text/x-wiki no877016xsvziq7vf4lun2hfahb26ul 2407847 2407844 2026-04-30T05:48:16Z SSapaty (WMF) 28955 /* Announcement template */ 2407847 wikitext text/x-wiki 965i6awvd7wo6g8jvdos3o3du4cqre9 2407848 2407847 2026-04-30T05:49:39Z SSapaty (WMF) 28955 /* Reminder template */ 2407848 wikitext text/x-wiki f7sfcqla5puija0bvx4s6l7u2m5pp6r 2407849 2407848 2026-04-30T05:51:52Z SSapaty (WMF) 28955 /* Bulk emailing instructions */ 2407849 wikitext text/x-wiki rmec0ps8513iy6u8s50qdkyevxoomie 2 460109 2407433 2026-04-29T13:24:32Z Atsuko-WMF 54082 notes 2407433 wikitext text/x-wiki tv82jnhfj9iwffbdjkn648256pt3bhx 2407586 2407433 2026-04-29T15:19:34Z Atsuko-WMF 54082 2407586 wikitext text/x-wiki huw7h3z2qpmuaak4gk4g3iio4jfc4nj 3 460110 2407676 2026-04-29T19:11:41Z StrikerBot 8475 Welcome to Toolforge! 2407676 wikitext text/x-wiki lge383qlzlw9jmg13o68y18kztg37i7 3 460111 2407681 2026-04-29T19:16:54Z StrikerBot 8475 Welcome to Toolforge! 2407681 wikitext text/x-wiki jx0x2obeaat5h75ep44is3m6qlxevgv 3 460112 2407723 2026-04-29T21:29:03Z StrikerBot 8475 Welcome to Toolforge! 2407723 wikitext text/x-wiki 9qh6alskainize73emyem458s2kf69i 2 460114 2407769 2026-04-29T23:53:41Z Abhipoojarybannur 55155 All notification acepted 2407769 wikitext text/x-wiki 4lky9ex0f44y4eubsoj3nh0h225ug1f 498 460115 2407983 2026-04-30T11:25:49Z Labslogbot 55 Auto update of instance info. 2407983 wikitext text/x-wiki o31dnzb4ri7aliu4bduml79oc6rgn5l 498 460116 2407984 2026-04-30T11:26:37Z Stashbot 7414 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project wise in eqiad1 (T424068) 2407984 wikitext text/x-wiki turm51w8qt00bz7s95rdigkqhb2w7ob