Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.46.0-wmf.24 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2404213 2402980 2026-04-20T15:24:36Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2404213 wikitext text/x-wiki taz1brsclxb6hqle5iz73y9w0qdlfnb 2404218 2404213 2026-04-20T15:32:13Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli 2404218 wikitext text/x-wiki cb6r0emtc5pt8qhy6ao5t4qhgwyha6t 0 4108 2404185 2404064 2026-04-20T14:40:01Z ScheduleDeploymentBot 37566 Add [[gerrit:1275428]] to Monday, April 20 UTC late backport window 2404185 wikitext text/x-wiki apdq1aw95ul6qr73i93xb2kbu6jkh3i 2404240 2404185 2026-04-20T15:57:50Z ScheduleDeploymentBot 37566 Add [[gerrit:1275428]] to Monday, April 20 UTC late backport window 2404240 wikitext text/x-wiki 4fs3512rajln39p6nxb4sloboz6zkt6 2404243 2404240 2026-04-20T15:59:17Z JDrewniak (WMF) 38238 /* Monday, April 20 */ 2404243 wikitext text/x-wiki jurdqzqn010sk0lafojrfaw9cjhlp81 2404278 2404243 2026-04-20T17:17:25Z ScheduleDeploymentBot 37566 Add [[gerrit:1275475]] to Monday, April 20 UTC late backport window 2404278 wikitext text/x-wiki g6u8zmw0a9hbucljfx8em0ofxap5f0b 2404281 2404278 2026-04-20T17:17:39Z ScheduleDeploymentBot 37566 Add [[gerrit:1275476]] to Monday, April 20 UTC late backport window 2404281 wikitext text/x-wiki 58mgsjk7eamti5w9iwstw1u7lmjsuqq 2404284 2404281 2026-04-20T17:17:55Z ScheduleDeploymentBot 37566 Add [[gerrit:1275477]] to Monday, April 20 UTC late backport window 2404284 wikitext text/x-wiki jpnx33ocixcuxzlknv8afbmk78u91hi 2404286 2404284 2026-04-20T17:18:03Z ScheduleDeploymentBot 37566 Add [[gerrit:1275478]] to Monday, April 20 UTC late backport window 2404286 wikitext text/x-wiki j4xj06ultducin76hriyjdd7ws5s4t8 2404335 2404286 2026-04-20T18:31:28Z PMiazga (WMF) 8979 Removed Attribution patches from Monday backport window. 2404335 wikitext text/x-wiki jurdqzqn010sk0lafojrfaw9cjhlp81 2404375 2404335 2026-04-20T22:17:01Z ScheduleDeploymentBot 37566 Add [[gerrit:1274387]] to Tuesday, April 21 UTC afternoon backport window 2404375 wikitext text/x-wiki oj2dj2854lonzb7x5scdprb1bz3z8au 2404376 2404375 2026-04-20T22:22:09Z ScheduleDeploymentBot 37566 Add [[gerrit:1275540]] to Tuesday, April 21 UTC afternoon backport window 2404376 wikitext text/x-wiki mn2o2hcpvf7hizqqmfvcxy55r79eyc1 2404377 2404376 2026-04-20T22:26:28Z ScheduleDeploymentBot 37566 Add [[gerrit:1275542]] to Tuesday, April 21 UTC afternoon backport window 2404377 wikitext text/x-wiki m2ba3ciov5333znlciacdjhwno9j0cx 2404378 2404377 2026-04-20T22:29:30Z Cscott 41391 Add vendor patch for deploy 2404378 wikitext text/x-wiki nnfujc1ufrlw175hqcqrtm9m66hibno 2404409 2404378 2026-04-21T00:28:46Z ScheduleDeploymentBot 37566 Add [[gerrit:1275560]] to Tuesday, April 21 UTC afternoon backport window 2404409 wikitext text/x-wiki of89k1z4nzobnvqb0azguvk519pkrgh 2404412 2404409 2026-04-21T00:29:43Z ScheduleDeploymentBot 37566 Add [[gerrit:1275561]] to Tuesday, April 21 UTC afternoon backport window 2404412 wikitext text/x-wiki mccxwt2zc5bkua8r4zhsl5z65acjy2v 2404421 2404412 2026-04-21T00:49:30Z ScheduleDeploymentBot 37566 Add [[gerrit:1275562]] to Tuesday, April 21 UTC afternoon backport window 2404421 wikitext text/x-wiki n5wg5749mcopgucgsouacpnf1hnnzka 2404531 2404421 2026-04-21T08:41:21Z ScheduleDeploymentBot 37566 Add [[gerrit:1275808]] to Wednesday, April 22 UTC morning backport window 2404531 wikitext text/x-wiki mjj0psmeea5hnyov76ntm34xkhiw8bu 2404630 2404531 2026-04-21T10:53:43Z ScheduleDeploymentBot 37566 Add [[gerrit:1275840]] to Tuesday, April 21 UTC afternoon backport window 2404630 wikitext text/x-wiki 8f70588olsrwmae9np8kv4w6jpdtjvy 2404631 2404630 2026-04-21T10:53:51Z ScheduleDeploymentBot 37566 Add [[gerrit:1275825]] to Tuesday, April 21 UTC afternoon backport window 2404631 wikitext text/x-wiki jcoes7x4gxegytxxr8egjau89d4qx3c 2404632 2404631 2026-04-21T10:53:56Z ScheduleDeploymentBot 37566 Add [[gerrit:1275834]] to Tuesday, April 21 UTC afternoon backport window 2404632 wikitext text/x-wiki bnlkmfhwzw8aojsbag7mi7kiicnl9gm 2404633 2404632 2026-04-21T10:54:04Z ScheduleDeploymentBot 37566 Add [[gerrit:1275835]] to Tuesday, April 21 UTC afternoon backport window 2404633 wikitext text/x-wiki qwypd25pkf6jgmd048wcemz8k37k01g 2404634 2404633 2026-04-21T10:54:14Z ScheduleDeploymentBot 37566 Add [[gerrit:1275836]] to Tuesday, April 21 UTC afternoon backport window 2404634 wikitext text/x-wiki d7spxttjpjn0gla935ynqwsh3xzj71h 2404635 2404634 2026-04-21T10:54:21Z ScheduleDeploymentBot 37566 Add [[gerrit:1275837]] to Tuesday, April 21 UTC afternoon backport window 2404635 wikitext text/x-wiki nvgn079c9zu7oypw9fbaubbv9tvqt87 2404636 2404635 2026-04-21T10:54:28Z ScheduleDeploymentBot 37566 Add [[gerrit:1275838]] to Tuesday, April 21 UTC afternoon backport window 2404636 wikitext text/x-wiki lolfhyyl3rwdqlv9482dyq8iqtuh4wj 2404637 2404636 2026-04-21T10:54:35Z ScheduleDeploymentBot 37566 Add [[gerrit:1275816]] to Tuesday, April 21 UTC afternoon backport window 2404637 wikitext text/x-wiki 2mlt3cwd82jiby83w2akmsws3ygw4eh 0 4120 2404128 2398209 2026-04-20T13:21:38Z SSingh (WMF) 13928 /* Setting up your SSH config */ bast1003 is deprecated 2404128 wikitext text/x-wiki bg31z16mdqjy17giifumvnqlu1ev2mf 2404336 2404128 2026-04-20T18:37:32Z DDuvall (WMF) 3208 bast1004 is the new eqiad bastion; bast1003 was decommissioned 2404336 wikitext text/x-wiki 1ydmaxnmlbgm1e0j5v3sp7pyn2g6ylt 0 4235 2404097 2286706 2026-04-20T12:14:14Z MVernon (WMF) 25893 Add note about preseed-test, which makes this less painful. 2404097 wikitext text/x-wiki nnqexs6tnj5zfppfrrshcpbqn0rhaq4 498 6726 2404205 2400602 2026-04-20T15:11:40Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2404205 wikitext text/x-wiki 76q8pw1krbv0jxgyems784j9yhg1ns3 2404209 2404205 2026-04-20T15:19:14Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli 2404209 wikitext text/x-wiki 9q17pivv1os1oeubtreve8v468ztjh4 0 7919 2404093 2404092 2026-04-20T12:02:41Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P91189 and previous config saved to /var/cache/conftool/dbconfig/20260420-120239-fceratto.json 2404093 wikitext text/x-wiki gt9jg090am2b2r81ozkl5j7k1wf7t79 2404094 2404093 2026-04-20T12:10:57Z Stashbot 7414 moritzm: installing edk2 security updates 2404094 wikitext text/x-wiki 8v66cbwvft015as6qmk60z6m2x9btek 2404095 2404094 2026-04-20T12:11:44Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.decommission for hosts moss-be[1001-1002].eqiad.wmnet 2404095 wikitext text/x-wiki 45509loxjkkmeva0io3lrplotn8yat7 2404096 2404095 2026-04-20T12:12:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P91190 and previous config saved to /var/cache/conftool/dbconfig/20260420-121247-fceratto.json 2404096 wikitext text/x-wiki lz9d9egcmegaxjffs9zr6oydgzsvvbw 2404098 2404096 2026-04-20T12:15:37Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1328-1334,1360-1374].eqiad.wmnet 2404098 wikitext text/x-wiki mbmzxl6fh18f5jt3gvcsl704gkxsu1g 2404099 2404098 2026-04-20T12:15:42Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1328-1334,1360-1374].eqiad.wmnet 2404099 wikitext text/x-wiki q7xmjkccs8f2i8l2vjketoijf9pjnhz 2404100 2404099 2026-04-20T12:16:33Z Stashbot 7414 moritzm: remove ganeti5006 from eqsin01 Ganeti cluster (running classic Ganeti) T421863 2404100 wikitext text/x-wiki qrey7cfyvshjl56bahj6c3adkv4qlkv 2404101 2404100 2026-04-20T12:17:33Z Stashbot 7414 zabe: Deployed patch for T423821 2404101 wikitext text/x-wiki eyap6j5jivieotw0x9oind9nrth4hl2 2404102 2404101 2026-04-20T12:23:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T419635)', diff saved to https://phabricator.wikimedia.org/P91191 and previous config saved to /var/cache/conftool/dbconfig/20260420-122256-fceratto.json 2404102 wikitext text/x-wiki 9sqencetqulqdsjzciqyrnl2jgrvj33 2404103 2404102 2026-04-20T12:23:15Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1231.eqiad.wmnet with reason: Maintenance 2404103 wikitext text/x-wiki jtoxjvv9sir9d0z6edjl6gujcwvtbt5 2404104 2404103 2026-04-20T12:23:25Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1231 (T419635)', diff saved to https://phabricator.wikimedia.org/P91192 and previous config saved to /var/cache/conftool/dbconfig/20260420-122321-fceratto.json 2404104 wikitext text/x-wiki ier40jc9cpdlktku7lv1wrucjt296t3 2404105 2404104 2026-04-20T12:25:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T419635)', diff saved to https://phabricator.wikimedia.org/P91193 and previous config saved to /var/cache/conftool/dbconfig/20260420-122534-fceratto.json 2404105 wikitext text/x-wiki 5qsyf2vjlle7td7vzh3qrt82hyowzui 2404106 2404105 2026-04-20T12:26:39Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.netbox 2404106 wikitext text/x-wiki 1n2dc4u1b3cgnn27dvl2xd00dwp0z99 2404107 2404106 2026-04-20T12:28:15Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1002-1005,1011-1012,1019-1020,1029-1031,1058-1063,1082-1083,1088-1092,1096,1098-1112,1166-1168].eqiad.wmnet 2404107 wikitext text/x-wiki joljnly0et7a7oaoydbaea4qg5s3qpa 2404108 2404107 2026-04-20T12:31:08Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: moss-be[1001-1002].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - mvernon@cumin2002" 2404108 wikitext text/x-wiki hdp2db4rkdw4qn7vsualgzt40yfwd8e 2404109 2404108 2026-04-20T12:35:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P91194 and previous config saved to /var/cache/conftool/dbconfig/20260420-123542-fceratto.json 2404109 wikitext text/x-wiki 7l2bu065grl31aeg27c3p3oqcynmyzf 2404110 2404109 2026-04-20T12:35:46Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: moss-be[1001-1002].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - mvernon@cumin2002" 2404110 wikitext text/x-wiki 6c55kd5va2tevp7dg4ltrke9i9be74i 2404111 2404110 2026-04-20T12:35:48Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2404111 wikitext text/x-wiki fwhjup252lpnl0oce94vr1nan7bjzah 2404112 2404111 2026-04-20T12:35:50Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts moss-be[1001-1002].eqiad.wmnet 2404112 wikitext text/x-wiki swgfgd9kpjtlbb23yjrdfj35fk7rbx6 2404113 2404112 2026-04-20T12:45:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P91195 and previous config saved to /var/cache/conftool/dbconfig/20260420-124550-fceratto.json 2404113 wikitext text/x-wiki 1clb3pg2pie5n4scupacxjvb5el9iw4 2404114 2404113 2026-04-20T12:56:04Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T419635)', diff saved to https://phabricator.wikimedia.org/P91196 and previous config saved to /var/cache/conftool/dbconfig/20260420-125559-fceratto.json 2404114 wikitext text/x-wiki nnxcz36y0r5wx9svze3abdaebvhfo3d 2404115 2404114 2026-04-20T12:56:18Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1236.eqiad.wmnet with reason: Maintenance 2404115 wikitext text/x-wiki 8gprl7qhesm2pizowkmzovf35o8ykln 2404116 2404115 2026-04-20T12:56:28Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1236 (T419635)', diff saved to https://phabricator.wikimedia.org/P91197 and previous config saved to /var/cache/conftool/dbconfig/20260420-125624-fceratto.json 2404116 wikitext text/x-wiki 3qraq8qptfrq5zj2kmv8s23qbyiw4qr 2404119 2404116 2026-04-20T12:58:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1236 (T419635)', diff saved to https://phabricator.wikimedia.org/P91198 and previous config saved to /var/cache/conftool/dbconfig/20260420-125837-fceratto.json 2404119 wikitext text/x-wiki bw6hgu2ml7xskgcjyr7kxz4q7wm7v6l 2404120 2404119 2026-04-20T12:58:49Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1002-1005,1011-1012,1019-1020,1029-1031,1058-1063,1082-1083,1088-1092,1096,1098-1112,1166-1168].eqiad.wmnet 2404120 wikitext text/x-wiki psha9zii76qu85tht7c74ox3j32jhcw 2404122 2404120 2026-04-20T13:08:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P91199 and previous config saved to /var/cache/conftool/dbconfig/20260420-130845-fceratto.json 2404122 wikitext text/x-wiki 4jlk2zstzm5yhr1qipkrtvit0xvvr8z 2404123 2404122 2026-04-20T13:12:30Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1247960|Remove unused JWT for bot password temporary config (T422367 T415007)]], [[gerrit:1273842|Enable ReadingLists beta feature for all Wikipedia wikis (T420881)]] 2404123 wikitext text/x-wiki atu39mia7mnsfmkxrqj4m7su9g65u1x 2404124 2404123 2026-04-20T13:15:42Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, aude, d3r1ck01: Backport for [[gerrit:1247960|Remove unused JWT for bot password temporary config (T422367 T415007)]], [[gerrit:1273842|Enable ReadingLists beta feature for all Wikipedia wikis (T420881)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2404124 wikitext text/x-wiki 55ayokt7jzhsg6w9sklpng4su3un0aj 2404125 2404124 2026-04-20T13:17:00Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, aude, d3r1ck01: Continuing with sync 2404125 wikitext text/x-wiki 6je52ulrboxaoa3aa97qofbakyxjxz4 2404126 2404125 2026-04-20T13:18:54Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P91200 and previous config saved to /var/cache/conftool/dbconfig/20260420-131853-fceratto.json 2404126 wikitext text/x-wiki bdkwf58j142ugxo35jfp6xh8kgw1dmv 2404127 2404126 2026-04-20T13:20:52Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1247960|Remove unused JWT for bot password temporary config (T422367 T415007)]], [[gerrit:1273842|Enable ReadingLists beta feature for all Wikipedia wikis (T420881)]] (duration: 08m 21s) 2404127 wikitext text/x-wiki 2o90r0b21i0tytbixhh4suvk2svy4qk 2404129 2404127 2026-04-20T13:22:08Z Stashbot 7414 phuedx@deploy1003: Started scap sync-world: Backport for [[gerrit:1275357|PHP SDK: Split measurement of unknown experiments (T422112)]] 2404129 wikitext text/x-wiki 46y3n25uoy2p7z6f0xti119e4qou6op 2404130 2404129 2026-04-20T13:23:43Z Stashbot 7414 phuedx@deploy1003: phuedx: Backport for [[gerrit:1275357|PHP SDK: Split measurement of unknown experiments (T422112)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2404130 wikitext text/x-wiki 6j0sl5d5ppmmffmwyqv96x5dffq2xcm 2404131 2404130 2026-04-20T13:26:09Z Stashbot 7414 phuedx@deploy1003: phuedx: Continuing with sync 2404131 wikitext text/x-wiki tleyi7sykuooij9vx10945yys1pd2pk 2404132 2404131 2026-04-20T13:29:04Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1236 (T419635)', diff saved to https://phabricator.wikimedia.org/P91202 and previous config saved to /var/cache/conftool/dbconfig/20260420-132901-fceratto.json 2404132 wikitext text/x-wiki 9y8onl8pgxz6dr3hwbevpsvs6djcjgj 2404133 2404132 2026-04-20T13:29:20Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1253.eqiad.wmnet with reason: Maintenance 2404133 wikitext text/x-wiki 6igys3ym9s3q1bzzvug2c3veawtw7hm 2404134 2404133 2026-04-20T13:29:30Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1253 (T419635)', diff saved to https://phabricator.wikimedia.org/P91203 and previous config saved to /var/cache/conftool/dbconfig/20260420-132926-fceratto.json 2404134 wikitext text/x-wiki risi0xh2vtao4vxgx8wapl380sqwebk 2404135 2404134 2026-04-20T13:29:59Z Stashbot 7414 phuedx@deploy1003: Finished scap sync-world: Backport for [[gerrit:1275357|PHP SDK: Split measurement of unknown experiments (T422112)]] (duration: 07m 51s) 2404135 wikitext text/x-wiki sx6r2wsj2m4j22iyst3mwttr00nyvaa 2404136 2404135 2026-04-20T13:30:10Z Stashbot 7414 eevans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on aqs1014.eqiad.wmnet with reason: Decommissioning — T412830 2404136 wikitext text/x-wiki oiv6jiib6ntxvc56xzj5or0xcstusdn 2404137 2404136 2026-04-20T13:31:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253 (T419635)', diff saved to https://phabricator.wikimedia.org/P91204 and previous config saved to /var/cache/conftool/dbconfig/20260420-133139-fceratto.json 2404137 wikitext text/x-wiki i5895u64nsqbi2s85oddjfrqn47vaz1 2404138 2404137 2026-04-20T13:32:39Z Stashbot 7414 urandom: decommissioning Cassandra, aqs1014 [a,b] — T412830 2404138 wikitext text/x-wiki c7k8ll7h6qgw8kor98rel4ks9mv3kcj 2404139 2404138 2026-04-20T13:34:17Z Stashbot 7414 Lucas_WMDE: UTC afternoon backport+config window done 2404139 wikitext text/x-wiki lmdw1tjkh870d6y3vc8jy3f7mqo2n88 2404140 2404139 2026-04-20T13:35:47Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance 2404140 wikitext text/x-wiki of1o99wus7m5lp3u93ue9bnyy1qdlui 2404141 2404140 2026-04-20T13:37:39Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.dns.netbox 2404141 wikitext text/x-wiki c6np4pvyrvfz7ysr56z2zvctj0zx6ql 2404142 2404141 2026-04-20T13:41:49Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P91205 and previous config saved to /var/cache/conftool/dbconfig/20260420-134148-fceratto.json 2404142 wikitext text/x-wiki tabe1n65wsr7ljmspwkznqjo4s2xpk7 2404143 2404142 2026-04-20T13:41:52Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance 2404143 wikitext text/x-wiki a9cxoy4cmedhs9xg87553otcnnjon2d 2404144 2404143 2026-04-20T13:42:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1157 (T419961)', diff saved to https://phabricator.wikimedia.org/P91206 and previous config saved to /var/cache/conftool/dbconfig/20260420-134158-fceratto.json 2404144 wikitext text/x-wiki p2340pshe8azm7lf1wafuygmroegraa 2404146 2404144 2026-04-20T13:44:17Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dbstore1010 to eqiad - jclark@cumin1003" 2404146 wikitext text/x-wiki 0c95z6auyxtk5s2k5mf7shz4cx9ev7c 2404147 2404146 2026-04-20T13:44:23Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dbstore1010 to eqiad - jclark@cumin1003" 2404147 wikitext text/x-wiki nrt9565azlc7fpjo6ai42o9juygd9ju 2404148 2404147 2026-04-20T13:44:25Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2404148 wikitext text/x-wiki qzmjyaiwlfeyc8ih4q0lpv102mo03ex 2404149 2404148 2026-04-20T13:45:03Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host dbstore1010.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2404149 wikitext text/x-wiki tckelzz4bhpuun5kzo7k89407jz45s1 2404150 2404149 2026-04-20T13:47:59Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.dns.netbox 2404150 wikitext text/x-wiki ex4q8stwjzjlna88b6riy7wlnu9z15s 2404151 2404150 2026-04-20T13:50:28Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T419961)', diff saved to https://phabricator.wikimedia.org/P91207 and previous config saved to /var/cache/conftool/dbconfig/20260420-135025-fceratto.json 2404151 wikitext text/x-wiki 7ech4jqed7gn6t3qy7hkzmpb6f07h72 2404152 2404151 2026-04-20T13:50:51Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2404152 wikitext text/x-wiki p0fsi1zu7vv7ofpm8odcuwvuioijs9r 2404153 2404152 2026-04-20T13:51:57Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P91208 and previous config saved to /var/cache/conftool/dbconfig/20260420-135155-fceratto.json 2404153 wikitext text/x-wiki hfbbutjwplx4hz048lgoi24e83bp12t 2404154 2404153 2026-04-20T13:52:48Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2152.codfw.wmnet with reason: Maintenance 2404154 wikitext text/x-wiki mtzfiq61go7yb0ajp0h5kuztgfaq9bc 2404155 2404154 2026-04-20T13:52:58Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2152 (T410589)', diff saved to https://phabricator.wikimedia.org/P91209 and previous config saved to /var/cache/conftool/dbconfig/20260420-135255-ladsgroup.json 2404155 wikitext text/x-wiki 6jgrby166wr42ekuu6ygh74unyoevoh 2404157 2404155 2026-04-20T13:59:09Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2144.codfw.wmnet,db1151.eqiad.wmnet with reason: Reimage to Trixie 2404157 wikitext text/x-wiki 80l509gib94xupee2gc2r24i21oexir 2404158 2404157 2026-04-20T14:00:03Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2404158 wikitext text/x-wiki mh7j5gwqzusybl3b5m4c2dptlvl8slp 2404159 2404158 2026-04-20T14:00:06Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1151.eqiad.wmnet with reason: Reimage to Trixie 2404159 wikitext text/x-wiki bgz457as8tunpy71a3v1xaek80p1le6 2404160 2404159 2026-04-20T14:00:08Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1151: Reimage to Trixie 2404160 wikitext text/x-wiki 8w0l775i78wtrn8tzhtbfyqh1zmb3gq 2404161 2404160 2026-04-20T14:00:10Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2404161 wikitext text/x-wiki hghqui1eab828p1edz3bdil8t218g2j 2404162 2404161 2026-04-20T14:00:16Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2404162 wikitext text/x-wiki efy8b7wkcwua324on22ukv0e3kilp8d 2404164 2404162 2026-04-20T14:00:19Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1151: Reimage to Trixie 2404164 wikitext text/x-wiki dpzradl9l8sc21vjlkcdbva9d5kvc2o 2404165 2404164 2026-04-20T14:00:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P91211 and previous config saved to /var/cache/conftool/dbconfig/20260420-140033-fceratto.json 2404165 wikitext text/x-wiki 57jxgtxc6b95yklzmvymrc6ynlgm65y 2404166 2404165 2026-04-20T14:00:44Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1151.eqiad.wmnet with OS trixie 2404166 wikitext text/x-wiki 1y0bidkyq5017xpry6sv17yus5h0okf 2404167 2404166 2026-04-20T14:02:07Z Stashbot 7414 urandom: upgrade envoyproxy, restbase — T419637 & T410975 2404167 wikitext text/x-wiki 3uen7ucovjdxsl5iup13mjunya5kayo 2404168 2404167 2026-04-20T14:02:18Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253 (T419635)', diff saved to https://phabricator.wikimedia.org/P91212 and previous config saved to /var/cache/conftool/dbconfig/20260420-140203-fceratto.json 2404168 wikitext text/x-wiki aa9e3ga9nywmvlbew9k2kw0vns62ucy 2404169 2404168 2026-04-20T14:02:22Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance 2404169 wikitext text/x-wiki senviy6nd1nd27hi4jabf4wwyalxg7v 2404170 2404169 2026-04-20T14:10:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P91213 and previous config saved to /var/cache/conftool/dbconfig/20260420-141042-fceratto.json 2404170 wikitext text/x-wiki lanzsyr2roocue19ffuy6e2ybssz8f2 2404171 2404170 2026-04-20T14:14:16Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host thanos-be2006.codfw.wmnet with OS bullseye 2404171 wikitext text/x-wiki nmoqza352ylydvl28qnqngb7daunfy1 2404172 2404171 2026-04-20T14:14:23Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1151.eqiad.wmnet with reason: host reimage 2404172 wikitext text/x-wiki iaz7cin09gc1byn4zmc6dxdx3gtzhz9 2404173 2404172 2026-04-20T14:15:10Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.provision for host dbstore1010.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2404173 wikitext text/x-wiki iyuheivt4lmyn49l4hyebo0nnkgyik0 2404174 2404173 2026-04-20T14:19:20Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1151.eqiad.wmnet with reason: host reimage 2404174 wikitext text/x-wiki f582rkpl4wlmgzvrbcuzghp8jw6zppg 2404175 2404174 2026-04-20T14:19:51Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudelastic1012.eqiad.wmnet 2404175 wikitext text/x-wiki s1u526657u5o25qlpocyyftaalrxs15 2404176 2404175 2026-04-20T14:20:53Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T419961)', diff saved to https://phabricator.wikimedia.org/P91214 and previous config saved to /var/cache/conftool/dbconfig/20260420-142050-fceratto.json 2404176 wikitext text/x-wiki 264ihuv038n8mir721futer090xu8m3 2404177 2404176 2026-04-20T14:21:14Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance 2404177 wikitext text/x-wiki j260jfhcaav2xxw0bthegloqbu71gfm 2404178 2404177 2026-04-20T14:21:23Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1166 (T419961)', diff saved to https://phabricator.wikimedia.org/P91215 and previous config saved to /var/cache/conftool/dbconfig/20260420-142120-fceratto.json 2404178 wikitext text/x-wiki hdb36squnvz7gggqkzdp61wyskdrvqd 2404179 2404178 2026-04-20T14:21:51Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudelastic1012.eqiad.wmnet 2404179 wikitext text/x-wiki hv5xbndnqhtaqsl9lore1h34s4wpd2k 2404180 2404179 2026-04-20T14:26:53Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2404180 wikitext text/x-wiki 7c9reaqbkoyxi4h5sdqnc7cffjrv76f 2404181 2404180 2026-04-20T14:36:00Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dbstore1010.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2404181 wikitext text/x-wiki rcm6tm18bx0h18h6lgx0tir7i58k333 2404182 2404181 2026-04-20T14:37:00Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be2006.codfw.wmnet with reason: host reimage 2404182 wikitext text/x-wiki 7ehiqg0wnqn6cgao3yu03srxh63es2u 2404183 2404182 2026-04-20T14:37:16Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dbstore1010.eqiad.wmnet with OS bookworm 2404183 wikitext text/x-wiki c13pcdgai5l2rjgy1n3llobm8toqzul 2404184 2404183 2026-04-20T14:37:18Z Stashbot 7414 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host dbstore1010.eqiad.wmnet with OS bookworm 2404184 wikitext text/x-wiki pw7u28cthinsiwxsd10fpmlwl7ar5a7 2404186 2404184 2026-04-20T14:40:41Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host dbstore1010.eqiad.wmnet with OS bookworm 2404186 wikitext text/x-wiki kv568vicdnigtsminbud8t48yqofor6 2404187 2404186 2026-04-20T14:42:09Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1151.eqiad.wmnet with OS trixie 2404187 wikitext text/x-wiki jb7mra5i9z0gxsjddlr20e1jdwez6pi 2404188 2404187 2026-04-20T14:43:08Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1151: after reimage to trixie 2404188 wikitext text/x-wiki qb94rd4sr7j4sa6y7uuuo5j0t6y5mpn 2404189 2404188 2026-04-20T14:43:10Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db1151: after reimage to trixie 2404189 wikitext text/x-wiki fhv2tnwywqojpn02n0xbf1irprxgz2n 2404190 2404189 2026-04-20T14:45:24Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be2006.codfw.wmnet with reason: host reimage 2404190 wikitext text/x-wiki 0ugfk5b2adeqio3u8fbu1lc7goyawyt 2404191 2404190 2026-04-20T14:45:55Z Stashbot 7414 cwhite@deploy1003: Started deploy [performance/arc-lamp@bd7b2ab]: T413127 2404191 wikitext text/x-wiki 1j8400zx2uxsg61p4ei360tro19wvc5 2404192 2404191 2026-04-20T14:46:04Z Stashbot 7414 cwhite@deploy1003: Finished deploy [performance/arc-lamp@bd7b2ab]: T413127 (duration: 00m 08s) 2404192 wikitext text/x-wiki qq6gr8myrzbvnuuj0u9t7zefxah2lhp 2404193 2404192 2026-04-20T14:51:25Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wikikube-worker2188'] 2404193 wikitext text/x-wiki 48cknri3xbfrxaen9mwz63efrlj9wj4 2404194 2404193 2026-04-20T14:52:03Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2404194 wikitext text/x-wiki s7bzbjgifc5as0lmut9uu275y4mylxi 2404196 2404194 2026-04-20T15:03:47Z Stashbot 7414 trueg@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply 2404196 wikitext text/x-wiki 5lxvennapevn7pozl4q7pbwktb1l3z8 2404197 2404196 2026-04-20T15:03:50Z Stashbot 7414 trueg@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply 2404197 wikitext text/x-wiki f401qzp0dhxl90opfh7165j7i5pdz04 2404198 2404197 2026-04-20T15:05:11Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dbstore1010.eqiad.wmnet with reason: host reimage 2404198 wikitext text/x-wiki 042i4vfw4hxg3jew8czaqxrhupc8g18 2404199 2404198 2026-04-20T15:05:40Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be2006.codfw.wmnet with OS bullseye 2404199 wikitext text/x-wiki k04na7qcsvh8iucl2d3jaejtxvztv45 2404200 2404199 2026-04-20T15:09:06Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbstore1010.eqiad.wmnet with reason: host reimage 2404200 wikitext text/x-wiki skxkiiolin9ucy3l92mlmu0wsltmjiv 2404202 2404200 2026-04-20T15:11:05Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1151: repool after maintenance 2404202 wikitext text/x-wiki bu1oqebrjyxmb32s6y5eeahblb41n9m 2404203 2404202 2026-04-20T15:11:07Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db1151: repool after maintenance 2404203 wikitext text/x-wiki ljodsq6s949oooobz3etb58qrnfjtnc 2404204 2404203 2026-04-20T15:11:22Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wikikube-worker2188'] 2404204 wikitext text/x-wiki 76vghyanymbpuavty4e97gctzjtsxm7 2404206 2404204 2026-04-20T15:13:10Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2036: Moving to another rack 2404206 wikitext text/x-wiki envcd8ej4xxx62hde5zndtjrvq1yafh 2404207 2404206 2026-04-20T15:13:28Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2036: Moving to another rack 2404207 wikitext text/x-wiki ds2u467wvtdv5zuskbs6gp37hph4r2q 2404208 2404207 2026-04-20T15:14:23Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Moved to anotehr rack 2404208 wikitext text/x-wiki 1i03lb6ys0jiwodeld992i5oey9cwq3 2404210 2404208 2026-04-20T15:20:12Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1166: Security update 2404210 wikitext text/x-wiki 2y5ah1p5ejde3hcphe6sazm43bk1n59 2404211 2404210 2026-04-20T15:23:35Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1159.eqiad.wmnet with reason: Maintenance 2404211 wikitext text/x-wiki bgp5ksdrpyjw5jtpbdywhqsbpel0wm1 2404212 2404211 2026-04-20T15:23:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1159 (T419635)', diff saved to https://phabricator.wikimedia.org/P91217 and previous config saved to /var/cache/conftool/dbconfig/20260420-152341-fceratto.json 2404212 wikitext text/x-wiki 17al7kwxeluz01uzkl5zmzkemrz8wgj 2404215 2404212 2026-04-20T15:25:48Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db1166: Security update 2404215 wikitext text/x-wiki 16iyc95yphfv4bf09oxbxswxdr1f8yj 2404216 2404215 2026-04-20T15:25:50Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reboot-single for host cloudelastic1012.eqiad.wmnet 2404216 wikitext text/x-wiki s41ufcfykt7g3pvx9aill1psa3kthum 2404217 2404216 2026-04-20T15:27:04Z Stashbot 7414 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2404217 wikitext text/x-wiki fyfq9wpxvsl6wx0ccbwn38wllnr54f2 2404219 2404217 2026-04-20T15:33:18Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2404219 wikitext text/x-wiki 4zmfkvwb2sbtotblj9ff0mihjoxb2ye 2404220 2404219 2026-04-20T15:34:50Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2404220 wikitext text/x-wiki qndlamy3o4stid1r7a6lgge9drowkgh 2404221 2404220 2026-04-20T15:34:54Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2404221 wikitext text/x-wiki rwsqjdm0hfzac6y68ne15sevukz8wwu 2404222 2404221 2026-04-20T15:35:49Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1166: Security update 2404222 wikitext text/x-wiki f76p7a6qums1aolkyyn835lb972lanv 2404223 2404222 2026-04-20T15:36:02Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2404223 wikitext text/x-wiki 9juga9q2q4b9wvemzo3a278wjg7e42d 2404224 2404223 2026-04-20T15:36:17Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" 2404224 wikitext text/x-wiki 9x0ybyiz7nft7wxriukjh5aqs5jid09 2404225 2404224 2026-04-20T15:36:19Z Stashbot 7414 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbstore1010.eqiad.wmnet with OS bookworm 2404225 wikitext text/x-wiki qwj8t40w7zdc8fvrq2fy7e44bynqtvt 2404226 2404225 2026-04-20T15:36:22Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host es2036 2404226 wikitext text/x-wiki 3xk6h50z2i4g7ued835269qw107qw26 2404227 2404226 2026-04-20T15:36:31Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host es2036 2404227 wikitext text/x-wiki 07bqeg6y3qzpk690xs7im5onbfq9gq7 2404228 2404227 2026-04-20T15:36:59Z Stashbot 7414 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cloudelastic1012.eqiad.wmnet 2404228 wikitext text/x-wiki ta6xlpsw11jfv90awbddu6f6r8mi9kd 2404229 2404228 2026-04-20T15:41:34Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2404229 wikitext text/x-wiki awwl4lbk5byqsoe8ayouqt2qa11c4q3 2404230 2404229 2026-04-20T15:41:38Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2404230 wikitext text/x-wiki jg0qesv3ufps5hrtfbpf48wif3cy57p 2404231 2404230 2026-04-20T15:48:49Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti5006.eqsin.wmnet with OS bookworm 2404231 wikitext text/x-wiki 0lzwfx17jj6f8klgsvoppztfaqh81jk 2404232 2404231 2026-04-20T15:50:11Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.hosts.remove-downtime for wikikube-worker2188.codfw.wmnet 2404232 wikitext text/x-wiki g2yjvi6t8v8nhgazel7h6xfyy43wndw 2404233 2404232 2026-04-20T15:50:13Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker2188.codfw.wmnet 2404233 wikitext text/x-wiki 5y0ebsso95w4xkca667ianbjca1vnk9 2404234 2404233 2026-04-20T15:50:46Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2036: Moving to another rack 2404234 wikitext text/x-wiki 1ha6xpqjibfo1466y9mvcrb2muqvcwa 2404235 2404234 2026-04-20T15:50:50Z Stashbot 7414 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool es2036: Moving to another rack 2404235 wikitext text/x-wiki tgh2xh8kdu639lza07vawnx0nm64jzf 2404236 2404235 2026-04-20T15:50:57Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2188.codfw.wmnet 2404236 wikitext text/x-wiki smrmameos5pfm9icmc3g3r3gcsgzp5p 2404237 2404236 2026-04-20T15:51:00Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2188.codfw.wmnet 2404237 wikitext text/x-wiki ftwigc63eefugjebaizfzannfqge870 2404238 2404237 2026-04-20T15:51:02Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2036: Moving to another rack 2404238 wikitext text/x-wiki hvyb5x4rbebay36qhon7w6p5p8r7dsx 2404239 2404238 2026-04-20T15:55:03Z Stashbot 7414 sukhe: sudo cumin -b31 "A:cp and not P{cp2041* or cp2042*}" "run-puppet-agent --enable 'merging CR 1272869'" 2404239 wikitext text/x-wiki q855jz455jlybbsx1onwpalicl26yfh 2404241 2404239 2026-04-20T15:57:56Z Stashbot 7414 moritzm: installing libvirt security updates 2404241 wikitext text/x-wiki 0p87sgckw5ut9zxkwblpd9yhsapjk8d 2404245 2404241 2026-04-20T16:06:30Z Stashbot 7414 bking@cumin2002: conftool action : set/pooled=no; selector: name=cloudelastic1012.eqiad.wmnet 2404245 wikitext text/x-wiki fy6qp2lkanrbv6j6ayztpmmcdq2v8mw 2404246 2404245 2026-04-20T16:17:31Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2404246 wikitext text/x-wiki 82b9fwg6z0jbn8s9mu07izurn3t53ml 2404247 2404246 2026-04-20T16:17:35Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2404247 wikitext text/x-wiki 6v65sm1zrf7nq1zia3ioq4xnip2lcvu 2404248 2404247 2026-04-20T16:17:52Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage 2404248 wikitext text/x-wiki sncw2x5u6mn3hzxxqgefc6ulbxh6yu7 2404249 2404248 2026-04-20T16:19:23Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM backupmon1001.eqiad.wmnet 2404249 wikitext text/x-wiki 93aeyzba2rkq9gk9yub7b6yss22utai 2404250 2404249 2026-04-20T16:21:16Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1166: Security update 2404250 wikitext text/x-wiki ihxllpk9rs551sp73oumjiinwqh84cu 2404251 2404250 2026-04-20T16:24:02Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T419635)', diff saved to https://phabricator.wikimedia.org/P91226 and previous config saved to /var/cache/conftool/dbconfig/20260420-162359-fceratto.json 2404251 wikitext text/x-wiki 4b36g28u8x3i4jm938s7qv44wl33tf7 2404252 2404251 2026-04-20T16:25:55Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5006.eqsin.wmnet with reason: host reimage 2404252 wikitext text/x-wiki 272yeg92t9cqac61h3tvriw07pk4xu4 2404253 2404252 2026-04-20T16:26:01Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2404253 wikitext text/x-wiki k94gtqanjtt8zk1mlqr13z0y3lfciad 2404254 2404253 2026-04-20T16:26:51Z Stashbot 7414 marostegui: Switchover m3 proxy (phabricator) 2404254 wikitext text/x-wiki iqb740tgrzd6yfeh68n0qj8240yuk2i 2404255 2404254 2026-04-20T16:27:28Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2404255 wikitext text/x-wiki dn7ywz7o52fje0ei86h7hcmygtk19la 2404256 2404255 2026-04-20T16:29:02Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM backupmon1001.eqiad.wmnet 2404256 wikitext text/x-wiki i70iq4miztqu0m8jne2z3iaruvu4r0h 2404257 2404256 2026-04-20T16:29:33Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM aux-k8s-etcd1003.eqiad.wmnet 2404257 wikitext text/x-wiki q4qlx5ccxnwqpkryolxa2kvgr4nz4p3 2404258 2404257 2026-04-20T16:33:35Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM aux-k8s-etcd1003.eqiad.wmnet 2404258 wikitext text/x-wiki 5fs6gs84hbkpw88c60egv8qgek9epb8 2404259 2404258 2026-04-20T16:34:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P91227 and previous config saved to /var/cache/conftool/dbconfig/20260420-163407-fceratto.json 2404259 wikitext text/x-wiki e007f73n1hmjzidstqyt6jfi4wz2fzv 2404260 2404259 2026-04-20T16:34:53Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM aux-k8s-etcd1003.eqiad.wmnet 2404260 wikitext text/x-wiki gkzin74239ur1ggxspq0gl44kt1kuxf 2404261 2404260 2026-04-20T16:36:26Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2036: Moving to another rack 2404261 wikitext text/x-wiki tpqzmj7j6gvzqdafp8bmrmc02rjtzl4 2404262 2404261 2026-04-20T16:38:53Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM aux-k8s-etcd1003.eqiad.wmnet 2404262 wikitext text/x-wiki dkpqb0348jqmnprkajzrxtazp3z9i55 2404263 2404262 2026-04-20T16:44:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P91229 and previous config saved to /var/cache/conftool/dbconfig/20260420-164415-fceratto.json 2404263 wikitext text/x-wiki 6qxwt56nsuih2zstgvtih5zdf6ica7f 2404264 2404263 2026-04-20T16:44:42Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM aux-k8s-etcd1003.eqiad.wmnet 2404264 wikitext text/x-wiki skdaw330joox0uaff4jbmw31bvlnwve 2404265 2404264 2026-04-20T16:48:42Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM aux-k8s-etcd1003.eqiad.wmnet 2404265 wikitext text/x-wiki nsu46369xpyysam19ju423xh4zrv66o 2404266 2404265 2026-04-20T16:48:44Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5006.eqsin.wmnet with OS bookworm 2404266 wikitext text/x-wiki jr8bw96fw1viid6dt13vv4wxp6228fk 2404267 2404266 2026-04-20T16:52:37Z Stashbot 7414 moritzm: installing imagemagick security updates 2404267 wikitext text/x-wiki f52ci18235pncnz0iiuc9nm62btvwag 2404268 2404267 2026-04-20T16:54:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T419635)', diff saved to https://phabricator.wikimedia.org/P91230 and previous config saved to /var/cache/conftool/dbconfig/20260420-165423-fceratto.json 2404268 wikitext text/x-wiki amoz9d55oom0u0ykwov956m5tgscmn3 2404269 2404268 2026-04-20T16:54:43Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1161.eqiad.wmnet with reason: Maintenance 2404269 wikitext text/x-wiki 1qy9m40wp7z3ot317tad7v9lffgl2hn 2404270 2404269 2026-04-20T16:54:52Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance 2404270 wikitext text/x-wiki fogrd870qdy1lobvn5srl5icqantydr 2404271 2404270 2026-04-20T16:55:02Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1161 (T419635)', diff saved to https://phabricator.wikimedia.org/P91231 and previous config saved to /var/cache/conftool/dbconfig/20260420-165459-fceratto.json 2404271 wikitext text/x-wiki ps4f8qwc7olioajcxq5cbx5ol91y9bg 2404272 2404271 2026-04-20T17:03:00Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reimage for host cloudelastic1012.eqiad.wmnet with OS trixie 2404272 wikitext text/x-wiki ow697dxiu1sbjl43cna4gxwxewi66y3 2404273 2404272 2026-04-20T17:14:43Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudelastic1012.eqiad.wmnet with reason: host reimage 2404273 wikitext text/x-wiki 1w0ukpplh4kds6dtp8987zor3av4iy7 2404274 2404273 2026-04-20T17:15:06Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host thanos-be1005.eqiad.wmnet with OS bullseye 2404274 wikitext text/x-wiki g0xald9hxbdmzxzqe721ys8j2ymevi4 2404275 2404274 2026-04-20T17:16:55Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply 2404275 wikitext text/x-wiki 65jcngakh4idra3zr4neyjo0h3n57sr 2404276 2404275 2026-04-20T17:17:24Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply 2404276 wikitext text/x-wiki 1w0uuyplq9wjmeuj8pm4ow8da0l4v0m 2404277 2404276 2026-04-20T17:17:25Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply 2404277 wikitext text/x-wiki jjaibyjfcphaz671ybu05ye0lthchxk 2404279 2404277 2026-04-20T17:17:34Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply 2404279 wikitext text/x-wiki 1prgiv1pqoa8wye210qrdlcbep35ofq 2404280 2404279 2026-04-20T17:17:36Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply 2404280 wikitext text/x-wiki 2ysbmcbbklknwrnt8mtm864rfd0hcin 2404282 2404280 2026-04-20T17:17:48Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply 2404282 wikitext text/x-wiki 35165lo6hylov3g9znq7c8jldufqfup 2404283 2404282 2026-04-20T17:17:49Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply 2404283 wikitext text/x-wiki 285j5nq6apch2ox5lvwy351wchllgrm 2404285 2404283 2026-04-20T17:18:03Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2404285 wikitext text/x-wiki k12z0hjb5i5rhb1ffftyannro8dfh0e 2404287 2404285 2026-04-20T17:18:05Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply 2404287 wikitext text/x-wiki rzqp5k8tj5ifrjrn2o8qr3c91rkdot8 2404288 2404287 2026-04-20T17:18:21Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply 2404288 wikitext text/x-wiki 0ep8l8wm0iqt9ra8qey7uv591eayal3 2404289 2404288 2026-04-20T17:18:23Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply 2404289 wikitext text/x-wiki pv5s5uqw7kkck1wkycerwgqaq48vrao 2404290 2404289 2026-04-20T17:18:41Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudelastic1012.eqiad.wmnet with reason: host reimage 2404290 wikitext text/x-wiki hoy2yfd9rkayws0xzc27o01ibqzn94e 2404291 2404290 2026-04-20T17:18:45Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply 2404291 wikitext text/x-wiki a3zhh153m0omb950ersvr1xjk9on3du 2404292 2404291 2026-04-20T17:21:46Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply 2404292 wikitext text/x-wiki e7wl3dcu8dqgwnnggpskemgruaq0log 2404293 2404292 2026-04-20T17:22:32Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply 2404293 wikitext text/x-wiki fsup6gpj01anpwg16fwcmfu6yv0w22z 2404294 2404293 2026-04-20T17:23:03Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply 2404294 wikitext text/x-wiki h1tkt2nnqrdefdrqd1m3orswf6jx7bi 2404295 2404294 2026-04-20T17:23:33Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply 2404295 wikitext text/x-wiki 6jme9p1kbpvbzu7stkc1dat69s6quiq 2404296 2404295 2026-04-20T17:24:05Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply 2404296 wikitext text/x-wiki l4vrab9cc04cvb4wl2xmn1wugl9eu8p 2404297 2404296 2026-04-20T17:24:21Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply 2404297 wikitext text/x-wiki iy5wb0zsfh6ri4ls4itm5pikgmlgkw3 2404298 2404297 2026-04-20T17:24:52Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply 2404298 wikitext text/x-wiki dwlv067d1degwagjsaqgudtzcswnztc 2404299 2404298 2026-04-20T17:25:15Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2404299 wikitext text/x-wiki 940x9wq5wu7l9gjphzkgg2ir9ck5wpj 2404300 2404299 2026-04-20T17:25:46Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply 2404300 wikitext text/x-wiki 674rvaqatvldh3xc05dy8qg6a3qna0p 2404301 2404300 2026-04-20T17:26:16Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply 2404301 wikitext text/x-wiki sm8vaz5sqqclg203jnmfds7c0ai7og0 2404302 2404301 2026-04-20T17:26:48Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply 2404302 wikitext text/x-wiki 0sf1i3s4nfzqwt1no4it2ekck09th52 2404303 2404302 2026-04-20T17:27:38Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be1005.eqiad.wmnet with reason: host reimage 2404303 wikitext text/x-wiki phpr2foudu3tgho8dmnejszgge3zn2e 2404304 2404303 2026-04-20T17:27:44Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply 2404304 wikitext text/x-wiki f38ifn24q3j7zxb786yci2xvxnb9k9u 2404306 2404304 2026-04-20T17:34:58Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be1005.eqiad.wmnet with reason: host reimage 2404306 wikitext text/x-wiki opu3fipos3d5jh99jwb23ev8w2qbvh5 2404307 2404306 2026-04-20T17:35:41Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudelastic1012.eqiad.wmnet with OS trixie 2404307 wikitext text/x-wiki bf65hdwi3na1qdmntwgrkfi5coe5qrp 2404308 2404307 2026-04-20T17:36:38Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host phab2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2404308 wikitext text/x-wiki 88afepiz5ag2cmuiay2x4pfeq1efxky 2404309 2404308 2026-04-20T17:37:12Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host phab2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2404309 wikitext text/x-wiki qfw7fw8bw9l8l8dqwrvjcl3vi0m6ayd 2404310 2404309 2026-04-20T17:38:07Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host phab2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2404310 wikitext text/x-wiki dvswkh5o1kgt9e99y4vpmqjo7bcxwfv 2404311 2404310 2026-04-20T17:41:50Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply 2404311 wikitext text/x-wiki 41tj8bw17ue27i3f595pnatuhlhzo9o 2404312 2404311 2026-04-20T17:42:37Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply 2404312 wikitext text/x-wiki g6qnxvb366taml1p7xejm50fi1h6z6b 2404313 2404312 2026-04-20T17:43:09Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply 2404313 wikitext text/x-wiki 4wnkvsn38i1gevcnm3e6fz9mta7bnpo 2404314 2404313 2026-04-20T17:43:46Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply 2404314 wikitext text/x-wiki 2wnyhrecbtuceuep2g1z0885ldk4eyr 2404315 2404314 2026-04-20T17:44:18Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply 2404315 wikitext text/x-wiki oty0p2l6ya55zhw5a0wyukjv5pk2bw2 2404316 2404315 2026-04-20T17:44:31Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply 2404316 wikitext text/x-wiki 3hf2tnkstlz12iylxc1smtxalbmeodt 2404317 2404316 2026-04-20T17:45:02Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply 2404317 wikitext text/x-wiki b8vvbkx5a2ejz5d2elhkirjas391cpp 2404318 2404317 2026-04-20T17:45:18Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2404318 wikitext text/x-wiki pf808ar2i72k79oy7cwtb5gaoppfm1s 2404319 2404318 2026-04-20T17:45:49Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply 2404319 wikitext text/x-wiki krpvr8dticxtf8lhsylkvhecb87twom 2404320 2404319 2026-04-20T17:46:11Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply 2404320 wikitext text/x-wiki ktcz22lub3cjmoi3uv8wrlxeqeklttm 2404321 2404320 2026-04-20T17:46:42Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply 2404321 wikitext text/x-wiki 6otwsmaszhz79lsgzqxxin2cxn5u0y1 2404322 2404321 2026-04-20T17:47:56Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply 2404322 wikitext text/x-wiki 30s9sn1pavkznouv09qw2sh18rpybd9 2404324 2404322 2026-04-20T17:56:52Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be1005.eqiad.wmnet with OS bullseye 2404324 wikitext text/x-wiki fivdc7orsci05z68w2l98bxloy1sooq 2404326 2404324 2026-04-20T18:05:38Z Stashbot 7414 herron@dns1004: START - running authdns-update 2404326 wikitext text/x-wiki f53x8wf21395yl6359qz7gh8cg1m8su 2404327 2404326 2026-04-20T18:07:09Z Stashbot 7414 herron@dns1004: END - running authdns-update 2404327 wikitext text/x-wiki cx5c3p6zx2m7fb2xknvwc7qznpksck9 2404328 2404327 2026-04-20T18:11:32Z Stashbot 7414 Amir1: drop of langlinks table on testcommonswiki (T421914) 2404328 wikitext text/x-wiki 4lbk510ebihyv893xwsybq4wmnfxcxc 2404331 2404328 2026-04-20T18:25:33Z Stashbot 7414 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1275475|Attribution: Clean up API spec descriptions (T422502)]], [[gerrit:1275476|i18n: Use {{doc-markdown}} template in Attribution qqq.json (T422502)]], [[gerrit:1275477|Attribution: Documentation copyedits]], [[gerrit:1275488|Attribution: Update contact and add call to action (T422502)]], [[gerrit:1275478|Attribution: Add localized texts for tren 2404331 wikitext text/x-wiki 8nfm06t5izw6wf5lcz75w4vv6oyoq2b 2404339 2404331 2026-04-20T18:42:29Z Stashbot 7414 jforrester@deploy1003: pmiazga, jforrester: Backport for [[gerrit:1275475|Attribution: Clean up API spec descriptions (T422502)]], [[gerrit:1275476|i18n: Use {{doc-markdown}} template in Attribution qqq.json (T422502)]], [[gerrit:1275477|Attribution: Documentation copyedits]], [[gerrit:1275488|Attribution: Update contact and add call to action (T422502)]], [[gerrit:1275478|Attribution: Add localized texts for trending 2404339 wikitext text/x-wiki qws1sb7he6yirt7vl4xobpfwwf2oylj 2404340 2404339 2026-04-20T18:44:17Z Stashbot 7414 jforrester@deploy1003: pmiazga, jforrester: Continuing with sync 2404340 wikitext text/x-wiki l0bp862dry3bavlqhpe88rvey0jxwet 2404342 2404340 2026-04-20T18:55:56Z Stashbot 7414 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1275475|Attribution: Clean up API spec descriptions (T422502)]], [[gerrit:1275476|i18n: Use {{doc-markdown}} template in Attribution qqq.json (T422502)]], [[gerrit:1275477|Attribution: Documentation copyedits]], [[gerrit:1275488|Attribution: Update contact and add call to action (T422502)]], [[gerrit:1275478|Attribution: Add localized texts for tre 2404342 wikitext text/x-wiki efp2nqfv4b3blu4913oke9utm2zren0 2404343 2404342 2026-04-20T18:56:26Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.249.0" for 2 host(s) 2404343 wikitext text/x-wiki eve6ig39dphtk53uz0e66cd3j2h710u 2404344 2404343 2026-04-20T18:58:19Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.249.0" completed for 2 hosts 2404344 wikitext text/x-wiki cei1wyhe5fx47pfuhfkos03sa9p4w1z 2404345 2404344 2026-04-20T19:00:30Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2190: Security update 2404345 wikitext text/x-wiki smg5h2c6zuu9p4elkpe5f8scppi6aht 2404346 2404345 2026-04-20T19:04:22Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host phab2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2404346 wikitext text/x-wiki shhe3w8c33ytzrge7h3ae5q6fmit6mf 2404355 2404346 2026-04-20T19:28:46Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host phab2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2404355 wikitext text/x-wiki 3ejianlyrd9mzaj04tto1yny7itsmns 2404356 2404355 2026-04-20T19:46:10Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2190: Security update 2404356 wikitext text/x-wiki h8wf0pehpvay9rqri4l9mdnedmd2qjf 2404357 2404356 2026-04-20T19:48:21Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host phab2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2404357 wikitext text/x-wiki ai5ghzhkx08e2qkdoijwtvzl0g73nm7 2404358 2404357 2026-04-20T20:05:07Z Stashbot 7414 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1275428|Do not show donate button on affiliate wikis (T423876)]] 2404358 wikitext text/x-wiki 2p1lrewhspyz1iyeujnhfc15mmcv0kn 2404359 2404358 2026-04-20T20:08:39Z Stashbot 7414 aude@deploy1003: aude: Backport for [[gerrit:1275428|Do not show donate button on affiliate wikis (T423876)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2404359 wikitext text/x-wiki rc8ld0fur6l6kd1eeoq99rgsavmvxka 2404362 2404359 2026-04-20T20:10:16Z Stashbot 7414 aude@deploy1003: aude: Continuing with sync 2404362 wikitext text/x-wiki 4pphh3yd909ksvbza7ddmco3fknjmfr 2404363 2404362 2026-04-20T20:16:04Z Stashbot 7414 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1275428|Do not show donate button on affiliate wikis (T423876)]] (duration: 10m 57s) 2404363 wikitext text/x-wiki olfimyx4gd1q78qvgwx6hes7f2lmezq 2404369 2404363 2026-04-20T21:33:12Z Stashbot 7414 maryum: Deployed security fix for T299359 2404369 wikitext text/x-wiki dm50cmkw8zcrh4hjaqmgwj9kb8mt7n8 2404370 2404369 2026-04-20T21:42:55Z Stashbot 7414 maryum: Deployed security fix for T406954 2404370 wikitext text/x-wiki 1qno2oyfpvem01a6cc724tpcqce9qg0 2404371 2404370 2026-04-20T21:57:07Z Stashbot 7414 rzl@deploy1003: Started scap sync-world: https://gerrit.wikimedia.org/r/1275463 T423311 T423624 2404371 wikitext text/x-wiki 90eovnge546yeple6r95p15nysj18du 2404372 2404371 2026-04-20T21:59:16Z Stashbot 7414 rzl@deploy1003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1275463 T423311 T423624 (duration: 03m 24s) 2404372 wikitext text/x-wiki oysx361k30vwogmwvzpmxaopwgoxeht 2404379 2404372 2026-04-20T22:52:32Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host pc2022.codfw.wmnet with OS trixie 2404379 wikitext text/x-wiki 8zh6ft8g70mw9jdc0mfxefzpo980ppk 2404380 2404379 2026-04-20T22:52:50Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host pc2023.codfw.wmnet with OS trixie 2404380 wikitext text/x-wiki 1ns0rk7ohawb328ts1vm43rqznnud62 2404381 2404380 2026-04-20T22:53:01Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host pc2024.codfw.wmnet with OS trixie 2404381 wikitext text/x-wiki 43omlylh5z2anf99id1tcsg82y9sx7u 2404382 2404381 2026-04-20T23:07:00Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on pc2022.codfw.wmnet with reason: host reimage 2404382 wikitext text/x-wiki 8vvvruz5ehiij65mpi5viixyd627t9d 2404383 2404382 2026-04-20T23:07:01Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on pc2023.codfw.wmnet with reason: host reimage 2404383 wikitext text/x-wiki gl7dah6s3wcgtwm596hg99532dpf7ng 2404384 2404383 2026-04-20T23:07:03Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on pc2024.codfw.wmnet with reason: host reimage 2404384 wikitext text/x-wiki aeyfzite3vqrcdqyny56lq79d1zcfzy 2404385 2404384 2026-04-20T23:10:43Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1275540|Revert "Skin: Avoid stretching low resolution images" (T421524 T423676)]] 2404385 wikitext text/x-wiki danvv36iru2z01rx619cvjcl4dlqrwi 2404386 2404385 2026-04-20T23:12:43Z Stashbot 7414 jdlrobson@deploy1003: cscott, jdlrobson: Backport for [[gerrit:1275540|Revert "Skin: Avoid stretching low resolution images" (T421524 T423676)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2404386 wikitext text/x-wiki rtquw4hav1crs49yucadigfba1fx18u 2404387 2404386 2026-04-20T23:12:48Z Stashbot 7414 jdlrobson@deploy1003: cscott, jdlrobson: Continuing with sync 2404387 wikitext text/x-wiki 7yp99qmnb6okmva2jy1xi7qbbc37wwl 2404388 2404387 2026-04-20T23:14:29Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc2022.codfw.wmnet with reason: host reimage 2404388 wikitext text/x-wiki biugoj4zx9naeyo8w3z4p6899s3bb1n 2404389 2404388 2026-04-20T23:16:40Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1275540|Revert "Skin: Avoid stretching low resolution images" (T421524 T423676)]] (duration: 05m 56s) 2404389 wikitext text/x-wiki 4zn0hpz83ai3z2v4la57wl16pajixyg 2404390 2404389 2026-04-20T23:19:22Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc2023.codfw.wmnet with reason: host reimage 2404390 wikitext text/x-wiki cf77cuzsbr79bocpv8b71gsooe0axn5 2404391 2404390 2026-04-20T23:20:18Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1275547|[Mobile Page Previews] Avoid syntax error on older browsers (T423959)]] 2404391 wikitext text/x-wiki k0qiwj2u0411vpe7l9suud1lnmxl8ns 2404393 2404391 2026-04-20T23:21:50Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1275547|[Mobile Page Previews] Avoid syntax error on older browsers (T423959)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2404393 wikitext text/x-wiki 5eb6m4oxu979u853cofb6nx017z2463 2404395 2404393 2026-04-20T23:24:05Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc2024.codfw.wmnet with reason: host reimage 2404395 wikitext text/x-wiki j9qcvtybndh9u3cc1lal481qewkdle1 2404396 2404395 2026-04-20T23:24:39Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Continuing with sync 2404396 wikitext text/x-wiki h5oht12kr97l577illclpcnx8w7hvhr 2404397 2404396 2026-04-20T23:28:28Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1275547|[Mobile Page Previews] Avoid syntax error on older browsers (T423959)]] (duration: 08m 13s) 2404397 wikitext text/x-wiki fsha2oct0sqlolxngif5xvlxx65urvv 2404398 2404397 2026-04-20T23:29:13Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc2022.codfw.wmnet with OS trixie 2404398 wikitext text/x-wiki ca25xzxel66nww6wbovanhwaaj4zxdu 2404399 2404398 2026-04-20T23:32:24Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1271862|Restore PageImages functionality to Wikisources and Wikibooks (T417538)]] 2404399 wikitext text/x-wiki 70tptqd8qt4nctojn3g2cnce1e76otr 2404400 2404399 2026-04-20T23:34:03Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson, ignaciorodrguez: Backport for [[gerrit:1271862|Restore PageImages functionality to Wikisources and Wikibooks (T417538)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2404400 wikitext text/x-wiki qwmp9buirp9lek2nsoaac5piyj3njx4 2404401 2404400 2026-04-20T23:34:19Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc2023.codfw.wmnet with OS trixie 2404401 wikitext text/x-wiki e6snevdb9i1u7k4an35sey1bhz7qs2z 2404402 2404401 2026-04-20T23:36:25Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson, ignaciorodrguez: Continuing with sync 2404402 wikitext text/x-wiki rgrjegz7q8bf6qaml6bek79bn6wbxim 2404403 2404402 2026-04-20T23:39:13Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc2024.codfw.wmnet with OS trixie 2404403 wikitext text/x-wiki l6ijl67o4s7izxwfbkunzu5vs5fh6xm 2404404 2404403 2026-04-20T23:40:16Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1271862|Restore PageImages functionality to Wikisources and Wikibooks (T417538)]] (duration: 07m 47s) 2404404 wikitext text/x-wiki 1om4yhxjq5a57sezdbbzxqdzc139u2x 2404407 2404404 2026-04-21T00:28:10Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2250.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2404407 wikitext text/x-wiki 5myzykq039im3ygmmu2rbs5qzpt7idt 2404408 2404407 2026-04-21T00:28:36Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2251.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2404408 wikitext text/x-wiki 6em675fhx2bur1uag93bs2rvg9i4tbt 2404410 2404408 2026-04-21T00:29:00Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2252.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2404410 wikitext text/x-wiki 6e5bjwk1jthn0etcr4hfthfhe6ej7ak 2404411 2404410 2026-04-21T00:29:26Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2253.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2404411 wikitext text/x-wiki 0t7p0uhdol480a4zg6ojnuw0w7zc528 2404413 2404411 2026-04-21T00:32:30Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2251.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2404413 wikitext text/x-wiki mw0zgwaqdgvflzz3ha6uroc9zq41np5 2404414 2404413 2026-04-21T00:33:15Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2252.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2404414 wikitext text/x-wiki iysmd3wak5v6nthcl1jmg2jsezbcg24 2404415 2404414 2026-04-21T00:33:17Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2253.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2404415 wikitext text/x-wiki qo84fucpzkrkm4kfgcdo273iftffy4k 2404416 2404415 2026-04-21T00:34:23Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2250.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2404416 wikitext text/x-wiki ei93dvskdyqq87w41rc9lw7j30xtn83 2404417 2404416 2026-04-21T00:35:10Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host db2250.codfw.wmnet with OS bookworm 2404417 wikitext text/x-wiki o0uyichyrzfjvpdjq1egc2pnodujxq7 2404418 2404417 2026-04-21T00:35:35Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host db2251.codfw.wmnet with OS bookworm 2404418 wikitext text/x-wiki dt8zorajnrilh3725vj10eq9cuz6gf5 2404419 2404418 2026-04-21T00:35:45Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host db2252.codfw.wmnet with OS bookworm 2404419 wikitext text/x-wiki gkid4kp72skjim2tbbjfbfrls6ma7kg 2404420 2404419 2026-04-21T00:36:00Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host db2253.codfw.wmnet with OS bookworm 2404420 wikitext text/x-wiki pui1x25i53eplkdr4ya3oev6oqnwkzu 2404422 2404420 2026-04-21T00:51:58Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2253.codfw.wmnet with reason: host reimage 2404422 wikitext text/x-wiki 4gvvhki21y33xeivw4wbbd38pjvqtxs 2404423 2404422 2026-04-21T00:52:15Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2252.codfw.wmnet with reason: host reimage 2404423 wikitext text/x-wiki owssgxb4ic53r6p2mx2hj2u1uxz1r1z 2404424 2404423 2026-04-21T00:52:24Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2251.codfw.wmnet with reason: host reimage 2404424 wikitext text/x-wiki 9d2l3tfmh4z0bmay5x3hky17qjkiyq6 2404425 2404424 2026-04-21T00:52:25Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2250.codfw.wmnet with reason: host reimage 2404425 wikitext text/x-wiki 60r9vy1sp8hhnzp91xc68oo8pnrsp60 2404426 2404425 2026-04-21T00:59:57Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2253.codfw.wmnet with reason: host reimage 2404426 wikitext text/x-wiki dnzj51vkq9dasm1wmnjd1fb5h9bo58g 2404427 2404426 2026-04-21T01:02:55Z Stashbot 7414 zabe: marked 543 revisions as bad # T393237 2404427 wikitext text/x-wiki 95nkkktp4akmjs3l52cucqyeewdsz02 2404428 2404427 2026-04-21T01:03:35Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2252.codfw.wmnet with reason: host reimage 2404428 wikitext text/x-wiki 3zf43ykzp6u881u15n47qpfudler9lr 2404429 2404428 2026-04-21T01:07:09Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2251.codfw.wmnet with reason: host reimage 2404429 wikitext text/x-wiki bce80cfaie6xw96qyjhox26m7v7zbuz 2404430 2404429 2026-04-21T01:12:01Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2250.codfw.wmnet with reason: host reimage 2404430 wikitext text/x-wiki nfojfj7t8jkwmjes4k2c4pq6sakmzz6 2404431 2404430 2026-04-21T01:17:17Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" 2404431 wikitext text/x-wiki 9kd46hoj8jvubnvz8xdbk2xa7s0p8hf 2404432 2404431 2026-04-21T01:18:16Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" 2404432 wikitext text/x-wiki 7oqfpv4dym97os3cuevtdzzx50czjhe 2404433 2404432 2026-04-21T01:18:18Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2253.codfw.wmnet with OS bookworm 2404433 wikitext text/x-wiki p6dcfhsnqzq6e3axj4mbuop444uam0n 2404434 2404433 2026-04-21T01:20:46Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" 2404434 wikitext text/x-wiki 3z2yfi3z4kky1za1r394r1s4tadjrbc 2404435 2404434 2026-04-21T01:23:36Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" 2404435 wikitext text/x-wiki fs65g2rqty6of7poohuicy0g0udxl2v 2404436 2404435 2026-04-21T01:23:38Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2252.codfw.wmnet with OS bookworm 2404436 wikitext text/x-wiki 2kgqh76tc78ogrbcj6c88ayg1irg4we 2404437 2404436 2026-04-21T01:24:08Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" 2404437 wikitext text/x-wiki jty39at70zgqs3nd83vm5psjyfcxyhb 2404438 2404437 2026-04-21T01:24:24Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" 2404438 wikitext text/x-wiki evxkxwhrazdq0z1v1agafxjr165urnj 2404439 2404438 2026-04-21T01:24:26Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2251.codfw.wmnet with OS bookworm 2404439 wikitext text/x-wiki 4ikleb6imrfp96z700rld5k17iwk0u8 2404440 2404439 2026-04-21T01:28:56Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" 2404440 wikitext text/x-wiki egosohmpy4wauy50u9wr4hib8ecap5a 2404441 2404440 2026-04-21T01:32:02Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" 2404441 wikitext text/x-wiki k3xlfrbm3f5o2tijn04ewi9v5eou5hh 2404442 2404441 2026-04-21T01:32:11Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db2250.codfw.wmnet with OS bookworm 2404442 wikitext text/x-wiki riyfge233haw1ifwr2lvi9pyuf40me7 2404443 2404442 2026-04-21T02:01:11Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2404443 wikitext text/x-wiki ghjemii91ict6nvclje7rothblj70rs 2404444 2404443 2026-04-21T02:07:14Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 03s) 2404444 wikitext text/x-wiki 7197h9z9m4j4lesc0079v68ec3s89f3 2404445 2404444 2026-04-21T02:22:23Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T410589)', diff saved to https://phabricator.wikimedia.org/P91238 and previous config saved to /var/cache/conftool/dbconfig/20260421-022219-ladsgroup.json 2404445 wikitext text/x-wiki 9o8f3563mgyee5gd883vbwwjuqtnxni 2404447 2404445 2026-04-21T02:32:30Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P91239 and previous config saved to /var/cache/conftool/dbconfig/20260421-023228-ladsgroup.json 2404447 wikitext text/x-wiki l6zs9kcdh3bysqbf902ka30woucqmo8 2404448 2404447 2026-04-21T02:42:38Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P91240 and previous config saved to /var/cache/conftool/dbconfig/20260421-024237-ladsgroup.json 2404448 wikitext text/x-wiki p4xav8fhphvpq76lqh2ibp55dcgn697 2404449 2404448 2026-04-21T02:52:50Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T410589)', diff saved to https://phabricator.wikimedia.org/P91241 and previous config saved to /var/cache/conftool/dbconfig/20260421-025245-ladsgroup.json 2404449 wikitext text/x-wiki mq336ney6bidoykyaz23y28xs8v7ww0 2404450 2404449 2026-04-21T02:53:04Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance 2404450 wikitext text/x-wiki q04pj4ipw78neqmz3h1873j1acs7knb 2404451 2404450 2026-04-21T02:53:14Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2154 (T410589)', diff saved to https://phabricator.wikimedia.org/P91242 and previous config saved to /var/cache/conftool/dbconfig/20260421-025311-ladsgroup.json 2404451 wikitext text/x-wiki 1dsidljj2q48cdg61hwqxkbkldcqtj4 2404454 2404451 2026-04-21T04:02:34Z Stashbot 7414 mwpresync@deploy1003: Pruned MediaWiki: 1.46.0-wmf.22 (duration: 02m 30s) 2404454 wikitext text/x-wiki j424z7zmtdktal0n4i52e70tzb6mr1n 2404457 2404454 2026-04-21T05:25:35Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb1025.eqiad.wmnet with reason: Clone s6 2404457 wikitext text/x-wiki l74y3f25d75q1zbekr0zop1v1d20x5m 2404458 2404457 2026-04-21T05:27:05Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb1015.eqiad.wmnet with reason: Clone s6 to clouddb1025 2404458 wikitext text/x-wiki 1hn1hfy7lpx7bszpydwm5eniqe84xj4 2404459 2404458 2026-04-21T05:40:08Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2404459 wikitext text/x-wiki 1y56xg3008mdaknlmzpe25bfh2xpmh8 2404460 2404459 2026-04-21T06:11:34Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2144.codfw.wmnet,db1151.eqiad.wmnet with reason: Reimage to Trixie 2404460 wikitext text/x-wiki crchyg04a5vnqekodmf0y3wn04573ge 2404461 2404460 2026-04-21T06:12:01Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2144.codfw.wmnet with reason: Reimage to Trixie 2404461 wikitext text/x-wiki 44tqgjsjiaug9ht85lp08epyiom656l 2404462 2404461 2026-04-21T06:12:07Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2144: Reimage to Trixie 2404462 wikitext text/x-wiki 1xem7wl60en1oj2avzlo2unjfta11fw 2404463 2404462 2026-04-21T06:12:09Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2404463 wikitext text/x-wiki 8syzlkv2s1cgkfmu2mdgnubc56g345g 2404464 2404463 2026-04-21T06:12:13Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2404464 wikitext text/x-wiki rgvik61obdujczq56er7b00kvewenut 2404465 2404464 2026-04-21T06:12:15Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2144: Reimage to Trixie 2404465 wikitext text/x-wiki 50njpnmk1kuxuxwiupiwy3flrlo7fs4 2404466 2404465 2026-04-21T06:12:36Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2144.codfw.wmnet with OS trixie 2404466 wikitext text/x-wiki 9lq8qrjoa9tdl54j1212tdcggdvgcni 2404467 2404466 2026-04-21T06:33:12Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2144.codfw.wmnet with reason: host reimage 2404467 wikitext text/x-wiki doy73tke0k8evajtyytrrqmld6rhhp8 2404468 2404467 2026-04-21T06:38:44Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2144.codfw.wmnet with reason: host reimage 2404468 wikitext text/x-wiki go58r3jekrfe94dmdk87zxve08rtd81 2404469 2404468 2026-04-21T07:02:25Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2144.codfw.wmnet with OS trixie 2404469 wikitext text/x-wiki 9qpaidojzh5sxn782xgufh7e8ril9dz 2404470 2404469 2026-04-21T07:03:26Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2144: after reimage to trixie 2404470 wikitext text/x-wiki 7zlc01nrux8jz6q86demmhuqtt47c74 2404471 2404470 2026-04-21T07:03:28Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2144: after reimage to trixie 2404471 wikitext text/x-wiki 4v4b47cc503ju5hviebgipe3ya5z09h 2404472 2404471 2026-04-21T07:04:17Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2144: After reimage 2404472 wikitext text/x-wiki k54w1ecimjsvgzcdwwroe0gmz2jsbqs 2404473 2404472 2026-04-21T07:04:19Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2404473 wikitext text/x-wiki 2pulu14a58qy5v76zapd6tr0vzipdhs 2404474 2404473 2026-04-21T07:04:44Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2404474 wikitext text/x-wiki 3l7utl7bhswj96mvagcvjjgouggjb51 2404475 2404474 2026-04-21T07:04:46Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2144: After reimage 2404475 wikitext text/x-wiki s53te15hxr817nxtiq440psmse9707e 2404476 2404475 2026-04-21T07:05:32Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc2011: Cloning pc2021 2404476 wikitext text/x-wiki sdcfv4zq76laqwjw6owls02i6d4s1li 2404477 2404476 2026-04-21T07:05:34Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2404477 wikitext text/x-wiki 7m1g7y0t3n7ecao0v28oef6xlobpm31 2404478 2404477 2026-04-21T07:05:40Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2404478 wikitext text/x-wiki tbqpn7w2n9robjqdyenrfemh9xsn5zv 2404479 2404478 2026-04-21T07:05:42Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc2011: Cloning pc2021 2404479 wikitext text/x-wiki 5d13e4qk9yyrevlg328znw0lfyolpv8 2404480 2404479 2026-04-21T07:06:20Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2011.codfw.wmnet,pc1011.eqiad.wmnet with reason: Cloning pc2021 from pc2011 2404480 wikitext text/x-wiki l828q6m32kt21h9lko5fvkxn0t14wzw 2404481 2404480 2026-04-21T07:16:12Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host thanos-be1006.eqiad.wmnet with OS bullseye 2404481 wikitext text/x-wiki dhzvurxvbrxyal4w80hy0314wu5zk0r 2404482 2404481 2026-04-21T07:17:17Z Stashbot 7414 mvernon@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be1064.eqiad.wmnet with reason: vacuum overlarge container dbs 2404482 wikitext text/x-wiki akzyiyntwi5b9ou6beq9s8wu7uwxb9r 2404483 2404482 2026-04-21T07:38:31Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be1006.eqiad.wmnet with reason: host reimage 2404483 wikitext text/x-wiki 6kk6qgk4lrmvrxnp96f6ra7xn2uu1j5 2404484 2404483 2026-04-21T07:39:49Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1011: Reimage to Trixie 2404484 wikitext text/x-wiki ergbjyub8p2nk4t7028mkb5bmf95xp8 2404485 2404484 2026-04-21T07:39:56Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2404485 wikitext text/x-wiki 962olm9i1x8s5k102qjr57iiosau2xd 2404486 2404485 2026-04-21T07:39:58Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2404486 wikitext text/x-wiki 8owr365hg1l2u3pedyq19o877ttp0wl 2404487 2404486 2026-04-21T07:39:59Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1011: Reimage to Trixie 2404487 wikitext text/x-wiki lzcaybuj65beqjjqysly9xiudt2jvqu 2404488 2404487 2026-04-21T07:45:51Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be1006.eqiad.wmnet with reason: host reimage 2404488 wikitext text/x-wiki e4a4s8za1p6rmxyr6216rz5tq3pijmn 2404489 2404488 2026-04-21T07:46:23Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.decommission for hosts wikikube-worker[1002-1005,1011-1012,1019-1020,1029-1031,1058-1061].eqiad.wmnet 2404489 wikitext text/x-wiki 1fqyynk7otwlwq0cfa8ajuj5bmenymo 2404490 2404489 2026-04-21T07:46:40Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.decommission for hosts wikikube-worker[1062-1063,1082-1083,1088-1092,1096-1101].eqiad.wmnet 2404490 wikitext text/x-wiki o2ijdooegkd0n34aymfumfja85k3g84 2404491 2404490 2026-04-21T07:46:47Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.decommission for hosts wikikube-worker[1102-1112,1166-1168].eqiad.wmnet 2404491 wikitext text/x-wiki kgssdt9tkadigvspilctdwp9zjp3qxt 2404492 2404491 2026-04-21T07:48:17Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.remove-downtime for ms-be1064.eqiad.wmnet 2404492 wikitext text/x-wiki t8jvxdzl5zzq772q1w040swko9t43sv 2404493 2404492 2026-04-21T07:48:19Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be1064.eqiad.wmnet 2404493 wikitext text/x-wiki rckgmq7zn891rpq7s9m8hlaix3sbcwt 2404494 2404493 2026-04-21T07:49:15Z Stashbot 7414 fnegri@cumin1003: conftool action : set/weight=100; selector: name=clouddb1025.eqiad.wmnet,service=s6 2404494 wikitext text/x-wiki 4n2y2fuqcvy8d6iz0xowlgo7vgox04r 2404495 2404494 2026-04-21T07:49:29Z Stashbot 7414 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1025.eqiad.wmnet,service=s6 2404495 wikitext text/x-wiki nvrsvjbhfr8ql0h3xgc9lbm2hb57ad0 2404496 2404495 2026-04-21T07:51:17Z Stashbot 7414 fnegri@cumin1003: conftool action : set/weight=100; selector: name=clouddb1025.eqiad.wmnet,service=s4 2404496 wikitext text/x-wiki 6orei7f7s15elurlehfc0u5zmnucj6i 2404497 2404496 2026-04-21T07:51:21Z Stashbot 7414 fnegri@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1025.eqiad.wmnet,service=s4 2404497 wikitext text/x-wiki rtk5uqxkf5xfcc6jq2h9qxi0xl5ussl 2404498 2404497 2026-04-21T08:03:07Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1159.eqiad.wmnet with reason: Maintenance 2404498 wikitext text/x-wiki 3ha8rselyezm82jm4im4q3aw267szny 2404499 2404498 2026-04-21T08:03:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1159 (T419635)', diff saved to https://phabricator.wikimedia.org/P91248 and previous config saved to /var/cache/conftool/dbconfig/20260421-080314-fceratto.json 2404499 wikitext text/x-wiki c2t1f6b742ix5justhlzfqdn7qb3b62 2404501 2404499 2026-04-21T08:04:33Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance 2404501 wikitext text/x-wiki tixebfknk54xnz3ccd0vpd54y5rue3q 2404502 2404501 2026-04-21T08:06:19Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be1006.eqiad.wmnet with OS bullseye 2404502 wikitext text/x-wiki 635bczq5sww8ngq2xdhoq5tvvwjwnjh 2404503 2404502 2026-04-21T08:08:03Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet 2404503 wikitext text/x-wiki fcm1fmy0skjp1rp0xh10279gw9hwtv5 2404504 2404503 2026-04-21T08:09:30Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance 2404504 wikitext text/x-wiki g6t8v5lxefsv4ootom6xqdhynter5cm 2404505 2404504 2026-04-21T08:09:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1157 (T419961)', diff saved to https://phabricator.wikimedia.org/P91249 and previous config saved to /var/cache/conftool/dbconfig/20260421-080936-fceratto.json 2404505 wikitext text/x-wiki 49p79e790bz4jmm8kgtkgntsep046fh 2404506 2404505 2026-04-21T08:10:57Z Stashbot 7414 musikanimal@deploy1003: Started scap sync-world: Backport for [[gerrit:1275760|ext.abuseFilter.edit.js: temporary locking of CodeMirror lineWrapping (T423773 T423756)]] 2404506 wikitext text/x-wiki 4qjkurcnkyhti7nfbapj2nllhnsx5tw 2404508 2404506 2026-04-21T08:12:38Z Stashbot 7414 musikanimal@deploy1003: musikanimal: Backport for [[gerrit:1275760|ext.abuseFilter.edit.js: temporary locking of CodeMirror lineWrapping (T423773 T423756)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2404508 wikitext text/x-wiki haa35v5to06tjs92og6af3vy78d3ehf 2404509 2404508 2026-04-21T08:14:05Z Stashbot 7414 musikanimal@deploy1003: musikanimal: Continuing with sync 2404509 wikitext text/x-wiki q9ldtma3hhqbw3xft0stga874dt2suz 2404510 2404509 2026-04-21T08:14:52Z Stashbot 7414 elukey: bootstrapping pki intermediate discovery2026 2404510 wikitext text/x-wiki d1fcmcggccehvyffwf12y553yscm9rj 2404511 2404510 2026-04-21T08:17:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T419961)', diff saved to https://phabricator.wikimedia.org/P91250 and previous config saved to /var/cache/conftool/dbconfig/20260421-081717-fceratto.json 2404511 wikitext text/x-wiki drshici5p4ddtlotu1qifda6kn968rn 2404512 2404511 2026-04-21T08:18:04Z Stashbot 7414 musikanimal@deploy1003: Finished scap sync-world: Backport for [[gerrit:1275760|ext.abuseFilter.edit.js: temporary locking of CodeMirror lineWrapping (T423773 T423756)]] (duration: 07m 01s) 2404512 wikitext text/x-wiki qm5hzo3z3s4sc0fm6rgwa4pqsnpmts3 2404513 2404512 2026-04-21T08:18:10Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet 2404513 wikitext text/x-wiki ni5xrkfdi81vhfh3ip2lfwcms1tlybw 2404514 2404513 2026-04-21T08:20:21Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1157: Security update 2404514 wikitext text/x-wiki qf9luaz0ugbtwtvi9a1ieljah0p72cc 2404517 2404514 2026-04-21T08:27:33Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.dns.netbox 2404517 wikitext text/x-wiki 83e88nvdlq3dszzef4f5sx83sbea1p7 2404518 2404517 2026-04-21T08:31:35Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker[1102-1112,1166-1168].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1003" 2404518 wikitext text/x-wiki bntsvit3qnoabpug84fj5d25lm0ov8d 2404519 2404518 2026-04-21T08:32:05Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker[1102-1112,1166-1168].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1003" 2404519 wikitext text/x-wiki isk7c6carwma2d115dom96c59zom3gv 2404520 2404519 2026-04-21T08:32:07Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2404520 wikitext text/x-wiki s62ox45x08ci341kvo80tbzx2k4lieu 2404521 2404520 2026-04-21T08:32:09Z Stashbot 7414 jayme@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts wikikube-worker[1102-1112,1166-1168].eqiad.wmnet 2404521 wikitext text/x-wiki nuonp4szw0r3w44wzxkl0c7gojawln3 2404522 2404521 2026-04-21T08:32:31Z Stashbot 7414 moritzm: installing gst-plugins-base1.0 security updates 2404522 wikitext text/x-wiki 3596gr416ph4mbo8tjlpujc2phqrdwd 2404523 2404522 2026-04-21T08:32:41Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.dns.netbox 2404523 wikitext text/x-wiki rfcg8um9esf5sawpwt7c2r72kt1bq3q 2404524 2404523 2026-04-21T08:38:12Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.dns.netbox 2404524 wikitext text/x-wiki 7hi6g0jx6hz984fnbskq8jiatq9vm59 2404525 2404524 2026-04-21T08:39:07Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker[1062-1063,1082-1083,1088-1092,1096-1101].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1003" 2404525 wikitext text/x-wiki peaghul74m926fv0zxulzcieoxnsum4 2404526 2404525 2026-04-21T08:39:12Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker[1062-1063,1082-1083,1088-1092,1096-1101].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1003" 2404526 wikitext text/x-wiki 8tdelrz58qciti1ltzw9x6f6apzx3is 2404527 2404526 2026-04-21T08:39:14Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2404527 wikitext text/x-wiki 1ih1al3vtvrjmm4sfp6jsy2etuirm6h 2404528 2404527 2026-04-21T08:39:16Z Stashbot 7414 jayme@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts wikikube-worker[1062-1063,1082-1083,1088-1092,1096-1101].eqiad.wmnet 2404528 wikitext text/x-wiki m3b8ej0wx59ojbh076oefemb8jhtmny 2404529 2404528 2026-04-21T08:40:56Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2404529 wikitext text/x-wiki 1xsf6gqmcmfmqhw7p976ck5ivtex120 2404530 2404529 2026-04-21T08:40:59Z Stashbot 7414 jayme@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts wikikube-worker[1002-1005,1011-1012,1019-1020,1029-1031,1058-1061].eqiad.wmnet 2404530 wikitext text/x-wiki 27hepdkggz8ae00wu86hr6yvtps3g2e 2404532 2404530 2026-04-21T08:43:48Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5006.eqsin.wmnet to cluster eqsin02 and group 01 2404532 wikitext text/x-wiki qy9ismn5nik188g85u5gnbvdmgv5leo 2404533 2404532 2026-04-21T08:45:04Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2143.codfw.wmnet,db1153.eqiad.wmnet with reason: Reimage to Trixie 2404533 wikitext text/x-wiki 20fajyco2omoo5gvg40abb9mn0fhntu 2404534 2404533 2026-04-21T08:45:25Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1153.eqiad.wmnet with reason: Reimage to Trixie 2404534 wikitext text/x-wiki clcm1glk030qk9s3l0pf0wejz5y6fsj 2404535 2404534 2026-04-21T08:45:30Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1153: Reimage to Trixie 2404535 wikitext text/x-wiki 3x7abhq4jc2awslecv3l5t22reyvtg0 2404536 2404535 2026-04-21T08:45:32Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2404536 wikitext text/x-wiki j5u4j6qzfpe99u8n4fant6yanp7dvt1 2404537 2404536 2026-04-21T08:45:37Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2404537 wikitext text/x-wiki kzkvrv4qdrto7jowgfciuu66y5pq4w6 2404538 2404537 2026-04-21T08:45:40Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1153: Reimage to Trixie 2404538 wikitext text/x-wiki 89xlfc912k8oaatlft44u2rwiuom1ex 2404539 2404538 2026-04-21T08:46:01Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1153.eqiad.wmnet with OS trixie 2404539 wikitext text/x-wiki 6a17rbf8ah64bxva65cy334ng9wtjrg 2404540 2404539 2026-04-21T08:46:23Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti5006.eqsin.wmnet to cluster eqsin02 and group 01 2404540 wikitext text/x-wiki bhcmpoycnde6xny5ksz5pdeexprscld 2404541 2404540 2026-04-21T08:50:42Z Stashbot 7414 jayme: homer 'cr*eqiad*' commit - T423863 2404541 wikitext text/x-wiki 8pzo9u0rdi7qusdsjtqo9psgm2p8pf7 2404542 2404541 2026-04-21T08:51:56Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1157: Security update 2404542 wikitext text/x-wiki 699kp107lgennadzwmoztf4wd9lf7d0 2404544 2404542 2026-04-21T08:56:28Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2404544 wikitext text/x-wiki 2ybzppm8kfgpt3rrv9ytf63z0aeubmy 2404546 2404544 2026-04-21T09:00:43Z Stashbot 7414 jayme: homer 'asw2-b-eqiad.mgmt.eqiad.wmnet' commit - T423863 2404546 wikitext text/x-wiki eos4zqplhodguxcnjj34ohlpa5zhydv 2404547 2404546 2026-04-21T09:00:48Z Stashbot 7414 jayme: homer 'asw2-a-eqiad.mgmt.eqiad.wmnet' commit - T423863 2404547 wikitext text/x-wiki if0ola9gum8ufcf8crfey4m7jrs922y 2404548 2404547 2026-04-21T09:01:30Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1153.eqiad.wmnet with reason: host reimage 2404548 wikitext text/x-wiki 14me9vnuww2mkcwbs64pi6x694zbr9g 2404549 2404548 2026-04-21T09:03:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T419635)', diff saved to https://phabricator.wikimedia.org/P91255 and previous config saved to /var/cache/conftool/dbconfig/20260421-090336-fceratto.json 2404549 wikitext text/x-wiki jevsnalnha8hahjybbwe9osvqxeivk3 2404550 2404549 2026-04-21T09:03:52Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance 2404550 wikitext text/x-wiki f5kl1f203pzgb160ea1shrf1v1k6obu 2404551 2404550 2026-04-21T09:04:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1175 (T419961)', diff saved to https://phabricator.wikimedia.org/P91256 and previous config saved to /var/cache/conftool/dbconfig/20260421-090358-fceratto.json 2404551 wikitext text/x-wiki tt80hiocjc4ic123emovxfe0fd1d93p 2404552 2404551 2026-04-21T09:05:30Z Stashbot 7414 fabfur: restarting pybal on lvs1019-1020 to clear alerts 2404552 wikitext text/x-wiki sgklg6ckn5m66d3kr999mv0ct15e7dz 2404553 2404552 2026-04-21T09:05:44Z Stashbot 7414 jayme: kubectl delete node $(nodeset -e wikikube-worker[1002-1005,1011-1012,1019-1020,1029-1031,1058-1063,1082-1083,1088-1092,1096-1112,1166-1168].eqiad.wmnet) - T423863 2404553 wikitext text/x-wiki 44c227fcexwixb8ptd5npme4vxgu87a 2404554 2404553 2026-04-21T09:05:55Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance 2404554 wikitext text/x-wiki c3lp42pxbqsnj087l25wrb1t1yiby1s 2404555 2404554 2026-04-21T09:11:05Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1153.eqiad.wmnet with reason: host reimage 2404555 wikitext text/x-wiki b816vmvsz6z3ondrestmym35yw806zz 2404556 2404555 2026-04-21T09:11:18Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance 2404556 wikitext text/x-wiki 550ym3dw1t6i8a2qvedhacyu22od36g 2404557 2404556 2026-04-21T09:11:28Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1198 (T419961)', diff saved to https://phabricator.wikimedia.org/P91257 and previous config saved to /var/cache/conftool/dbconfig/20260421-091124-fceratto.json 2404557 wikitext text/x-wiki lmkb0k9061a1jmzpj6ruv3dc2g1ar8b 2404558 2404557 2026-04-21T09:13:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P91258 and previous config saved to /var/cache/conftool/dbconfig/20260421-091344-fceratto.json 2404558 wikitext text/x-wiki p10a4nor9896ktmervpzt6gif9uk0r1 2404559 2404558 2026-04-21T09:17:07Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2404559 wikitext text/x-wiki qna881rygzmmculca8mzn8po040jvih 2404560 2404559 2026-04-21T09:19:51Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T419961)', diff saved to https://phabricator.wikimedia.org/P91259 and previous config saved to /var/cache/conftool/dbconfig/20260421-091949-fceratto.json 2404560 wikitext text/x-wiki 1a5rgt2lop48r4ictgss5ccmu5frbje 2404561 2404560 2026-04-21T09:21:58Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db1198: Security update 2404561 wikitext text/x-wiki oh4o4xlot8vcvkryv5fuw7iqnruepnq 2404562 2404561 2026-04-21T09:23:54Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P91260 and previous config saved to /var/cache/conftool/dbconfig/20260421-092352-fceratto.json 2404562 wikitext text/x-wiki m57ad50brxzgd8c8wnu9kj9531bja7p 2404563 2404562 2026-04-21T09:26:51Z Stashbot 7414 moritzm: imported debdeploy 0.0.99.15 for trixie-wikimedia (compat release for Cumin 6) 2404563 wikitext text/x-wiki 5wrl5fsvh0f5en2qpvmgl3plxiufxs9 2404564 2404563 2026-04-21T09:34:06Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T419635)', diff saved to https://phabricator.wikimedia.org/P91262 and previous config saved to /var/cache/conftool/dbconfig/20260421-093401-fceratto.json 2404564 wikitext text/x-wiki js3tq4ngg6fhi6o7fa7tuippspx814x 2404565 2404564 2026-04-21T09:34:09Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1153.eqiad.wmnet with OS trixie 2404565 wikitext text/x-wiki 29fc4fyledva008psvyk1s5yt58jhnd 2404566 2404565 2026-04-21T09:34:22Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1161.eqiad.wmnet with reason: Maintenance 2404566 wikitext text/x-wiki jozqrpiiu0eamqq8366ytugb4p8374z 2404567 2404566 2026-04-21T09:34:32Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance 2404567 wikitext text/x-wiki db5n1xy0gkkk1gigpodnion60pqazhj 2404568 2404567 2026-04-21T09:35:06Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1153: after reimage to trixie 2404568 wikitext text/x-wiki 3wj4zbjqb9xzms83mi6vtn5u9kh80b6 2404569 2404568 2026-04-21T09:35:08Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db1153: after reimage to trixie 2404569 wikitext text/x-wiki rqqyn6povis2g2xusluh3nyfgr3axfd 2404570 2404569 2026-04-21T09:37:12Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1153: repool after maintenance 2404570 wikitext text/x-wiki bk2sgpn49t1imuyxf9o7c7tnlivu109 2404571 2404570 2026-04-21T09:37:14Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db1153: repool after maintenance 2404571 wikitext text/x-wiki 348h4tn68kom71rnsszsu0bfoqcelzw 2404572 2404571 2026-04-21T09:38:03Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2404572 wikitext text/x-wiki 7pso048b3g1s350vn5ylqy8ctoxccma 2404573 2404572 2026-04-21T09:39:10Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1153: repool after maintenance 2404573 wikitext text/x-wiki f0ue389jhk6lumpssepnd7lfdp89gsz 2404574 2404573 2026-04-21T09:39:12Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2404574 wikitext text/x-wiki m79yrnvdwtgmrbedwz2dzysuryz53sy 2404575 2404574 2026-04-21T09:39:24Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2404575 wikitext text/x-wiki 5pqfu85h0igolcvxdgnlbjl6ccbdpkp 2404576 2404575 2026-04-21T09:39:26Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1153: repool after maintenance 2404576 wikitext text/x-wiki 5yoivb402e7jepjkumczw0g8c4xf69m 2404577 2404576 2026-04-21T09:45:56Z Stashbot 7414 moritzm: updating debdeploy on trixie to 0.0.99.15 2404577 wikitext text/x-wiki rdexn5c7vakly1qndz3da4rv8hyqxjg 2404578 2404577 2026-04-21T09:47:28Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2404578 wikitext text/x-wiki 093ls3w565hx9hnecrhve5d9a3pl1z5 2404579 2404578 2026-04-21T09:50:58Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host tcp-proxy5003.eqsin.wmnet 2404579 wikitext text/x-wiki qlf78qjxfp51vru4xnyedtxtl5gejf9 2404580 2404579 2026-04-21T09:50:59Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2404580 wikitext text/x-wiki 2uav88mn3tvodasyaaklvgte2laflyq 2404581 2404580 2026-04-21T09:54:40Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM tcp-proxy5003.eqsin.wmnet - jmm@cumin2002" 2404581 wikitext text/x-wiki p8puv3hk7a1plgqfhst08yh586wvjf3 2404582 2404581 2026-04-21T09:54:46Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM tcp-proxy5003.eqsin.wmnet - jmm@cumin2002" 2404582 wikitext text/x-wiki mfq3jg3wnocm0awhrbq2vx2zke5ysaw 2404583 2404582 2026-04-21T09:54:48Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2404583 wikitext text/x-wiki jcbsvueqj3lilotvb5qhfa323em1779 2404584 2404583 2026-04-21T09:54:50Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache tcp-proxy5003.eqsin.wmnet on all recursors 2404584 wikitext text/x-wiki o54stpfyzygvtaqbklvtc2lyuc4jthi 2404585 2404584 2026-04-21T09:54:52Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) tcp-proxy5003.eqsin.wmnet on all recursors 2404585 wikitext text/x-wiki ljjezes2o2umqs07m1ue2gvlr1w17c7 2404586 2404585 2026-04-21T09:54:54Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1198: Security update 2404586 wikitext text/x-wiki a6gc7gc7nf6fkzsmvrnsbvmjsm5fg0m 2404587 2404586 2026-04-21T09:55:24Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM tcp-proxy5003.eqsin.wmnet - jmm@cumin2002" 2404587 wikitext text/x-wiki g0qzu1jz3lfx90ltbk4vf2b3f9j4lqk 2404588 2404587 2026-04-21T09:55:29Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM tcp-proxy5003.eqsin.wmnet - jmm@cumin2002" 2404588 wikitext text/x-wiki m77zaandqzmfas32uh5wamsn0nccddc 2404589 2404588 2026-04-21T09:59:21Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1159.eqiad.wmnet with reason: Maintenance 2404589 wikitext text/x-wiki kwmf6930nssjs7n97oj8e3d2xzxdze7 2404590 2404589 2026-04-21T09:59:31Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1159 (T419635)', diff saved to https://phabricator.wikimedia.org/P91267 and previous config saved to /var/cache/conftool/dbconfig/20260421-095928-fceratto.json 2404590 wikitext text/x-wiki fpr19tygoht71u1k5ti7vctsp3abgcg 2404591 2404590 2026-04-21T10:00:14Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host thanos-be1007.eqiad.wmnet with OS bullseye 2404591 wikitext text/x-wiki em4v9k9vs8xi3oce110inwxszn345o2 2404592 2404591 2026-04-21T10:00:21Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance 2404592 wikitext text/x-wiki p4w8o35k30lpg47jiolwmito8qz10d9 2404593 2404592 2026-04-21T10:00:45Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance 2404593 wikitext text/x-wiki jl55hjx5hlclccpe0qesbvrxce1b8az 2404594 2404593 2026-04-21T10:00:54Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1212 (T419961)', diff saved to https://phabricator.wikimedia.org/P91268 and previous config saved to /var/cache/conftool/dbconfig/20260421-100051-fceratto.json 2404594 wikitext text/x-wiki 6h4s1nhqq3rk3htm29f5rsqri9ommoq 2404595 2404594 2026-04-21T10:03:05Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host tcp-proxy5003.eqsin.wmnet with OS trixie 2404595 wikitext text/x-wiki 9y9dtewvw04oip0q23osjzy4ban4xg2 2404596 2404595 2026-04-21T10:07:43Z Stashbot 7414 claime: Disabling puppet on A:cp to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1271804 - T422804 2404596 wikitext text/x-wiki o1kzlhmfw5qau86ec12s1kd96wpy7k3 2404597 2404596 2026-04-21T10:08:56Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T419961)', diff saved to https://phabricator.wikimedia.org/P91269 and previous config saved to /var/cache/conftool/dbconfig/20260421-100849-fceratto.json 2404597 wikitext text/x-wiki b6pn6dxcoa4atdhzrijnnev1z8d371q 2404598 2404597 2026-04-21T10:10:57Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2404598 wikitext text/x-wiki eqn1p5fna3wk9t25x3jx9x9rig8v5nn 2404599 2404598 2026-04-21T10:12:21Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply 2404599 wikitext text/x-wiki at5igdlcu7xt582du2wbk82fdcshr9o 2404600 2404599 2026-04-21T10:12:43Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply 2404600 wikitext text/x-wiki 93n2dge44l3yeqv8zz6qutple96h913 2404601 2404600 2026-04-21T10:13:10Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply 2404601 wikitext text/x-wiki b2gcvbrlqtxdyg82a51p3msn2o67mi5 2404602 2404601 2026-04-21T10:13:25Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply 2404602 wikitext text/x-wiki 95jvat8b74mk27uoenvxbdk7o67zsbj 2404603 2404602 2026-04-21T10:13:31Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply 2404603 wikitext text/x-wiki 5oem2xkts6vfs8ngy56y3uaa0p7ct6r 2404604 2404603 2026-04-21T10:13:49Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply 2404604 wikitext text/x-wiki fsqj4uljuy6em7hsn451amou5awrqq4 2404605 2404604 2026-04-21T10:19:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P91270 and previous config saved to /var/cache/conftool/dbconfig/20260421-101857-fceratto.json 2404605 wikitext text/x-wiki 82bzlm8rhkzgmrmden1mibcy26yw0fl 2404606 2404605 2026-04-21T10:22:10Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be1007.eqiad.wmnet with reason: host reimage 2404606 wikitext text/x-wiki 5y32zmx0hh290hf23lmqd592o3xon3c 2404607 2404606 2026-04-21T10:29:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P91271 and previous config saved to /var/cache/conftool/dbconfig/20260421-102907-fceratto.json 2404607 wikitext text/x-wiki ms6fs43ik8gy8vp5xi64lfsj37v8uas 2404608 2404607 2026-04-21T10:30:29Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be1007.eqiad.wmnet with reason: host reimage 2404608 wikitext text/x-wiki e412c8qlwnt4jzes4ks51g4fpx4bmqi 2404610 2404608 2026-04-21T10:35:53Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2143.codfw.wmnet,db1153.eqiad.wmnet with reason: Reimage to Trixie 2404610 wikitext text/x-wiki 16t63jnj3z55kue99umkf2tdu4iqe2o 2404611 2404610 2026-04-21T10:36:42Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2404611 wikitext text/x-wiki 2qlg6nssplj4yrz3593gyun918fjr5v 2404612 2404611 2026-04-21T10:37:32Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2143.codfw.wmnet with reason: Reimage to Trixie 2404612 wikitext text/x-wiki qc86rxqd2qvm5xtvamoegyq2ibhp4z8 2404613 2404612 2026-04-21T10:37:37Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2143: Reimage to Trixie 2404613 wikitext text/x-wiki 7coquibmzrzs8q33jgvhl139m4o7tni 2404614 2404613 2026-04-21T10:37:39Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2404614 wikitext text/x-wiki e84qtkl6m36xbuivfamkpxeso18f44e 2404615 2404614 2026-04-21T10:37:45Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2404615 wikitext text/x-wiki cvw2nitugy9epudvvjpif2v7gdflwr6 2404616 2404615 2026-04-21T10:37:47Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2143: Reimage to Trixie 2404616 wikitext text/x-wiki fafiyrtyu99qiszl9l7j121x60tyev4 2404617 2404616 2026-04-21T10:38:08Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2143.codfw.wmnet with OS trixie 2404617 wikitext text/x-wiki b4hwontvlqu7n8ofe6fe4xhqaxrhw7e 2404618 2404617 2026-04-21T10:39:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T419961)', diff saved to https://phabricator.wikimedia.org/P91273 and previous config saved to /var/cache/conftool/dbconfig/20260421-103915-fceratto.json 2404618 wikitext text/x-wiki em6smp353datwgngo4szsfmazsb1l0r 2404619 2404618 2026-04-21T10:39:38Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance 2404619 wikitext text/x-wiki 8tjvsdr8mgfyyuk9wq9jd7xet01b5pm 2404620 2404619 2026-04-21T10:39:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1223 (T419961)', diff saved to https://phabricator.wikimedia.org/P91274 and previous config saved to /var/cache/conftool/dbconfig/20260421-103945-fceratto.json 2404620 wikitext text/x-wiki 1gwzuggplmtfui5ytunu54il09sxx5l 2404621 2404620 2026-04-21T10:43:22Z Stashbot 7414 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply 2404621 wikitext text/x-wiki iam8i4e2pm3j4yrr0bh5x5vzkp79ayd 2404622 2404621 2026-04-21T10:44:15Z Stashbot 7414 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply 2404622 wikitext text/x-wiki 7ha8npkd4k9pxt9g9tcyu071a0nhtua 2404623 2404622 2026-04-21T10:47:11Z Stashbot 7414 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply 2404623 wikitext text/x-wiki 84vc9pnt1yy1newk5kllw3ez5cc9t5g 2404624 2404623 2026-04-21T10:48:33Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be1007.eqiad.wmnet with OS bullseye 2404624 wikitext text/x-wiki t43xgcu7qt9uat79ydww2q8jk805sm4 2404625 2404624 2026-04-21T10:49:36Z Stashbot 7414 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply 2404625 wikitext text/x-wiki iarbqhtdpcjm4aq1l37yvfofdhhcb0j 2404626 2404625 2026-04-21T10:49:56Z Stashbot 7414 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply 2404626 wikitext text/x-wiki t33ov54cmptda8cs41dasm7c4jde9fm 2404627 2404626 2026-04-21T10:50:47Z Stashbot 7414 jiji@deploy1003: Locking from deployment [ALL REPOSITORIES]: Upgrading mw-mcrouter - effie 2404627 wikitext text/x-wiki j2cudf4n3m1pro00gv36c0y9hk9ty4d 2404628 2404627 2026-04-21T10:50:54Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2404628 wikitext text/x-wiki 23ytubnxfc6rqwwcwtfoyyeehxumu0m 2404629 2404628 2026-04-21T10:51:08Z Stashbot 7414 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply 2404629 wikitext text/x-wiki qvh06pqtkjj91rlrafcswijrwzwnm1g 2404638 2404629 2026-04-21T10:56:47Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2143.codfw.wmnet with reason: host reimage 2404638 wikitext text/x-wiki c32wjvn7xgc8yx88m0jouj1u86bryhr 2404639 2404638 2026-04-21T10:56:57Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2404639 wikitext text/x-wiki h61qv4qma05ryqat2497f78sv8g9wox 2404640 2404639 2026-04-21T10:57:15Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host tcp-proxy5003.eqsin.wmnet with OS trixie 2404640 wikitext text/x-wiki 317tqd7vfoaqtod9fqkqew10ejjpwno 2404641 2404640 2026-04-21T10:57:17Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host tcp-proxy5003.eqsin.wmnet 2404641 wikitext text/x-wiki 22y45i3plo82x0bxf5i8uchc0hvp3ub 2404642 2404641 2026-04-21T10:59:49Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T419635)', diff saved to https://phabricator.wikimedia.org/P91275 and previous config saved to /var/cache/conftool/dbconfig/20260421-105945-fceratto.json 2404642 wikitext text/x-wiki 17p79mx0a9l3sj7xosphq4ozwmyd56j 2404643 2404642 2026-04-21T10:59:52Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host tcp-proxy5003.eqsin.wmnet 2404643 wikitext text/x-wiki 2vlmkjh23q4y6mpyf9egv7i12z3yfsl 2404644 2404643 2026-04-21T10:59:54Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2404644 wikitext text/x-wiki ljq9al8h7xmhhl16p4r3w7wflwm6h8y 2404645 2404644 2026-04-21T11:00:46Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2143.codfw.wmnet with reason: host reimage 2404645 wikitext text/x-wiki nze1yy0ir096uzjc600x5f2gldyh9ii 2404646 2404645 2026-04-21T11:07:49Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) 2404646 wikitext text/x-wiki ktjt9osmxjf91v29pmj75270tuwb9w4 2404647 2404646 2026-04-21T11:07:56Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host tcp-proxy5003.eqsin.wmnet 2404647 wikitext text/x-wiki k7siwpas5zmjsterxinx97zhifetln4 2404648 2404647 2026-04-21T11:09:05Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T419961)', diff saved to https://phabricator.wikimedia.org/P91276 and previous config saved to /var/cache/conftool/dbconfig/20260421-110903-fceratto.json 2404648 wikitext text/x-wiki eqex5n840njk4asitgvwali21nxuzic 2404649 2404648 2026-04-21T11:09:25Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply 2404649 wikitext text/x-wiki 7nwtyuat5l63o38ztqqv09q9bdw17vd 2404650 2404649 2026-04-21T11:09:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P91277 and previous config saved to /var/cache/conftool/dbconfig/20260421-110954-fceratto.json 2404650 wikitext text/x-wiki roop7x1srdzm4rgyzmqhhrr6a9l7bef 2404651 2404650 2026-04-21T11:11:10Z Stashbot 7414 claime: Enabling puppet on A:cp to deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/1271804 - T422804 2404651 wikitext text/x-wiki 6viqwwroxpy1txmo5ol5b5pz72hrh2t 2404652 2404651 2026-04-21T11:19:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P91278 and previous config saved to /var/cache/conftool/dbconfig/20260421-111911-fceratto.json 2404652 wikitext text/x-wiki ehvlz31ay1fpti39ol6i1271hkct73l 2404653 2404652 2026-04-21T11:20:03Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P91279 and previous config saved to /var/cache/conftool/dbconfig/20260421-112001-fceratto.json 2404653 wikitext text/x-wiki 64cs2ge7xtqga0rnxi16v8j1q13l6j6 2404654 2404653 2026-04-21T11:21:05Z Stashbot 7414 klausman@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply 2404654 wikitext text/x-wiki nk8z0t15z4v4zx945sa3jfx154lmk8h 2404655 2404654 2026-04-21T11:21:38Z Stashbot 7414 klausman@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply 2404655 wikitext text/x-wiki 1hy34yuj6jsrgi0j2jhk6rmngofonig 2404656 2404655 2026-04-21T11:23:47Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2143.codfw.wmnet with OS trixie 2404656 wikitext text/x-wiki kq817291hl233hb3w6o34sterqwcaa9 2404657 2404656 2026-04-21T11:24:48Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2143: after reimage to trixie 2404657 wikitext text/x-wiki csv9cbpmw5ehmz1dx5nxv2gdj5sclvn 2404658 2404657 2026-04-21T11:24:49Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2143: after reimage to trixie 2404658 wikitext text/x-wiki iupl0ubmy1wr6n9n5hynwvw2znzzer9 2404659 2404658 2026-04-21T11:25:01Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2143: repool after maintenance 2404659 wikitext text/x-wiki 7hq6jvfckubgp8jb1tlmbi362ic1vpm 2404660 2404659 2026-04-21T11:25:02Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2404660 wikitext text/x-wiki kvmxixb0dg29bmmu3ahqhc3st41n6wt 2404661 2404660 2026-04-21T11:26:50Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2404661 wikitext text/x-wiki 3sik0sllohdi3eucs5kco2da172ldoe 2404662 2404661 2026-04-21T11:26:52Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2143: repool after maintenance 2404662 wikitext text/x-wiki kmrqghj2jrjnm6uqulzp14t4i71ovk2 2404663 2404662 2026-04-21T11:26:56Z Stashbot 7414 klausman@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: apply 2404663 wikitext text/x-wiki 3r69g60yg4ssz08240x3vqwx96x1i62 2404664 2404663 2026-04-21T11:27:26Z Stashbot 7414 klausman@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply 2404664 wikitext text/x-wiki 4mtuiuz43ivy04a3vygwo038pe7jiy9 2404665 2404664 2026-04-21T11:29:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P91281 and previous config saved to /var/cache/conftool/dbconfig/20260421-112919-fceratto.json 2404665 wikitext text/x-wiki tuu4vrxr0zc5tussc0prvd5on6t7w6x 2404666 2404665 2026-04-21T11:30:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T419635)', diff saved to https://phabricator.wikimedia.org/P91282 and previous config saved to /var/cache/conftool/dbconfig/20260421-113010-fceratto.json 2404666 wikitext text/x-wiki jxwrwt9fdbyg8ingf94hfqy6il6x2z9 2404667 2404666 2026-04-21T11:30:18Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1161.eqiad.wmnet with reason: Maintenance 2404667 wikitext text/x-wiki 5wjq94zd5vx4fryoxfzvuwuqk690nbj 2404668 2404667 2026-04-21T11:30:38Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance 2404668 wikitext text/x-wiki lz3a9902frgg7djqbvj4yc8wv3a2vys 2404669 2404668 2026-04-21T11:31:16Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2404669 wikitext text/x-wiki o4qqoh7tjw0lrp7q9im5l6thq9fna4i 2404670 2404669 2026-04-21T11:33:45Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2404670 wikitext text/x-wiki frjtp3x6vuxtyezsgvi5nz4ivgw6fwi 2404671 2404670 2026-04-21T11:38:17Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: test current diff - jmm@cumin2002" 2404671 wikitext text/x-wiki trlw9xivmbih4cyrmkq3e762qpejhpl 2404672 2404671 2026-04-21T11:38:23Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: test current diff - jmm@cumin2002" 2404672 wikitext text/x-wiki gtvw3ziox5u3uyqtwf2kl0b7e5hylwc 2404673 2404672 2026-04-21T11:38:25Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2404673 wikitext text/x-wiki 3340aa9h4u4nmqxup0r744ddu1j0sqj 2404674 2404673 2026-04-21T11:39:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T419961)', diff saved to https://phabricator.wikimedia.org/P91283 and previous config saved to /var/cache/conftool/dbconfig/20260421-113927-fceratto.json 2404674 wikitext text/x-wiki 52i4ghpht4h7igdqxhe0xrvs8e5xyhl 2404675 2404674 2026-04-21T11:39:50Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance 2404675 wikitext text/x-wiki tafa47cpwibvubbdfkncln38ckuz6dp 2404676 2404675 2026-04-21T11:41:23Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host tcp-proxy5003.eqsin.wmnet 2404676 wikitext text/x-wiki jlaihp4gyq2l77m0y9r8u177indiyzs 2404677 2404676 2026-04-21T11:41:25Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.netbox 2404677 wikitext text/x-wiki bscal9suepg1kgc4row3gpnf543nw4o 2404678 2404677 2026-04-21T11:41:32Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply 2404678 wikitext text/x-wiki 4hlh0c4twpwoymiyvaacn7l9vz8tnpl 2404679 2404678 2026-04-21T11:44:59Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM tcp-proxy5003.eqsin.wmnet - jmm@cumin2002" 2404679 wikitext text/x-wiki 0u72h999aohlcys86z3kncmdrbeqvxw 2404680 2404679 2026-04-21T11:45:04Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM tcp-proxy5003.eqsin.wmnet - jmm@cumin2002" 2404680 wikitext text/x-wiki b4ollmen88m50glgv1kao7h5vmp6smm 2404681 2404680 2026-04-21T11:45:11Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2404681 wikitext text/x-wiki s6dyp45ihjej6s9l9m1guwyyijobh4k 2404682 2404681 2026-04-21T11:45:13Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache tcp-proxy5003.eqsin.wmnet on all recursors 2404682 wikitext text/x-wiki d8ryt3tlxswzlib1256oda0wpj3qz74 2404683 2404682 2026-04-21T11:45:15Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) tcp-proxy5003.eqsin.wmnet on all recursors 2404683 wikitext text/x-wiki t6og8zds099mu50d2ys4isez0a2kxna 2404684 2404683 2026-04-21T11:45:41Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM tcp-proxy5003.eqsin.wmnet - jmm@cumin2002" 2404684 wikitext text/x-wiki k3551r7fg0cv7lifwbsvtd2ijlimhp8 2404685 2404684 2026-04-21T11:45:46Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM tcp-proxy5003.eqsin.wmnet - jmm@cumin2002" 2404685 wikitext text/x-wiki n7y72ml6m57zr39axm652tdkh7xzk1s 2404686 2404685 2026-04-21T11:47:20Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Remove pc2011 and add pc2021 as replacement', diff saved to https://phabricator.wikimedia.org/P91285 and previous config saved to /var/cache/conftool/dbconfig/20260421-114718-marostegui.json 2404686 wikitext text/x-wiki ov4xqtq1xjbpftirjrhahs3x0v8u6gd 2404687 2404686 2026-04-21T11:48:46Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host tcp-proxy5003.eqsin.wmnet with OS trixie 2404687 wikitext text/x-wiki khc7mna4tvd35z3v8xxu69xeo3joaqt 2404688 2404687 2026-04-21T11:48:50Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2404688 wikitext text/x-wiki t1fliloalhx82p3050q9rcmr5zkxsoh 2404689 2404688 2026-04-21T11:50:48Z Stashbot 7414 moritzm: installing Tornado security updates 2404689 wikitext text/x-wiki pc4jhev9vvyp9kxc19oogj800rd0obt 2404690 2404689 2026-04-21T11:52:11Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'add pc2021 to pc1', diff saved to https://phabricator.wikimedia.org/P91286 and previous config saved to /var/cache/conftool/dbconfig/20260421-115209-marostegui.json 2404690 wikitext text/x-wiki c2kwr2yhes6300zknuxq9dkdrexarga 2404691 2404690 2026-04-21T11:53:16Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2404691 wikitext text/x-wiki j4xb0r2tos82p8xz4owb30fnjbipk24 2404692 2404691 2026-04-21T11:56:29Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool pc2021: Pool pc2021 into pc 2404692 wikitext text/x-wiki kgmn3igef6r6hcwbj9f2b65fkf9wgxi 2404693 2404692 2026-04-21T11:56:32Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2404693 wikitext text/x-wiki ivemz0mbm8dnuif1ti6o6xhpemfcwjn 2404694 2404693 2026-04-21T11:56:34Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.parsercache (exit_code=99) 2404694 wikitext text/x-wiki hmtx2c4x0e5j0g0nqpk3ouw4plyfzq1 2404695 2404694 2026-04-21T11:56:36Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc2021: Pool pc2021 into pc 2404695 wikitext text/x-wiki 8m6u8laoz083j11752zo4p0bfvy932b 2404696 2404695 2026-04-21T11:57:02Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool pc2021: Pool pc2021 into pc 2404696 wikitext text/x-wiki fmel9fmkcj6ia05tj69rqwwphdxl4q6 2404697 2404696 2026-04-21T11:57:05Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2404697 wikitext text/x-wiki r6p8piljxq0r4x4dp0ljstp0nysotou 2404698 2404697 2026-04-21T11:57:06Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.parsercache (exit_code=99) 2404698 wikitext text/x-wiki rhn0vqjjtv90wi6cno1wnbensdnsloe 2404699 2404698 2026-04-21T11:57:08Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc2021: Pool pc2021 into pc 2404699 wikitext text/x-wiki 0a1qk6houl2nk5wklha5945tsnrpm0v 2404700 2404699 2026-04-21T11:57:23Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool pc1011: Pool pc2021 into pc 2404700 wikitext text/x-wiki csza46v8ppkhqkp4yfpiezu9l4xyw2b 2404701 2404700 2026-04-21T11:57:25Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2404701 wikitext text/x-wiki t0kyhnqdo1yski4u8ulygc1jsaiet5u 2404702 2404701 2026-04-21T11:57:27Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.parsercache (exit_code=99) 2404702 wikitext text/x-wiki tsfhdcv7yagwn5o3ow86gbvq0q6ozmt 2404703 2404702 2026-04-21T11:57:29Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1011: Pool pc2021 into pc 2404703 wikitext text/x-wiki hsl7ws6949filpknpb4qk3tk6is5b8q 2404704 2404703 2026-04-21T11:58:50Z Stashbot 7414 jiji@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Upgrading mw-mcrouter - effie (duration: 68m 02s) 2404704 wikitext text/x-wiki i58ybn7t1notvc9g68eed6yla6vmy8n 0 14218 2404455 2402155 2026-04-21T04:08:23Z Novem Linguae 23682 /* x4 */ add details 2404455 wikitext text/x-wiki 1kdyazknjmsfbui3njl9xt4xgy52ahw 2404456 2404455 2026-04-21T04:10:58Z Novem Linguae 23682 /* x4 */ add 2404456 wikitext text/x-wiki 3o95ks9crjdhsuhfe08lpr7ijlnxx7c 498 15050 2404360 2403924 2026-04-20T20:08:43Z Stashbot 7414 wmbot~lokal-profil@tools-bastion-15: removed old jobs.yml file (now maintained in repo 2404360 wikitext text/x-wiki gcemexeef0k1ltkdsh5lck3v60txiuu 2404364 2404360 2026-04-20T20:26:04Z Stashbot 7414 wmbot~lokal-profil@tools-bastion-15: Deploy 8cf1305, db7d1ac (T55271), e885dad 2404364 wikitext text/x-wiki k2uo9dzdkhxeyatgwab85gqh0q9gm66 0 17290 2404195 2403892 2026-04-20T15:03:16Z Stashbot 7414 James_F: Docker: Bump ci-bullseye/-bookworm/-trixie for mirrors.wm.org removal, T423622 2404195 wikitext text/x-wiki i7npo0sni2n56bmtnvywwnhpk890xml 2404329 2404195 2026-04-20T18:18:06Z Stashbot 7414 bd808: Unblock 113.128.0.0/15 2404329 wikitext text/x-wiki 3wizwohh9sk82e30kvijini0txcycxk 2404330 2404329 2026-04-20T18:19:04Z Stashbot 7414 James_F: jjb: All 486 (!) jobs now updated for T423622 2404330 wikitext text/x-wiki 03y583l0tuffn7bnmyek0r5b0du3ngg 2404332 2404330 2026-04-20T18:28:25Z Stashbot 7414 dancy: Upgrading gitlab cloud runners (staging) (T423726) 2404332 wikitext text/x-wiki 3dnbbe2o6fngnye5af37n6khdrm11dl 2404333 2404332 2026-04-20T18:28:59Z Stashbot 7414 dancy: Upgrading gitlab cloud runners (staging) to 1.33.9-do.2 (T423726) 2404333 wikitext text/x-wiki ba1hcdjxbdkqhlfmgrlckmkj3lnicok 2404405 2404333 2026-04-20T23:56:05Z Stashbot 7414 bd808: Unblock 76.157.0.0/16 2404405 wikitext text/x-wiki 43ezyplpue6chn39mku7p8ji6s47is7 1 26157 2404121 2359397 2026-04-20T13:01:50Z PAssimacopoulos-WMF 54622 /* Question about the note beneath 'Setting up your SSH config' code */ new section 2404121 wikitext text/x-wiki j9h40cxculk4cx3ixu3it1ihp61gv7q 10 26999 2404341 2395010 2026-04-20T18:44:19Z DDuvall (WMF) 3208 bast1004 is the current eqiad bastion 2404341 wikitext text/x-wiki c2qilt70t6qc5z6prwlx9plrbt69ne8 498 442831 2404325 2349347 2026-04-20T18:03:54Z Stashbot 7414 mutante: added new contint hosts contint1003 / contint2003 to security groups to allow ssh access to puppet-compiler instances T418521 2404325 wikitext text/x-wiki fe4mbzkcp0qs7h0ju2l5bt815rg42je 0 443686 2404353 2401068 2026-04-20T19:25:40Z KHerron (WMF) 10463 updating pyrra references to sloth replacements 2404353 wikitext text/x-wiki jdi10bmx6caf9gmjxrme5h397y0zep9 498 443947 2404117 2401423 2026-04-20T12:57:02Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 135b3fa3d2 (l10n updates: ko) 2404117 wikitext text/x-wiki 4kx47eag1kx5r2n4xm401yu7kmx4kew 0 447612 2404338 1906977 2026-04-20T18:41:16Z DDuvall (WMF) 3208 decommissioned 2404338 wikitext text/x-wiki 4v7wh0p5zefcny6sby4z3062o8lvnaz 2404609 2404338 2026-04-21T10:31:19Z Lucas Werkmeister (WMDE) 10037 link replacement 2404609 wikitext text/x-wiki i146xq1gs8ljnt7uc8quk501iroo5l4 0 447732 2404368 2396435 2026-04-20T21:05:26Z AndrewBogott (WMF) 51047 /* guest images */ 2404368 wikitext text/x-wiki 5qbebwad4o0n5cgjigm6t1zqhgxf3r8 0 447982 2404351 2302507 2026-04-20T19:22:13Z KHerron (WMF) 10463 2404351 wikitext text/x-wiki c2v9e69iid8s89iqumybv8ad9dm6eay 0 448004 2404348 2341969 2026-04-20T19:16:28Z KHerron (WMF) 10463 2404348 wikitext text/x-wiki b47dr3zq4cbts0onjbjtk3qr18k5q7d 0 448736 2404242 2404090 2026-04-20T15:58:46Z JMeybohm (WMF) 16709 /* Removing a node */ 2404242 wikitext text/x-wiki qan85otw0rbx2jditn2mo6grcptkgto 2404244 2404242 2026-04-20T16:05:50Z JMeybohm (WMF) 16709 /* Decommission */ 2404244 wikitext text/x-wiki 0q3lu8fp83xka9f230cc1xos1omgzxv 2404515 2404244 2026-04-21T08:24:33Z JMeybohm (WMF) 16709 /* Decommission */ 2404515 wikitext text/x-wiki prtwpyp5aznugi448efa1m7ykl2exlz 2404516 2404515 2026-04-21T08:24:59Z JMeybohm (WMF) 16709 /* Decommission */ 2404516 wikitext text/x-wiki ky8clvejx6w1ax2gmhhs8mejqghf7jw 2404545 2404516 2026-04-21T08:57:55Z JMeybohm (WMF) 16709 /* Decommission */ 2404545 wikitext text/x-wiki 2db6rvbz36bxkrs3fvt7uud8nmtcih8 0 449160 2404145 2403956 2026-04-20T13:43:15Z Dexbot 30554 Bot: Updating the report 2404145 wikitext text/x-wiki eaioy0yu9xfcge8dn7dxht5n3jcemws 2404156 2404145 2026-04-20T13:55:18Z Dexbot 30554 Bot: Updating the report 2404156 wikitext text/x-wiki e2k2hr6d9hqflf1ord3s6lm53tyh2k0 2404214 2404156 2026-04-20T15:25:30Z Dexbot 30554 Bot: Updating the report 2404214 wikitext text/x-wiki 3co6etpd0wfa77at1areun71716hfko 2404406 2404214 2026-04-21T00:02:36Z Dexbot 30554 Bot: Updating the report 2404406 wikitext text/x-wiki 9ps52s8ljq0v3dpymz5vz1sz9qlwxqx 2404446 2404406 2026-04-21T02:23:52Z Dexbot 30554 Bot: Updating the report 2404446 wikitext text/x-wiki ouyrw1tz4l6g0pz89vcmokn1yar4r2j 2404500 2404446 2026-04-21T08:03:34Z Dexbot 30554 Bot: Updating the report 2404500 wikitext text/x-wiki 9rfispazzkmv4uqdb6q49gxq7ks8dk0 2404507 2404500 2026-04-21T08:12:35Z Dexbot 30554 Bot: Updating the report 2404507 wikitext text/x-wiki 56xge8plfv7ucpx35sfbk22cqq4rksv 0 450684 2404118 2403378 2026-04-20T12:57:06Z Atsuko-WMF 54082 /* Development instance */ --changes-only feature 2404118 wikitext text/x-wiki dh0erwrgq2y9p0lyiepwqei9a54ph6s 0 451075 2404361 2399449 2026-04-20T20:10:14Z DAdedoyin (WMF) 39150 Rotation 2404361 wikitext text/x-wiki 3qktgu9jnl7ikqjm24xay3q1vhmlnsv 498 452890 2404394 2403277 2026-04-20T23:22:46Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24695320410 (https://github.com/cluebotng/component-configs/commits/279edf060f43353ea66e6d057773bfdb883b16a1) 2404394 wikitext text/x-wiki 7uxs05cvi6fvkcwj4fqhjeezko3l9a9 498 453144 2404334 2095165 2026-04-20T18:30:53Z Stashbot 7414 lucaswerkmeister: deleted two files (T423940) 2404334 wikitext text/x-wiki ictnca8ewngigiz4gw83abb7rvs24rl 116 453906 2404347 2403879 2026-04-20T19:15:17Z Gitlabaccountapprovalbot 37332 fitch was rejected. 2404347 wikitext text/x-wiki pcly1cquh3fqycf9h0sfimcauv4x371 498 453941 2404305 2262493 2026-04-20T17:31:13Z Stashbot 7414 wmbot~bd808@tools-bastion-14: Built new image from a4744ea4 2404305 wikitext text/x-wiki arqiqfqqeq4hjjculujln90g4qyzb32 0 455989 2404452 2403486 2026-04-21T03:54:39Z CMing (WMF) 23277 /* Measurement plan */ 2404452 wikitext text/x-wiki 32te8trpmvjtk6ppqrhvtqf0kjfh3jw 2404453 2404452 2026-04-21T03:57:17Z CMing (WMF) 23277 /* Experiment design: non-cache splitting experiments */ 2404453 wikitext text/x-wiki 18e2o6yurih4nencmqpa42oqrxzmi0d 0 458812 2404349 2350406 2026-04-20T19:21:08Z KHerron (WMF) 10463 updating pyrra references to sloth 2404349 wikitext text/x-wiki eu4w4ad5icfovje8ozffsagw69o623q 0 459229 2404354 2343722 2026-04-20T19:26:52Z KHerron (WMF) 10463 2404354 wikitext text/x-wiki l35eij8q92812qcpr5mz9vu25y1fxt5 498 459244 2404374 2403502 2026-04-20T22:04:37Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24692640197 (https://github.com/cluebotng/component-configs/commits/6ccc54ff83a9536189370f93b0f3b7f9225e4259) 2404374 wikitext text/x-wiki ab8ymql23jjbmkt75stm2wroc6u4tuj 498 459255 2404373 2403501 2026-04-20T22:04:11Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24692640192 (https://github.com/cluebotng/component-configs/commits/6ccc54ff83a9536189370f93b0f3b7f9225e4259) 2404373 wikitext text/x-wiki 2k2ns4whx9usr28ct72louihbjfz198 0 459695 2404392 2398087 2026-04-20T23:21:02Z APaskulin (WMF) 14880 /* Endpoint map */ Add GrowthExperiments module endpoint 2404392 wikitext text/x-wiki 94motqjo7koh8asrtkn73lut0gzppcq 0 459699 2404201 2403279 2026-04-20T15:09:55Z Stashbot 7414 ejegg: payments-wiki upgraded from 86a42498 to 6e78ef91 2404201 wikitext text/x-wiki av39uos80d025rvem6wjkyi20001iba 2404350 2404201 2026-04-20T19:21:10Z Stashbot 7414 larssandergreen: tools upgraded from 26ab0125 to edca3f63 2404350 wikitext text/x-wiki p89iex69zdb7vaheoxfa14c7ma5cey7 0 459719 2404163 2404086 2026-04-20T14:00:19Z BTullis (WMF) 25295 Updated the SLIs to reflect the realistic targets defined. 2404163 wikitext text/x-wiki 5wq97zc8fq12wlrbjn7t4f1tbi1di86 0 459721 2404352 2381636 2026-04-20T19:22:29Z KHerron (WMF) 10463 2404352 wikitext text/x-wiki 3qbxcay95csj1d51ctdm7yx0lxeix4a 0 459743 2404366 2400335 2026-04-20T20:53:17Z LErickson-WMF 51715 2404366 wikitext text/x-wiki gfav0fkuoyzcijcztkbe425j4mrc4qe 2404367 2404366 2026-04-20T20:54:06Z LErickson-WMF 51715 2404367 wikitext text/x-wiki db6ku4dab0bpcr0ubnbl2dmeqs6m35k 0 459745 2404365 2381118 2026-04-20T20:48:56Z LErickson-WMF 51715 2404365 wikitext text/x-wiki pdjbyn4ek7nneon58aqdvrvr47rqg7h 2 460039 2404543 2401882 2026-04-21T08:54:53Z Atsuko-WMF 54082 session simeout 2404543 wikitext text/x-wiki nn9cgkypiyq4wcljisl48w4vqkfk8f2 8 460076 2404323 2026-04-20T17:52:21Z BryanDavis 1604 borrow some common block reasons from [[:mw:MediaWiki:Ipbreason-indef-dropdown]] 2404323 wikitext text/x-wiki ll73yh5956c88dfmnoder3bfvboank9 0 460077 2404337 2026-04-20T18:40:00Z DDuvall (WMF) 3208 new eqiad bastion 2404337 wikitext text/x-wiki 5puzpee5v5lfpjxcbt6be5l3qx5zktq