Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.7 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 0 3146 2427551 2427292 2026-06-16T14:04:42Z BJensen-WMF 48942 Initial page describing apache configuration, primarily for mw-on-k8s 2427551 wikitext text/x-wiki rx0ack2885cfcex6dh1vz5veotvcvuf 0 4108 2427389 2427338 2026-06-16T12:08:56Z ScheduleDeploymentBot 37566 Add [[gerrit:1302804]] to Tuesday, June 16 UTC afternoon backport window 2427389 wikitext text/x-wiki kwznns0hhkkav235ttps561voruepck 2427558 2427389 2026-06-16T14:47:24Z ScheduleDeploymentBot 37566 Add [[gerrit:1302890]] to Tuesday, June 16 UTC late backport window 2427558 wikitext text/x-wiki oeuitc6kf0di5rz57gy88okym29b499 2427592 2427558 2026-06-16T15:50:23Z ScheduleDeploymentBot 37566 Add [[gerrit:1302320]] to Tuesday, June 16 UTC late backport window 2427592 wikitext text/x-wiki rel60hoxydh8r7xr3o49rcolaw18er0 2427660 2427592 2026-06-16T18:17:47Z ScheduleDeploymentBot 37566 Add [[gerrit:1302934]] to Tuesday, June 16 UTC late backport window 2427660 wikitext text/x-wiki 6lpoyf1aqma7lxl9abex8kx6gyv3mo5 2427812 2427660 2026-06-17T01:31:47Z ScheduleDeploymentBot 37566 Add [[gerrit:1302998]] to Wednesday, June 17 UTC afternoon backport window 2427812 wikitext text/x-wiki 2rjdgvoi7fy3md74mbvk76lyblywl5g 2427820 2427812 2026-06-17T03:45:16Z ScheduleDeploymentBot 37566 Add [[gerrit:1303012]] to Wednesday, June 17 UTC afternoon backport window 2427820 wikitext text/x-wiki q0lvmqeufmq3vlqdqux99unkxfl736x 2427821 2427820 2026-06-17T03:46:49Z ScheduleDeploymentBot 37566 Add [[gerrit:1303010]] to Wednesday, June 17 UTC morning backport window 2427821 wikitext text/x-wiki dpebqndthvauakxs8zwccenz1h980i0 2427822 2427821 2026-06-17T03:47:12Z ScheduleDeploymentBot 37566 Add [[gerrit:1303009]] to Wednesday, June 17 UTC morning backport window 2427822 wikitext text/x-wiki 877vc85rnyok2xybik416z258cijwqc 2427870 2427822 2026-06-17T07:54:09Z ScheduleDeploymentBot 37566 Add [[gerrit:1303323]] to Wednesday, June 17 UTC morning backport window 2427870 wikitext text/x-wiki qg9lf3e5oojlvkgq2ftle2ig1b6b4qc 2427871 2427870 2026-06-17T07:54:38Z ScheduleDeploymentBot 37566 Add [[gerrit:1303297]] to Wednesday, June 17 UTC morning backport window 2427871 wikitext text/x-wiki rem9xkmg2wp8h7vczr4a5p2eo5cj3av 2427896 2427871 2026-06-17T08:31:26Z APatro (WMF) 38793 /* {{Deployment_day|date=2026-06-17}} */ 2427896 wikitext text/x-wiki 5rkjf9v2l8vlq2kjcjsbfo99ezcida2 2427959 2427896 2026-06-17T09:48:14Z ScheduleDeploymentBot 37566 Add [[gerrit:1303365]] to Wednesday, June 17 UTC late backport window 2427959 wikitext text/x-wiki 9xlrep3yzzk68dh1fne52axde0pbe5k 2427961 2427959 2026-06-17T09:48:31Z ScheduleDeploymentBot 37566 Add [[gerrit:1303364]] to Wednesday, June 17 UTC late backport window 2427961 wikitext text/x-wiki 0599hrc95bbvd54tvetgatk62yyypge 0 7919 2427368 2427360 2026-06-16T12:01:46Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging1006.eqiad.wmnet with OS trixie 2427368 wikitext text/x-wiki hyo25uymzu7mmx4l3v369d2r6qgig9n 2427369 2427368 2026-06-16T12:01:58Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2012.codfw.wmnet 2427369 wikitext text/x-wiki 478uih8f3czfwws08nbptbpuoviqfa6 2427370 2427369 2026-06-16T12:02:11Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2001.codfw.wmnet 2427370 wikitext text/x-wiki nevx4r9mjyz3af2th8kmmk599c1qy63 2427373 2427370 2026-06-16T12:02:33Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2012.codfw.wmnet 2427373 wikitext text/x-wiki b5ocdvitl1brqbllamjwpo5cpq2pyia 2427374 2427373 2026-06-16T12:03:03Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2013.codfw.wmnet 2427374 wikitext text/x-wiki 5k8lwg787en1jmywj5j3ht9ma45ir41 2427375 2427374 2026-06-16T12:03:39Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2013.codfw.wmnet 2427375 wikitext text/x-wiki jkoqic3ixs0oc1agkhiol6dqfxgn93r 2427376 2427375 2026-06-16T12:03:45Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2014.codfw.wmnet 2427376 wikitext text/x-wiki 4qxvsilze5ddnn11gl5snya1m62ff70 2427378 2427376 2026-06-16T12:04:18Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2014.codfw.wmnet 2427378 wikitext text/x-wiki eehhgf66z34fb7h3e8dgxvtwtsfvmjz 2427379 2427378 2026-06-16T12:04:23Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2017.codfw.wmnet 2427379 wikitext text/x-wiki tvi5eaq0oterlm1cflvcqmmnib97dyz 2427380 2427379 2026-06-16T12:04:57Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2017.codfw.wmnet 2427380 wikitext text/x-wiki j0ak3vf71id7wzeg4dv88os9e6ltc6q 2427381 2427380 2026-06-16T12:05:07Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2018.codfw.wmnet 2427381 wikitext text/x-wiki pef9mybrlrcchtaerjhbukv6kxtinp6 2427382 2427381 2026-06-16T12:05:40Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2018.codfw.wmnet 2427382 wikitext text/x-wiki c93aei3jtm7ykdgtbem5kzakojml768 2427383 2427382 2026-06-16T12:05:46Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet 2427383 wikitext text/x-wiki ddtgr13x3kda7fqoth9g01oh4ccl9lc 2427384 2427383 2026-06-16T12:06:19Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet 2427384 wikitext text/x-wiki 701rv096j95tgkahzqzcbbk9ey3gu76 2427385 2427384 2026-06-16T12:06:24Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2044.codfw.wmnet 2427385 wikitext text/x-wiki cu68vh1wvy1y27d6e6n3z34qmhqc9i5 2427386 2427385 2026-06-16T12:06:56Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2044.codfw.wmnet 2427386 wikitext text/x-wiki 6pkucw76yvgmewvxg0b7uqy8l64pufc 2427387 2427386 2026-06-16T12:06:58Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-a5-codfw,lsw1-a5-codfw IPv6,lsw1-a5-codfw.mgmt,ssw1-a[1,8]-codfw.mgmt with reason: switch upgrrade 2427387 wikitext text/x-wiki s8tkwgc40qjiq9tzrpp6rlniqzrmyep 2427388 2427387 2026-06-16T12:07:02Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2051.codfw.wmnet 2427388 wikitext text/x-wiki hc67vur2zbaz24b4xt37mqpzasuv7t7 2427390 2427388 2026-06-16T12:10:46Z Stashbot 7414 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 29 hosts with reason: lsw1-a5-codfw JunOS upgrade 2427390 wikitext text/x-wiki kzdpbqvwb1e7ieg4c7kwcntw1qeam3m 2427391 2427390 2026-06-16T12:12:05Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2051.codfw.wmnet 2427391 wikitext text/x-wiki ohwsquvfijyu1oaiy9j8utdpuzp0bod 2427392 2427391 2026-06-16T12:12:10Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2074.codfw.wmnet 2427392 wikitext text/x-wiki aykufz5ltph3o8voxdeu49ui190grux 2427393 2427392 2026-06-16T12:12:46Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2074.codfw.wmnet 2427393 wikitext text/x-wiki blx6q8lbei9e1yobrkd02da1kjdnow1 2427394 2427393 2026-06-16T12:12:51Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2075.codfw.wmnet 2427394 wikitext text/x-wiki jcswkd6zxqptnb453jjyngybyroj0ne 2427395 2427394 2026-06-16T12:13:25Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2075.codfw.wmnet 2427395 wikitext text/x-wiki gs70po0q1gizxalodx92h708yr71gvq 2427396 2427395 2026-06-16T12:13:30Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2076.codfw.wmnet 2427396 wikitext text/x-wiki oqrg9hhuzphyvp94cd1aspo50t6yxlg 2427397 2427396 2026-06-16T12:14:03Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2076.codfw.wmnet 2427397 wikitext text/x-wiki gseed7h2ydvkpv28navrzp1h78qgsl3 2427398 2427397 2026-06-16T12:14:08Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2077.codfw.wmnet 2427398 wikitext text/x-wiki rsxq3odtmywas0gfvaqq264a9hb7ubz 2427399 2427398 2026-06-16T12:14:45Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2077.codfw.wmnet 2427399 wikitext text/x-wiki 9tif9cq5in4w7n0yw0wxscret2kn84e 2427400 2427399 2026-06-16T12:14:50Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2078.codfw.wmnet 2427400 wikitext text/x-wiki lrt7fzqggdj8vmqp4qbtt3neliui9n4 2427402 2427400 2026-06-16T12:15:23Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2078.codfw.wmnet 2427402 wikitext text/x-wiki bjr6uvjdequdfk0f73q7ippd8e21tx9 2427403 2427402 2026-06-16T12:15:28Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2091.codfw.wmnet 2427403 wikitext text/x-wiki 1ou6f9xeonhm1ff1a9m7g53czk4mlrb 2427404 2427403 2026-06-16T12:16:01Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2091.codfw.wmnet 2427404 wikitext text/x-wiki lhkfi6xs0tzvbtj1dwgdmhqu6khp54m 2427405 2427404 2026-06-16T12:16:07Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2092.codfw.wmnet 2427405 wikitext text/x-wiki lp3fnov8elkthlktrc982ts9imoqg3c 2427406 2427405 2026-06-16T12:16:40Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2092.codfw.wmnet 2427406 wikitext text/x-wiki bj6mbencz9icb9rdzu7bm6ikg99x9my 2427407 2427406 2026-06-16T12:16:45Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2242.codfw.wmnet 2427407 wikitext text/x-wiki 8k02nrhwg505n09c91o01uj8r3w6v51 2427408 2427407 2026-06-16T12:17:20Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2242.codfw.wmnet 2427408 wikitext text/x-wiki qgdhro2oe445gkttv72cqg4dlpfy04m 2427410 2427408 2026-06-16T12:17:25Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2243.codfw.wmnet 2427410 wikitext text/x-wiki egqb5omhyrk9b6oh31sygd3p5mx09b6 2427411 2427410 2026-06-16T12:17:59Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2243.codfw.wmnet 2427411 wikitext text/x-wiki ezvizy1yzjbxq8794oixa2sjl7unt9e 2427412 2427411 2026-06-16T12:18:05Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2254.codfw.wmnet 2427412 wikitext text/x-wiki 7ss57hfvh7es6lfm4kw3rm6trkgg1ao 2427414 2427412 2026-06-16T12:19:13Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2254.codfw.wmnet 2427414 wikitext text/x-wiki dah9529jj9sve60bsd6i6lrfoxz9osp 2427415 2427414 2026-06-16T12:19:18Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2255.codfw.wmnet 2427415 wikitext text/x-wiki 6rngg0z54plztdk85xzu2snk7o82zya 2427416 2427415 2026-06-16T12:19:52Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2255.codfw.wmnet 2427416 wikitext text/x-wiki mizdba93x8f3aeylm5j8ifgbwbb3igg 2427418 2427416 2026-06-16T12:22:15Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1006.eqiad.wmnet with reason: host reimage 2427418 wikitext text/x-wiki m51d8etmvzdgpgsnp9zlo94wn8gk9gs 2427419 2427418 2026-06-16T12:23:44Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging1007.eqiad.wmnet with OS trixie 2427419 wikitext text/x-wiki 80hs9xcmuuzvionlx1dv342csoyanp1 2427421 2427419 2026-06-16T12:24:16Z Stashbot 7414 topranks: reboot lsw1-a5-codfw to complete JunOS upgrade T428020 2427421 wikitext text/x-wiki 3t73d7h9dfd2g483b7kypodadejfcq0 2427422 2427421 2026-06-16T12:24:21Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging1008.eqiad.wmnet with OS trixie 2427422 wikitext text/x-wiki tli1p2c3oc3xnlvempq2jncmnzes4vr 2427423 2427422 2026-06-16T12:28:32Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2427423 wikitext text/x-wiki olc3m2lre7s31ar3mzuiihe9qvxz9m2 2427424 2427423 2026-06-16T12:28:55Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1006.eqiad.wmnet with reason: host reimage 2427424 wikitext text/x-wiki c2er6kqvkr4plpnj8vg6rl3wu4m1xpk 2427428 2427424 2026-06-16T12:40:19Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1008.eqiad.wmnet with reason: host reimage 2427428 wikitext text/x-wiki m5xbyyfk7c5bhuh0cpm605631msld1q 2427429 2427428 2026-06-16T12:43:16Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet 2427429 wikitext text/x-wiki kod4loa5euyklpk58nqvcgr375tkk22 2427430 2427429 2026-06-16T12:44:41Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1008.eqiad.wmnet with reason: host reimage 2427430 wikitext text/x-wiki rarq0zr9s2pvsoqs2em8d3uy6c47wna 2427431 2427430 2026-06-16T12:45:26Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet 2427431 wikitext text/x-wiki nbwvbxeizt6tf1twtjizgypm6mrjqax 2427432 2427431 2026-06-16T12:45:28Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet 2427432 wikitext text/x-wiki hsqcgne4znp4jdnv0duceohhw0vmffk 2427433 2427432 2026-06-16T12:45:34Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet 2427433 wikitext text/x-wiki 067wf3cd2vvhb9tyskksm63fp5s8p44 2427434 2427433 2026-06-16T12:45:36Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2013.codfw.wmnet 2427434 wikitext text/x-wiki c0nbym4ydn744prup0vctcghcx54euf 2427435 2427434 2026-06-16T12:45:40Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2014.codfw.wmnet 2427435 wikitext text/x-wiki c5ul59n8vsmazrd8fw5yxv3r26mwcnj 2427436 2427435 2026-06-16T12:45:42Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2014.codfw.wmnet 2427436 wikitext text/x-wiki hn6ow5r1yhp4qlcld4lpfcfkc1rygc7 2427437 2427436 2026-06-16T12:45:47Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2017.codfw.wmnet 2427437 wikitext text/x-wiki ec9namegyj0sdhlksnq3ds1cnobfwro 2427438 2427437 2026-06-16T12:45:49Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2017.codfw.wmnet 2427438 wikitext text/x-wiki 2axzriyabylme30b5pjc0jekf77omrs 2427439 2427438 2026-06-16T12:45:55Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2018.codfw.wmnet 2427439 wikitext text/x-wiki ptkk6mwmcbtkabo2viyxxgag47cjg46 2427440 2427439 2026-06-16T12:45:57Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2018.codfw.wmnet 2427440 wikitext text/x-wiki tc7t22jgv3tx4akr37s9fnkgrha5172 2427441 2427440 2026-06-16T12:45:59Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2001.codfw.wmnet 2427441 wikitext text/x-wiki qxs3y6k0z7nhjvf4x4gghzspyqaimo4 2427442 2427441 2026-06-16T12:46:01Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2427442 wikitext text/x-wiki menvp5u96ly2hekw548f9k3ilnmo3zt 2427443 2427442 2026-06-16T12:46:02Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2001.codfw.wmnet 2427443 wikitext text/x-wiki d8coi6az0pie667tvcljc5wdgkbtalk 2427444 2427443 2026-06-16T12:46:04Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet 2427444 wikitext text/x-wiki 6jc1nknxg893xka1rtrafo5a3018uz3 2427445 2427444 2026-06-16T12:46:06Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet 2427445 wikitext text/x-wiki 9wxuikq6jarojg86kv3dna7kbjz3t2m 2427446 2427445 2026-06-16T12:46:08Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2044.codfw.wmnet 2427446 wikitext text/x-wiki otc09wnijq9y7o02gyky8ckhkxq2f2m 2427447 2427446 2026-06-16T12:46:10Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2044.codfw.wmnet 2427447 wikitext text/x-wiki ls5iw06rmr8bdkggoqkxxj3ndxw8jh6 2427448 2427447 2026-06-16T12:46:15Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2051.codfw.wmnet 2427448 wikitext text/x-wiki 5g778bb8c2k87pv0we0cg00or26g1ea 2427449 2427448 2026-06-16T12:46:17Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2051.codfw.wmnet 2427449 wikitext text/x-wiki dsn145mhbtnbuoufj7s1z8jdexll4o2 2427450 2427449 2026-06-16T12:46:22Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2074.codfw.wmnet 2427450 wikitext text/x-wiki 7eudq26nnsfrffcc9oy8f5qt21jvf85 2427451 2427450 2026-06-16T12:46:25Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2074.codfw.wmnet 2427451 wikitext text/x-wiki 7whp8ooxfw5tky3ki3thbzm98ny9kja 2427452 2427451 2026-06-16T12:46:29Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2075.codfw.wmnet 2427452 wikitext text/x-wiki cdc1xe3ignx40buavzi9bblj89vxqbl 2427453 2427452 2026-06-16T12:46:31Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2075.codfw.wmnet 2427453 wikitext text/x-wiki f4hscettqshzrbt0iwkcxq7fimxzz78 2427454 2427453 2026-06-16T12:46:36Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2076.codfw.wmnet 2427454 wikitext text/x-wiki cfmadal1b4rnxovotdvi5s5ng0tiulf 2427455 2427454 2026-06-16T12:46:39Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.hosts.remove-downtime for 29 hosts 2427455 wikitext text/x-wiki e3gnff2e7kqryxd6yixvkmuckzpdz3x 2427456 2427455 2026-06-16T12:46:41Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2076.codfw.wmnet 2427456 wikitext text/x-wiki s35600bpj9m58ppy8a4eom23d5dy513 2427457 2427456 2026-06-16T12:46:44Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2077.codfw.wmnet 2427457 wikitext text/x-wiki 5th0kn4nshii2aqbepodfol0r5uznaa 2427458 2427457 2026-06-16T12:46:46Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2077.codfw.wmnet 2427458 wikitext text/x-wiki q8nqgshaadwquhl7ps719i8b385edja 2427459 2427458 2026-06-16T12:46:51Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2078.codfw.wmnet 2427459 wikitext text/x-wiki 0c9kb32jpn1wx8bz86v0ugrxpw05ewp 2427460 2427459 2026-06-16T12:46:53Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2078.codfw.wmnet 2427460 wikitext text/x-wiki 6jzb7gtzmtfzulxvhfr76fxriskte7v 2427461 2427460 2026-06-16T12:46:54Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 29 hosts 2427461 wikitext text/x-wiki cne2o2wafdo4lcy3l7ozkdpvi4lwkd7 2427462 2427461 2026-06-16T12:46:58Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2091.codfw.wmnet 2427462 wikitext text/x-wiki 2hk1y7172o0rd4d85ruiur3twt0y6k4 2427463 2427462 2026-06-16T12:47:00Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2091.codfw.wmnet 2427463 wikitext text/x-wiki ayuxbvpmslvvuwjqo8peaox40gm3doe 2427464 2427463 2026-06-16T12:47:05Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2092.codfw.wmnet 2427464 wikitext text/x-wiki rxuqcnkaz2gkstyry7phrtsblf8xxkc 2427465 2427464 2026-06-16T12:47:07Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2092.codfw.wmnet 2427465 wikitext text/x-wiki phyf7h97r6fyhls616lajgm1qcq1nts 2427466 2427465 2026-06-16T12:47:10Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2427466 wikitext text/x-wiki 44b3uz04q2frmrs1z6eex0qe57xx2kd 2427467 2427466 2026-06-16T12:47:12Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2242.codfw.wmnet 2427467 wikitext text/x-wiki ozfs9gudo20dep9jzkiman8va6u2wni 2427468 2427467 2026-06-16T12:47:14Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2242.codfw.wmnet 2427468 wikitext text/x-wiki pykbvoal25thczuaclkymzovfs8hs7p 2427469 2427468 2026-06-16T12:47:19Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2243.codfw.wmnet 2427469 wikitext text/x-wiki qclz68awfc1mom2ultcppnpcj0o6b0o 2427470 2427469 2026-06-16T12:47:21Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2243.codfw.wmnet 2427470 wikitext text/x-wiki 2f1j3mk0wmxh5r4u1ylmkgmngtrftdf 2427471 2427470 2026-06-16T12:47:27Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2254.codfw.wmnet 2427471 wikitext text/x-wiki b0jd5dnzbe000anaim2h8dn7ric3mzr 2427472 2427471 2026-06-16T12:47:29Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2254.codfw.wmnet 2427472 wikitext text/x-wiki ieg733cy99i1ae047ghy3eh6fdlqtlw 2427473 2427472 2026-06-16T12:47:34Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2255.codfw.wmnet 2427473 wikitext text/x-wiki cmrmna6ie0m6kz9ti3d6kbrgvirgwyl 2427474 2427473 2026-06-16T12:47:36Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2255.codfw.wmnet 2427474 wikitext text/x-wiki aa77t63ts3dzh5dewmr3zsmgvzrkj1i 2427475 2427474 2026-06-16T12:48:54Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.mysql.pool pool db2153: codfw rack a5 depool for switch maintenance T428020 2427475 wikitext text/x-wiki 16cek1i5kmd3ihcrp8xujroiaznq0gb 2427476 2427475 2026-06-16T12:49:21Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet 2427476 wikitext text/x-wiki 8w7462wurpdz2uyslr4d773jvax290g 2427477 2427476 2026-06-16T12:50:54Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2427477 wikitext text/x-wiki 7mdzi44ii4144xob3p4bhzb0cazbzyt 2427478 2427477 2026-06-16T12:50:56Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging1006.eqiad.wmnet with OS trixie 2427478 wikitext text/x-wiki ckn0xve61fgdn8432756ufoev1x8zik 2427479 2427478 2026-06-16T12:51:21Z Stashbot 7414 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-logging1007.eqiad.wmnet with OS trixie 2427479 wikitext text/x-wiki qb73ichxzq4kog5rb34tjmt03nmr5ym 2427481 2427479 2026-06-16T12:52:01Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.mysql.pool pool db2154: codfw rack a5 depool for switch maintenance T428020 2427481 wikitext text/x-wiki qx3b4nos779cqqwxs8n4d2fqz4q9uab 2427482 2427481 2026-06-16T12:52:03Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1079.eqiad.wmnet with OS trixie 2427482 wikitext text/x-wiki i8t79nim4oopbpea2kpxbeps3p22iq3 2427484 2427482 2026-06-16T12:57:06Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2427484 wikitext text/x-wiki duevmhz7teun6h5u9itk2sgilxhivk8 2427486 2427484 2026-06-16T13:01:23Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1080.eqiad.wmnet with OS trixie 2427486 wikitext text/x-wiki gv1nyjbezq51miul8gbt4gt7ipkpgu4 2427487 2427486 2026-06-16T13:02:10Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2427487 wikitext text/x-wiki jygg65z74qzh3k5155x8w6sgjr4ms1r 2427488 2427487 2026-06-16T13:02:12Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2427488 wikitext text/x-wiki a81302bospkbbw5i9mwfz9v9384xdnj 2427489 2427488 2026-06-16T13:02:49Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1079.eqiad.wmnet with reason: host reimage 2427489 wikitext text/x-wiki gbc9xz3onbvg1k0vilhmmile85x8uy4 2427490 2427489 2026-06-16T13:03:45Z Stashbot 7414 mfossati@deploy1003: Started scap sync-world: Backport for [[gerrit:1298875|Remove custom streams (T423148)]] 2427490 wikitext text/x-wiki artiac70g18tpx393t95h2i4e0xm3fg 2427491 2427490 2026-06-16T13:05:30Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2427491 wikitext text/x-wiki id0txffqgpydrpddtxbejkcgpf3kej1 2427492 2427491 2026-06-16T13:05:42Z Stashbot 7414 mfossati@deploy1003: ksarabia, mfossati: Backport for [[gerrit:1298875|Remove custom streams (T423148)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2427492 wikitext text/x-wiki gvyawwmoqbqgwm557n6zz9t6bdkh5lq 2427493 2427492 2026-06-16T13:06:07Z Stashbot 7414 mfossati@deploy1003: ksarabia, mfossati: Continuing with deployment 2427493 wikitext text/x-wiki 6nl71skudulofv4mbxcdzqijdpb09hc 2427494 2427493 2026-06-16T13:07:14Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2427494 wikitext text/x-wiki coaeh809ys2cw1fhs806ftyrtalzbwx 2427495 2427494 2026-06-16T13:08:22Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2427495 wikitext text/x-wiki dkuy9pdvyycje9utzci1aitqo6efwbt 2427496 2427495 2026-06-16T13:08:24Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging1008.eqiad.wmnet with OS trixie 2427496 wikitext text/x-wiki bcucfog0gui63c4a8ui739uzauqpmpo 2427497 2427496 2026-06-16T13:08:57Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1079.eqiad.wmnet with reason: host reimage 2427497 wikitext text/x-wiki t41hlxhgh8xy7iotpigahdv9lve471c 2427498 2427497 2026-06-16T13:12:20Z Stashbot 7414 mfossati@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298875|Remove custom streams (T423148)]] (duration: 08m 35s) 2427498 wikitext text/x-wiki trh4zzpunpyflq3hw0hgzpwpo6os0w1 2427499 2427498 2026-06-16T13:12:26Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1080.eqiad.wmnet with reason: host reimage 2427499 wikitext text/x-wiki 8412wuefhtyu73njme23ueasb887l5c 2427500 2427499 2026-06-16T13:15:01Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2427500 wikitext text/x-wiki 9mg8ynoldbec09t4ix7hngtdphashgb 2427501 2427500 2026-06-16T13:16:29Z Stashbot 7414 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1299626|Replace wgNewUserMessageOnAutoCreate with wgNewUserMessageOnFirstEdit (T426206)]] 2427501 wikitext text/x-wiki 5lu0lykcn778bzs62uabdvhep1ucytk 2427502 2427501 2026-06-16T13:18:33Z Stashbot 7414 dcausse@deploy1003: dcausse, neriah: Backport for [[gerrit:1299626|Replace wgNewUserMessageOnAutoCreate with wgNewUserMessageOnFirstEdit (T426206)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2427502 wikitext text/x-wiki 9hc7ssm4hb0ucvn2sdqhhkdopsg1tuf 2427504 2427502 2026-06-16T13:20:02Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1080.eqiad.wmnet with reason: host reimage 2427504 wikitext text/x-wiki 4n7r9raxyztlfz4qtbmzsceo8nt6r89 2427505 2427504 2026-06-16T13:20:23Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2427505 wikitext text/x-wiki irm57nl4m3388rvuyor83og3qq1swjs 2427506 2427505 2026-06-16T13:20:35Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2427506 wikitext text/x-wiki m8mwppj4siktuuqihv54anknzjrapss 2427507 2427506 2026-06-16T13:21:01Z Stashbot 7414 dcausse@deploy1003: dcausse, neriah: Continuing with deployment 2427507 wikitext text/x-wiki b0vir9ya0sq8o3vmq80qiym75sj1pgf 2427508 2427507 2026-06-16T13:21:34Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2427508 wikitext text/x-wiki k6v678l06azgxmzch0zi04cflv2t9aj 2427509 2427508 2026-06-16T13:22:04Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2427509 wikitext text/x-wiki m3u926h3my3y7a94514q0las7zev31b 2427512 2427509 2026-06-16T13:25:16Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2427512 wikitext text/x-wiki 4zhtxm0oc2gnyi5udqmsaz03l7nwaw2 2427513 2427512 2026-06-16T13:25:19Z Stashbot 7414 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1299626|Replace wgNewUserMessageOnAutoCreate with wgNewUserMessageOnFirstEdit (T426206)]] (duration: 08m 50s) 2427513 wikitext text/x-wiki qroh8q6izzqniq2vfyw1cm6bew4ou9j 2427514 2427513 2026-06-16T13:28:09Z Stashbot 7414 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1302197|translate: remove CirrusSearch endpoints (T425377)]] 2427514 wikitext text/x-wiki gbid0zmi6d73enwerfp0lk3ue1j2npq 2427515 2427514 2026-06-16T13:30:10Z Stashbot 7414 atsuko@deploy1003: atsuko: Backport for [[gerrit:1302197|translate: remove CirrusSearch endpoints (T425377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2427515 wikitext text/x-wiki fjjr5vbqus38r7wbenzzhc8syq3eykl 2427516 2427515 2026-06-16T13:32:47Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2427516 wikitext text/x-wiki r6ewx044zu3ozr9swfyaom739qscgcp 2427517 2427516 2026-06-16T13:32:49Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1079.eqiad.wmnet with OS trixie 2427517 wikitext text/x-wiki liqa89lw3xs21qzwbmaazt7wnnqlb7l 2427518 2427517 2026-06-16T13:34:22Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2153: codfw rack a5 depool for switch maintenance T428020 2427518 wikitext text/x-wiki o730yl93xs8bm3zb9getwzu4i9eyy9q 2427519 2427518 2026-06-16T13:36:14Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2427519 wikitext text/x-wiki t4i7vx7ybrxdapck07r8ombalqf7uym 2427520 2427519 2026-06-16T13:36:30Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2427520 wikitext text/x-wiki n32yr57ti35u6jcfdga8zdpfs7jo3ks 2427521 2427520 2026-06-16T13:36:32Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1080.eqiad.wmnet with OS trixie 2427521 wikitext text/x-wiki 528e8cvmituzlo0g4r1211ofx4apgdt 2427522 2427521 2026-06-16T13:37:05Z Stashbot 7414 atsuko@deploy1003: atsuko: Rolling back deployment 2427522 wikitext text/x-wiki q675zzt0mrfmcwkvoga4y24sc58f2zp 2427523 2427522 2026-06-16T13:39:24Z Stashbot 7414 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1302197|translate: remove CirrusSearch endpoints (T425377)]] (duration: 11m 16s) 2427523 wikitext text/x-wiki qima6ctoxpe9d3iyh2gtudajbyyx5on 2427524 2427523 2026-06-16T13:39:27Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2427524 wikitext text/x-wiki lbaa97u0ysoyafs0m5jmziigbarg1tc 2427525 2427524 2026-06-16T13:40:05Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2154: codfw rack a5 depool for switch maintenance T428020 2427525 wikitext text/x-wiki 6g4m5kjrowfc458vauzth5ls8w7lhcz 2427526 2427525 2026-06-16T13:40:10Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.mysql.pool pool db2157: codfw rack a5 depool for switch maintenance T428020 2427526 wikitext text/x-wiki 9t7w2zu9t7ng0agmf2ovx6wtbcbjg4m 2427527 2427526 2026-06-16T13:41:18Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2427527 wikitext text/x-wiki kaq7ycix2kbl59tjkvzyo7e0frfleic 2427528 2427527 2026-06-16T13:43:12Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2427528 wikitext text/x-wiki 4stb7wykvnoejddlxt15ti0jhaa7tc1 2427529 2427528 2026-06-16T13:43:23Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2427529 wikitext text/x-wiki p95x0szakxg4hntj1es8ek096s95bh2 2427530 2427529 2026-06-16T13:43:57Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2427530 wikitext text/x-wiki mbfvaxr24g03hl1rcvbml4h9jfh4qzw 2427531 2427530 2026-06-16T13:44:10Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2427531 wikitext text/x-wiki azotqqvv9s17slxfk3aatb6yrjuj30v 2427532 2427531 2026-06-16T13:44:13Z Stashbot 7414 atsuko@deploy1003: Started scap sync-world: Backport for [[gerrit:1302850|Revert "translate: remove CirrusSearch endpoints"]] 2427532 wikitext text/x-wiki 8jbd1am1s5nqr979k9o8reh0qjb2mz7 2427533 2427532 2026-06-16T13:46:14Z Stashbot 7414 atsuko@deploy1003: atsuko: Backport for [[gerrit:1302850|Revert "translate: remove CirrusSearch endpoints"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2427533 wikitext text/x-wiki e2mio1epfbmec2slp15x3o5r1k2m9nf 2427534 2427533 2026-06-16T13:47:37Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2427534 wikitext text/x-wiki 0zibs3unaeuylicx8iquf0kcomo0e50 2427535 2427534 2026-06-16T13:47:58Z Stashbot 7414 atsuko@deploy1003: atsuko: Rolling back deployment 2427535 wikitext text/x-wiki 9i6ef2npyew89wlj4qdqtr4yvu9jg18 2427536 2427535 2026-06-16T13:48:24Z Stashbot 7414 atsuko@deploy1003: Finished scap sync-world: Backport for [[gerrit:1302850|Revert "translate: remove CirrusSearch endpoints"]] (duration: 04m 10s) 2427536 wikitext text/x-wiki ts4kb8am5s2iwux5ewoxw5aj69ochnq 2427538 2427536 2026-06-16T13:51:54Z Stashbot 7414 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2427538 wikitext text/x-wiki ivo2dqli4hvlz7mrtto344t433z8v4o 2427539 2427538 2026-06-16T13:52:19Z Stashbot 7414 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2427539 wikitext text/x-wiki 7ma86pidtwxwqm0hs71mjmwunelmaoy 2427540 2427539 2026-06-16T13:52:21Z Stashbot 7414 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2427540 wikitext text/x-wiki 1w1ht17n1liu52fjmwa3svlbni6br09 2427541 2427540 2026-06-16T13:52:47Z Stashbot 7414 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2427541 wikitext text/x-wiki pp5rxiy0p14n5ifhqcqt0ehtaqjg1o7 2427542 2427541 2026-06-16T13:54:42Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2427542 wikitext text/x-wiki qcubfcmgffs4wtsrr95tl581py7me3y 2427543 2427542 2026-06-16T13:56:09Z Stashbot 7414 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1302792|Bump wikimedia/parsoid to 0.24.0-a10 (T417530 T428105 T429187)]], [[gerrit:1302793|Bump wikimedia/parsoid to 0.24.0-a10 (T429187)]] 2427543 wikitext text/x-wiki tmfl47exqzrrjdce63tr1o93ho4ecre 2427544 2427543 2026-06-16T13:57:14Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2427544 wikitext text/x-wiki pd990mt2dfqmhtbe8o6lu4puct9bisd 2427545 2427544 2026-06-16T13:57:38Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2427545 wikitext text/x-wiki nclyz0xnpif7vq6l3fj7vu6yiipp52x 2427546 2427545 2026-06-16T13:58:07Z Stashbot 7414 dcausse@deploy1003: jgiannelos, dcausse: Backport for [[gerrit:1302792|Bump wikimedia/parsoid to 0.24.0-a10 (T417530 T428105 T429187)]], [[gerrit:1302793|Bump wikimedia/parsoid to 0.24.0-a10 (T429187)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2427546 wikitext text/x-wiki 80zl1kmsncea7pgbe4xw4j60fq5j7uy 2427547 2427546 2026-06-16T13:59:57Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2427547 wikitext text/x-wiki ox0si76cu8s2mnnqyn7jqalgraolfc6 2427548 2427547 2026-06-16T14:00:07Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2427548 wikitext text/x-wiki 4ngzslrwow75zkevnxlyvofw7ngxqay 2427549 2427548 2026-06-16T14:02:07Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2427549 wikitext text/x-wiki 2cxxcaw21t1bolki8gg6x7a8yp8xzfl 2427550 2427549 2026-06-16T14:03:16Z Stashbot 7414 dcausse@deploy1003: jgiannelos, dcausse: Continuing with deployment 2427550 wikitext text/x-wiki q36n33xidpkhr7do1huotplngvcbyf4 2427552 2427550 2026-06-16T14:07:15Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2427552 wikitext text/x-wiki 85riki4c9iwnn0jd5fg16ghv965o2sg 2427553 2427552 2026-06-16T14:07:37Z Stashbot 7414 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1302792|Bump wikimedia/parsoid to 0.24.0-a10 (T417530 T428105 T429187)]], [[gerrit:1302793|Bump wikimedia/parsoid to 0.24.0-a10 (T429187)]] (duration: 11m 29s) 2427553 wikitext text/x-wiki kmesj1rlmjvilx9twa3zv9fpnxlyzhm 2427554 2427553 2026-06-16T14:28:14Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2157: codfw rack a5 depool for switch maintenance T428020 2427554 wikitext text/x-wiki 0r7g71x6yr9x9pefcnzw24w5g5a6ij3 2427555 2427554 2026-06-16T14:28:18Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.mysql.pool pool db2175: codfw rack a5 depool for switch maintenance T428020 2427555 wikitext text/x-wiki amvdxahuo12txnyt833c0qcvy1sfsr0 2427557 2427555 2026-06-16T14:46:48Z Stashbot 7414 aokoth@deploy1003: Started deploy [phabricator/deployment@73e57ce]: deploy phab 2427557 wikitext text/x-wiki 9byofr8kn2qxp62bjrc4nuf3i0sx6jz 2427559 2427557 2026-06-16T14:48:57Z Stashbot 7414 aokoth@deploy1003: Finished deploy [phabricator/deployment@73e57ce]: deploy phab (duration: 02m 09s) 2427559 wikitext text/x-wiki ds9kgmoe7j5zu730oxby31uin4e4frq 2427561 2427559 2026-06-16T14:51:32Z Stashbot 7414 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1302804|Hotfix for T428620 (T428620)]] 2427561 wikitext text/x-wiki 8dztdkyqsx2a299a79qt8pokw9gyi10 2427562 2427561 2026-06-16T14:53:34Z Stashbot 7414 awight@deploy1003: seanleong-wmde, awight: Backport for [[gerrit:1302804|Hotfix for T428620 (T428620)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2427562 wikitext text/x-wiki crnw2pr73fwrq5v7ftu24yjic78rx0t 2427563 2427562 2026-06-16T14:54:43Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update records for frproto1001 (formerly payments1008) - cmooney@cumin1003" 2427563 wikitext text/x-wiki 2wnsu66wfm4c5677xip34g5hrshxdtv 2427564 2427563 2026-06-16T14:54:47Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update records for frproto1001 (formerly payments1008) - cmooney@cumin1003" 2427564 wikitext text/x-wiki mxh7tuoydo1cgs2k9gur8mzt7rpkel0 2427565 2427564 2026-06-16T14:54:49Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2427565 wikitext text/x-wiki 246zt134yirmo4yrznce0ccshnrhpbp 2427566 2427565 2026-06-16T14:55:29Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments purgeUserOptions.php --login-age 1 growthexperiments-tour-help-panel # T429352 2427566 wikitext text/x-wiki 0ivlypkuxb3e5lo7r12dowce98vg744 2427568 2427566 2026-06-16T14:57:15Z Stashbot 7414 awight@deploy1003: seanleong-wmde, awight: Continuing with deployment 2427568 wikitext text/x-wiki 9kq5bm9o5k72uvknqvxltou2q1xxg2o 2427570 2427568 2026-06-16T15:01:33Z Stashbot 7414 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1302804|Hotfix for T428620 (T428620)]] (duration: 10m 00s) 2427570 wikitext text/x-wiki mjgj758ye8bt859aqqbzjapapv41y6j 2427571 2427570 2026-06-16T15:03:48Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments purgeUserOptions.php --login-age 1 growthexperiments-tour-homepage-mentorship # T429352 2427571 wikitext text/x-wiki 9n7sx2b808ctg3hgwoar5g1g30eq6rk 2427572 2427571 2026-06-16T15:06:14Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments purgeUserOptions.php --login-age 1 growthexperiments-tour-homepage-discovery # T429352 2427572 wikitext text/x-wiki f2xfryxf065cfuidod12azeqtp7whtf 2427573 2427572 2026-06-16T15:07:00Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments purgeUserOptions.php --login-age 1 growthexperiments-tour-homepage-welcome # T429352 2427573 wikitext text/x-wiki 6a4vt9laxilffsljd2gyarajv794qp1 2427577 2427573 2026-06-16T15:16:22Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2175: codfw rack a5 depool for switch maintenance T428020 2427577 wikitext text/x-wiki 7xuk245qb66caamq7hipyf5u0oiasv0 2427578 2427577 2026-06-16T15:16:26Z Stashbot 7414 cmooney@cumin2002: START - Cookbook sre.mysql.pool pool db2176: codfw rack a5 depool for switch maintenance T428020 2427578 wikitext text/x-wiki b8r4sd793b02r3o0hp83wdtajc653mn 2427581 2427578 2026-06-16T15:33:01Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@a640ed9]: test deploy phab2003 - T427286 2427581 wikitext text/x-wiki sfrlb4zaswjywykrkjj6txuknr1kive 2427583 2427581 2026-06-16T15:33:50Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@a640ed9]: test deploy phab2003 - T427286 (duration: 00m 49s) 2427583 wikitext text/x-wiki mcmc0o5dir3gehyfkc1wfa7mgr1y1a9 2427584 2427583 2026-06-16T15:36:26Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.269.0" for 2 host(s) 2427584 wikitext text/x-wiki 35k1bkmz49ynu136akz33wvu2bft6v6 2427585 2427584 2026-06-16T15:38:18Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.269.0" completed for 2 hosts 2427585 wikitext text/x-wiki 6huedwk403qsgkrnhjuulotuwrq34gi 2427586 2427585 2026-06-16T15:38:48Z Stashbot 7414 urbanecm: Remove `migrateMentorStatusAwayToCommunityConfiguration` from `updatelog` on all wikis in `growthexperiments.dblist` (T409170) 2427586 wikitext text/x-wiki 5j85pgksrz9l3ixweo60edceysu4eyp 2427587 2427586 2026-06-16T15:40:00Z Stashbot 7414 moritzm: installing Tomcat security updates 2427587 wikitext text/x-wiki bkrzlxr9hxl65qzmyngxnlqvf8w7vfm 2427590 2427587 2026-06-16T15:42:47Z Stashbot 7414 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:migrateMentorStatusAway --wiki=abwiki --dry-run # T409170 2427590 wikitext text/x-wiki 5ust6eejitbk9vh4e0x3t9vbkndh9qi 2427599 2427590 2026-06-16T16:04:30Z Stashbot 7414 cmooney@cumin2002: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2176: codfw rack a5 depool for switch maintenance T428020 2427599 wikitext text/x-wiki 8406cfaqg7v6ho955rnp9mphqslnuar 2427600 2427599 2026-06-16T16:06:27Z Stashbot 7414 aokoth@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2002.codfw.wmnet with reason: Phorge Deploy 2427600 wikitext text/x-wiki pyr1qqg0jn5u00edhxk5s8z5n2096s7 2427601 2427600 2026-06-16T16:07:25Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@a640ed9]: deploy phab2002 - T429350 2427601 wikitext text/x-wiki pqc1772ls7lvtx5vqwqrv5r77xwxv3g 2427602 2427601 2026-06-16T16:08:12Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@a640ed9]: deploy phab2002 - T429350 (duration: 00m 47s) 2427602 wikitext text/x-wiki pnicr9si6cl7dvuzk1rm1acg3gkauvc 2427603 2427602 2026-06-16T16:08:36Z Stashbot 7414 aokoth@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phorge Deploy 2427603 wikitext text/x-wiki 6kg54tbdmoh28o8ndln55s386vzy43v 2427604 2427603 2026-06-16T16:08:52Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@a640ed9]: deploy phab1004 - T429350 2427604 wikitext text/x-wiki er5g6rl5ov55x03um545qj0c30cbd6f 2427605 2427604 2026-06-16T16:09:41Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@a640ed9]: deploy phab1004 - T429350 (duration: 00m 45s) 2427605 wikitext text/x-wiki bu47i42u0tl5ao738p2wb47sd3gn2ec 2427613 2427605 2026-06-16T16:35:53Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2427613 wikitext text/x-wiki qd7m1nylz27nnhvsm66xbpm8injtgk7 2427624 2427613 2026-06-16T17:11:11Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1302912|Revert^2 "hCaptcha: Enable for UploadWizard on all wikis with it"]], [[gerrit:1302909|PublishCaptchaHandler: Only require CAPTCHA for UploadWizard (T429322)]], [[gerrit:1302908|PublishCaptchaHandler: Only require CAPTCHA for UploadWizard (T429322)]] 2427624 wikitext text/x-wiki 4vhwgvec7g9k6f6t7mz4irzdlu3ldbw 2427627 2427624 2026-06-16T17:20:46Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging1007.eqiad.wmnet with OS trixie 2427627 wikitext text/x-wiki rzjs96jenfsc2bzp6njqw6arvhtmor6 2427628 2427627 2026-06-16T17:25:51Z Stashbot 7414 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-logging1007.eqiad.wmnet with OS trixie 2427628 wikitext text/x-wiki jzjiqbg3kel3efymlpxipt18s6lrcr7 2427629 2427628 2026-06-16T17:27:49Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host conf2007.codfw.wmnet with OS trixie 2427629 wikitext text/x-wiki h3mqv0mnti7qnk42zf0f8qj11d4ncik 2427630 2427629 2026-06-16T17:30:00Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1302912|Revert^2 "hCaptcha: Enable for UploadWizard on all wikis with it"]], [[gerrit:1302909|PublishCaptchaHandler: Only require CAPTCHA for UploadWizard (T429322)]], [[gerrit:1302908|PublishCaptchaHandler: Only require CAPTCHA for UploadWizard (T429322)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified t 2427630 wikitext text/x-wiki cl2etz581s8dh5pyt0f55quk5l3vgnj 2427632 2427630 2026-06-16T17:30:34Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2427632 wikitext text/x-wiki 5cl95mprk8rib99t2zfsdhdiiq6k1lq 2427635 2427632 2026-06-16T17:38:03Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2427635 wikitext text/x-wiki k7o3nfvh0g5kkbv6cci2767kqa37yz0 2427636 2427635 2026-06-16T17:43:29Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1302912|Revert^2 "hCaptcha: Enable for UploadWizard on all wikis with it"]], [[gerrit:1302909|PublishCaptchaHandler: Only require CAPTCHA for UploadWizard (T429322)]], [[gerrit:1302908|PublishCaptchaHandler: Only require CAPTCHA for UploadWizard (T429322)]] (duration: 32m 19s) 2427636 wikitext text/x-wiki 6gdvej4gdg4g4xl26zdhkku3x8x5lwz 2427637 2427636 2026-06-16T17:43:54Z Stashbot 7414 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host conf2007.codfw.wmnet with OS trixie 2427637 wikitext text/x-wiki p4ppxkpl97earyf4oodyir9sgkif7es 2427638 2427637 2026-06-16T17:44:28Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change mgmt name for frproto1001 - cmooney@cumin1003" 2427638 wikitext text/x-wiki iv70n4gs22tkixn5pjyqi8v7qjjec4d 2427639 2427638 2026-06-16T17:46:16Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reimage for host dns5004.wikimedia.org with OS bookworm 2427639 wikitext text/x-wiki o2e0x9numqouy9ya7q9toq691xjagsj 2427640 2427639 2026-06-16T17:46:51Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.move-vlan for host dns5004 2427640 wikitext text/x-wiki iq2ww17w0sf9zwd3wuadekoje6uv99t 2427641 2427640 2026-06-16T17:47:48Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change mgmt name for frproto1001 - cmooney@cumin1003" 2427641 wikitext text/x-wiki qfdajbqwrlqo9bq85846vwkfhe7o72w 2427642 2427641 2026-06-16T17:47:50Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2427642 wikitext text/x-wiki p1mzi0g2uuz2iij6ire0m6uwjvy1bm5 2427644 2427642 2026-06-16T17:53:27Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=dns5004.* 2427644 wikitext text/x-wiki ai0e91rov751bvm8zsrgki5cgexkdle 2427646 2427644 2026-06-16T17:59:44Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2427646 wikitext text/x-wiki 3686vdlss325pfrscstyhsvcwj9enww 2427647 2427646 2026-06-16T18:00:23Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2427647 wikitext text/x-wiki hurw8a3k3c61onixtkdp49sr05sebaa 2427648 2427647 2026-06-16T18:00:50Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.dns.netbox 2427648 wikitext text/x-wiki lj04ugaumma4kyigi09il1wcfz53skb 2427649 2427648 2026-06-16T18:02:08Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2427649 wikitext text/x-wiki fgjyy2cl3fdp4s2dht7drqtb94m9txk 2427650 2427649 2026-06-16T18:08:09Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host dns5004 - brett@cumin2002" 2427650 wikitext text/x-wiki 49136uoxgp6784x2ppcet5lzd3ap002 2427651 2427650 2026-06-16T18:08:14Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host dns5004 - brett@cumin2002" 2427651 wikitext text/x-wiki cmvfj84gwtsf8ay0r3hhw1wujdghxdh 2427652 2427651 2026-06-16T18:08:17Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2427652 wikitext text/x-wiki bboq84mr0sscy9co0sxg6i5j26cvuvq 2427653 2427652 2026-06-16T18:08:19Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.dns.wipe-cache dns5004.wikimedia.org 8.166.102.103.in-addr.arpa 8.0.0.0.6.6.1.0.2.0.1.0.3.0.1.0.1.0.0.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors 2427653 wikitext text/x-wiki 1s0vlsh29xyt0whqzoiedqkgdxs4b1d 2427654 2427653 2026-06-16T18:08:21Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dns5004.wikimedia.org 8.166.102.103.in-addr.arpa 8.0.0.0.6.6.1.0.2.0.1.0.3.0.1.0.1.0.0.0.0.0.5.e.2.f.d.0.1.0.0.2.ip6.arpa on all recursors 2427654 wikitext text/x-wiki h9a1c6k81hobl48ujhi3vpv20rh8f73 2427655 2427654 2026-06-16T18:08:22Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host dns5004 2427655 wikitext text/x-wiki o2oq014bz76b6a2cvda7kioiyysy9bf 2427657 2427655 2026-06-16T18:12:09Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dns5004 2427657 wikitext text/x-wiki s8hs9z9wqlia8uxj28ccxqe19gwceie 2427658 2427657 2026-06-16T18:12:11Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host dns5004 2427658 wikitext text/x-wiki p8n9h8fndb8uj5wargerf6q7zed5x4f 2427659 2427658 2026-06-16T18:12:42Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2427659 wikitext text/x-wiki n0oewn36guxxa39fksbscq5nkve3leg 2427661 2427659 2026-06-16T18:23:42Z Stashbot 7414 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.7 refs T423916 2427661 wikitext text/x-wiki 4wbhxheqsf2fbt16ewasvo1qpmtaion 2427662 2427661 2026-06-16T18:30:00Z Stashbot 7414 robh@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS trixie 2427662 wikitext text/x-wiki p3gk150qtofivvdzx4449cox0i6ettn 2427663 2427662 2026-06-16T18:33:22Z Stashbot 7414 robh@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-ctrl2006.codfw.wmnet with OS trixie 2427663 wikitext text/x-wiki qat8i5j0ifqw8n58kds4rwjjvpc25g2 2427664 2427663 2026-06-16T18:34:46Z Stashbot 7414 robh@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS trixie 2427664 wikitext text/x-wiki s07bto59faniccpeoczkj202lnvlu2e 2427665 2427664 2026-06-16T18:35:57Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2427665 wikitext text/x-wiki dq5672ogt608uyydjwphy52zeexqjth 2427666 2427665 2026-06-16T18:39:48Z Stashbot 7414 eevans@deploy1003: helmfile [staging] START helmfile.d/services/data-gateway: apply 2427666 wikitext text/x-wiki 328zgu15mz87zc9wolm461gdce03y8f 2427667 2427666 2026-06-16T18:39:54Z Stashbot 7414 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply 2427667 wikitext text/x-wiki g2399qd2f3dli95lc6j4ho8beszc45p 2427668 2427667 2026-06-16T18:39:58Z Stashbot 7414 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/data-gateway: apply 2427668 wikitext text/x-wiki amt1i5hj7musjsjmpmz96v3ff0q7p6h 2427669 2427668 2026-06-16T18:40:23Z Stashbot 7414 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply 2427669 wikitext text/x-wiki nz7r3hk7ve8l3v9g1wla50nkxd2ocp1 2427670 2427669 2026-06-16T18:41:09Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on dns5004.wikimedia.org with reason: host reimage 2427670 wikitext text/x-wiki 9j94ps7oc6ef53j99b59j0uoszkhv17 2427671 2427670 2026-06-16T18:41:23Z Stashbot 7414 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/data-gateway: apply 2427671 wikitext text/x-wiki 2yayi7dr957ks4csc8ylet41nbh4uad 2427672 2427671 2026-06-16T18:41:42Z Stashbot 7414 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/data-gateway: apply 2427672 wikitext text/x-wiki j0ncey96r8b0e8szmuacn3e8wp99cli 2427673 2427672 2026-06-16T18:44:28Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns5004.wikimedia.org with reason: host reimage 2427673 wikitext text/x-wiki 7381sltwhov4ppc8zzaw6fumvngvjqn 2427674 2427673 2026-06-16T18:45:45Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427674 wikitext text/x-wiki cfna332qtdds3bcioo0bls7bucehbly 2427676 2427674 2026-06-16T18:48:48Z Stashbot 7414 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1302274|Disable ShortUrl on hiwiki, hiwikiversity, maiwiki, knwiki, knwikisource, tcywiki (T107188)]] 2427676 wikitext text/x-wiki bb32kr5y6jol7ri3h00xau6qs2sf5sy 2427677 2427676 2026-06-16T18:51:09Z Stashbot 7414 krinkle@deploy1003: krinkle: Backport for [[gerrit:1302274|Disable ShortUrl on hiwiki, hiwikiversity, maiwiki, knwiki, knwikisource, tcywiki (T107188)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2427677 wikitext text/x-wiki s27l0rc4nthtdlru4rfw6wl0vs04inj 2427678 2427677 2026-06-16T18:52:56Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427678 wikitext text/x-wiki lxghjtbsw0dyvg95foes2pt4fq535f8 2427679 2427678 2026-06-16T18:55:40Z Stashbot 7414 krinkle@deploy1003: krinkle: Continuing with deployment 2427679 wikitext text/x-wiki ckj6tc2pw7gdbvz1j359v9qz7zz0pch 2427680 2427679 2026-06-16T18:56:16Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427680 wikitext text/x-wiki llhgt9mdwqfmqzue37uwctfy3sfmqin 2427681 2427680 2026-06-16T18:56:42Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2427681 wikitext text/x-wiki 21ya3sdf50dxf3tar3i0osh1fpkwvbk 2427682 2427681 2026-06-16T18:58:24Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2427682 wikitext text/x-wiki hns55y6208zoor4j0keail9fcqp53hq 2427683 2427682 2026-06-16T19:00:06Z Stashbot 7414 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1302274|Disable ShortUrl on hiwiki, hiwikiversity, maiwiki, knwiki, knwikisource, tcywiki (T107188)]] (duration: 11m 18s) 2427683 wikitext text/x-wiki 8irjb0gr3qvkjsj0bmi00gm191kso2s 2427684 2427683 2026-06-16T19:02:54Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427684 wikitext text/x-wiki bsyqsrm46uhv72zj2xyenj6m5nglq7v 2427685 2427684 2026-06-16T19:08:17Z Stashbot 7414 robh@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-ctrl2006.codfw.wmnet with OS trixie 2427685 wikitext text/x-wiki o4t1j7d8idw4zmoa7x6vja5dp7goe5p 2427686 2427685 2026-06-16T19:08:19Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS trixie 2427686 wikitext text/x-wiki tvoje35ychi54odmrsm44pjt24ris1o 2427688 2427686 2026-06-16T19:18:18Z Stashbot 7414 topranks: restarting grpc server on eqiad SR-Linux switches to recover from problem of no free threads T429242 2427688 wikitext text/x-wiki oc8erliqk3lcqu2tom0d0gmhwnp0se6 2427689 2427688 2026-06-16T19:27:23Z Stashbot 7414 jasmine@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-ctrl2006.codfw.wmnet with OS trixie 2427689 wikitext text/x-wiki n65wgc2z769sw52j52ybn6dyuzsjux8 2427690 2427689 2026-06-16T19:30:07Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-restart-haproxy rolling restart of HAProxy on A:cp - OpenSSL update () 2427690 wikitext text/x-wiki s0i1zzndw3r0kpa0ke1cx9586x0dsnl 2427691 2427690 2026-06-16T19:31:00Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.dns.netbox 2427691 wikitext text/x-wiki 3f6ci0wbe0qc9m55jaq3lih7c8znu1o 2427692 2427691 2026-06-16T19:34:04Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2427692 wikitext text/x-wiki hwoqtcs9rbk8skq0hosd2d8igzobbts 2427693 2427692 2026-06-16T19:35:01Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-restart-haproxy rolling restart of HAProxy on A:cp - OpenSSL update () 2427693 wikitext text/x-wiki 9tf2g77o2bd6a3ww9zn0fk06owgfgfm 2427694 2427693 2026-06-16T19:39:13Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427694 wikitext text/x-wiki j1rv9w5kmtxe2a2fft8308pqfy5abt1 2427695 2427694 2026-06-16T19:45:17Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1001.eqiad.wmnet with OS bookworm 2427695 wikitext text/x-wiki i7rkojbinrbtfcjm0mys97hcw2ubzag 2427696 2427695 2026-06-16T19:45:19Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs2001.codfw.wmnet with OS bookworm 2427696 wikitext text/x-wiki 0zp6tk3ovvyki08zhmk3f9igatjzj5j 2427697 2427696 2026-06-16T19:46:25Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427697 wikitext text/x-wiki 7577a6vnz9datxdgbo108xi7b2knx9x 2427698 2427697 2026-06-16T19:47:49Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427698 wikitext text/x-wiki o8u95le5tzhcgunujsxt7ty58wth55s 2427699 2427698 2026-06-16T19:54:34Z Stashbot 7414 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427699 wikitext text/x-wiki rfg37gic7llvt2xq9xkuegfcv8oh4bs 2427700 2427699 2026-06-16T19:55:08Z Stashbot 7414 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS trixie 2427700 wikitext text/x-wiki eycncjlfy59bkhn3gxffyv36fvlqr7s 2427701 2427700 2026-06-16T19:55:39Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs2001.codfw.wmnet with OS bookworm 2427701 wikitext text/x-wiki 9knvafdrtys73efuumoie3qcd1l7e6p 2427702 2427701 2026-06-16T19:56:30Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-wdqs1001.eqiad.wmnet with reason: host reimage 2427702 wikitext text/x-wiki 5lej920okpz9ww9n4tipz4ywla2e7j7 2427703 2427702 2026-06-16T20:00:28Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-wdqs1001.eqiad.wmnet with reason: host reimage 2427703 wikitext text/x-wiki 42ugt06qlfhubwnxjfjuyfyouofwv5v 2427707 2427703 2026-06-16T20:09:39Z Stashbot 7414 jasmine@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-ctrl2006.codfw.wmnet with OS trixie 2427707 wikitext text/x-wiki syvotvh7gftp3d2xizuczap59eiov38 2427710 2427707 2026-06-16T20:17:47Z Stashbot 7414 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1302320|EditChecks: Namespace tracking object for seen/shown/used checks]] 2427710 wikitext text/x-wiki p577tcqwa99su68sf9q7fio3xzour43 2427711 2427710 2026-06-16T20:18:13Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003" 2427711 wikitext text/x-wiki oh4cqvdirvvhfltehwsw8yxcjhnkk7h 2427713 2427711 2026-06-16T20:19:42Z Stashbot 7414 kemayo@deploy1003: kemayo: Backport for [[gerrit:1302320|EditChecks: Namespace tracking object for seen/shown/used checks]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2427713 wikitext text/x-wiki 9hhpu62gmq3rds6ipmj3nn7yzpb3txq 2427715 2427713 2026-06-16T20:23:16Z Stashbot 7414 kemayo@deploy1003: kemayo: Continuing with deployment 2427715 wikitext text/x-wiki m9cpth8qb153dgrbh6zgl1smse7sier 2427717 2427715 2026-06-16T20:25:56Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns5004.*,service=authdns-update 2427717 wikitext text/x-wiki 53j628lybyy95les1fvry5i5avw7grd 2427719 2427717 2026-06-16T20:26:01Z Stashbot 7414 brett@dns5004: START - running authdns-update 2427719 wikitext text/x-wiki pkuel22dlc4v5zqkg20dpsa00r4m44j 2427720 2427719 2026-06-16T20:26:24Z Stashbot 7414 brett@dns5004: START - running authdns-update 2427720 wikitext text/x-wiki gajx0fio6ywos3oqrojx4kdcfpaxwrh 2427721 2427720 2026-06-16T20:27:37Z Stashbot 7414 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1302320|EditChecks: Namespace tracking object for seen/shown/used checks]] (duration: 09m 50s) 2427721 wikitext text/x-wiki qbexnr0xk2qqsqythninuwhuafrbcrc 2427724 2427721 2026-06-16T20:28:08Z Stashbot 7414 brett@dns5004: FAIL - running authdns-update 2427724 wikitext text/x-wiki 75uys3vq2lzqkm23218jztmd5pkets7 2427725 2427724 2026-06-16T20:29:03Z Stashbot 7414 brett@dns5004: START - running authdns-update 2427725 wikitext text/x-wiki 3yhwqxdv1l30yayk4n01we8pd6o7ral 2427726 2427725 2026-06-16T20:30:45Z Stashbot 7414 brett@dns5004: FAIL - running authdns-update 2427726 wikitext text/x-wiki 10vdss900tudo62exw76l6144qsm7cz 2427727 2427726 2026-06-16T20:30:55Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns5004.wikimedia.org with OS bookworm 2427727 wikitext text/x-wiki f0cfjinsb8x04y2yaso9ah25ibkgn07 2427728 2427727 2026-06-16T20:31:30Z Stashbot 7414 brett@dns1004: START - running authdns-update 2427728 wikitext text/x-wiki 9terv8lux0v3xphvdsmoj2fq50u7ta5 2427730 2427728 2026-06-16T20:33:11Z Stashbot 7414 brett@dns1004: END - running authdns-update 2427730 wikitext text/x-wiki 6nzmtwdonvrj6tzlfsdzy0t2b89ph3a 2427731 2427730 2026-06-16T20:40:14Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns5004.* 2427731 wikitext text/x-wiki 9ctyjfmu2qkxy7w47cy5bzeccko86ty 2427732 2427731 2026-06-16T20:41:52Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1302890|Guard round function with a supports query (T424596)]], [[gerrit:1302935|Add wprov parameter to home link (T429268)]] 2427732 wikitext text/x-wiki qc62o706szba54tu0ukrkdesf0z55ql 2427733 2427732 2026-06-16T20:43:47Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1302890|Guard round function with a supports query (T424596)]], [[gerrit:1302935|Add wprov parameter to home link (T429268)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2427733 wikitext text/x-wiki ddraojxlypc0f20dvt2e61jfkek9i74 2427735 2427733 2026-06-16T20:47:01Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Continuing with deployment 2427735 wikitext text/x-wiki ovwa314oyihyaz3tk653ul20gkldak1 2427737 2427735 2026-06-16T20:51:20Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1302890|Guard round function with a supports query (T424596)]], [[gerrit:1302935|Add wprov parameter to home link (T429268)]] (duration: 09m 28s) 2427737 wikitext text/x-wiki 8qza0yh28r620jke3njp3z8ezvx25v4 2427739 2427737 2026-06-16T20:54:42Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2043.* 2427739 wikitext text/x-wiki 848izxch18ipltefdk3ctnknbpkqktn 2427745 2427739 2026-06-16T21:08:09Z Stashbot 7414 robh@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS trixie 2427745 wikitext text/x-wiki lz2r61fg3j8sg846uyevgxx438u3lx6 2427746 2427745 2026-06-16T21:10:49Z Stashbot 7414 robh@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-ctrl2006.codfw.wmnet with OS trixie 2427746 wikitext text/x-wiki 5tzajas4mfk34jyb7s4srmchb5cxy53 2427747 2427746 2026-06-16T21:15:34Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1302934|Update definition of html heading to match Parsoid/core (T417530 T417531 T428677)]] 2427747 wikitext text/x-wiki 85zh34uozpy1nz4uwivgy89lbo7umjv 2427750 2427747 2026-06-16T21:17:28Z Stashbot 7414 cscott@deploy1003: arlolra, cscott: Backport for [[gerrit:1302934|Update definition of html heading to match Parsoid/core (T417530 T417531 T428677)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2427750 wikitext text/x-wiki dm8kh1lt41gwgekm6cax01p6l1bzbcs 2427753 2427750 2026-06-16T21:20:05Z Stashbot 7414 robh@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS bookworm 2427753 wikitext text/x-wiki 69lwkxb01imsbkdt8kdu885klgp9obz 2427754 2427753 2026-06-16T21:20:42Z Stashbot 7414 urbanecm@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply 2427754 wikitext text/x-wiki hiytfscnet387sy8u20zsohp2wwuvzk 2427755 2427754 2026-06-16T21:21:43Z Stashbot 7414 urbanecm@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply 2427755 wikitext text/x-wiki 70jab8wqvdr9s7orj2wt5krsuqhvzf7 2427756 2427755 2026-06-16T21:23:10Z Stashbot 7414 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply 2427756 wikitext text/x-wiki imojj0svzfrfec4c6seovoo1kbcsnnt 2427757 2427756 2026-06-16T21:24:42Z Stashbot 7414 robh@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-ctrl2006.codfw.wmnet with OS bookworm 2427757 wikitext text/x-wiki 7djybzqdfti2o0pf6an6ux4rryisflq 2427758 2427757 2026-06-16T21:24:51Z Stashbot 7414 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply 2427758 wikitext text/x-wiki doglribst1i1xvyv7cpzx693riht5sp 2427759 2427758 2026-06-16T21:25:04Z Stashbot 7414 urbanecm@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply 2427759 wikitext text/x-wiki d9cq2bzbpajnjwt05n0n8uq9kg0hqad 2427761 2427759 2026-06-16T21:26:51Z Stashbot 7414 urbanecm@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply 2427761 wikitext text/x-wiki 63zn8eaf9bl4b81s7airvhitz97zvx0 2427762 2427761 2026-06-16T21:29:54Z Stashbot 7414 cscott@deploy1003: arlolra, cscott: Continuing with deployment 2427762 wikitext text/x-wiki gusnwhmkw4dlli0viuaihlndtjee9w7 2427764 2427762 2026-06-16T21:30:42Z Stashbot 7414 robh@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427764 wikitext text/x-wiki 8dlucq8rd8kei1gpvofgktqn5u57xwt 2427765 2427764 2026-06-16T21:31:21Z Stashbot 7414 robh@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427765 wikitext text/x-wiki nlwchvci3044simb9endbaiorui4v7u 2427766 2427765 2026-06-16T21:32:25Z Stashbot 7414 robh@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427766 wikitext text/x-wiki ki2mb7o01tg5qd3zhdse8x3hny9lb1u 2427767 2427766 2026-06-16T21:34:15Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1302934|Update definition of html heading to match Parsoid/core (T417530 T417531 T428677)]] (duration: 18m 41s) 2427767 wikitext text/x-wiki 8d87de4eo719ws2i5b7i0v1rkzeryyh 2427772 2427767 2026-06-16T21:39:00Z Stashbot 7414 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427772 wikitext text/x-wiki 5bvx26ubuwwhewj87vp281ep4za7m83 2427773 2427772 2026-06-16T21:45:01Z Stashbot 7414 robh@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS trixie 2427773 wikitext text/x-wiki 8h5dyi7r4yaw3ula4656zvrqu14pa9y 2427774 2427773 2026-06-16T21:46:08Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2427774 wikitext text/x-wiki nh8qfl11itv1qnoi89p7gh412vkr52u 2427775 2427774 2026-06-16T21:46:21Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2427775 wikitext text/x-wiki 7nu4mtg4uqvr74bfljm6g5yien90b53 2427776 2427775 2026-06-16T21:46:25Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2427776 wikitext text/x-wiki 8oxhedr8f1vbvo86ejgotc8s6k6ori0 2427777 2427776 2026-06-16T21:46:44Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2427777 wikitext text/x-wiki 6x6ze9j4i436idwl86qj5ahdkabc84p 2427778 2427777 2026-06-16T21:46:48Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2427778 wikitext text/x-wiki 09jg2swlkh2iqxk4m1a2jxz3lp8ovs8 2427780 2427778 2026-06-16T21:48:27Z Stashbot 7414 robh@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-ctrl2006.codfw.wmnet with OS trixie 2427780 wikitext text/x-wiki 7m7ew3cn7bs8jat41ohioan1wvgeyt2 2427781 2427780 2026-06-16T21:48:41Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2427781 wikitext text/x-wiki 64u8kthkohdlab0mqutz8hjfwqpmdwv 2427782 2427781 2026-06-16T21:49:17Z Stashbot 7414 robh@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS bookworm 2427782 wikitext text/x-wiki 35ebqnxpt6987q0pi9f2do77r9xxgvv 2427783 2427782 2026-06-16T21:50:29Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2427783 wikitext text/x-wiki hvy2knmfk8va8fxrw9yd0uyr2tkmqsm 2427784 2427783 2026-06-16T21:52:25Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2427784 wikitext text/x-wiki g3w401i8k6gwk00frhl597b97j3swe3 2427785 2427784 2026-06-16T21:59:15Z Stashbot 7414 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1302953|Update VE core submodule to master (0930c3a9e) (T406841 T429174 T397501 T424632 T429355)]], [[gerrit:1302952|Update VE core submodule to master (0930c3a9e) (T397501 T424632 T429355)]] 2427785 wikitext text/x-wiki nkp75uro28w9597rf6ht50vmw6ars7z 2427786 2427785 2026-06-16T22:01:20Z Stashbot 7414 kemayo@deploy1003: kemayo: Backport for [[gerrit:1302953|Update VE core submodule to master (0930c3a9e) (T406841 T429174 T397501 T424632 T429355)]], [[gerrit:1302952|Update VE core submodule to master (0930c3a9e) (T397501 T424632 T429355)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2427786 wikitext text/x-wiki 69zzpeu4f452yk8f8rbncma7jfn79i9 2427787 2427786 2026-06-16T22:02:51Z Stashbot 7414 kemayo@deploy1003: kemayo: Continuing with deployment 2427787 wikitext text/x-wiki biq1vxv6o9gdnb8zz5w5hx0o9ertdxv 2427788 2427787 2026-06-16T22:07:19Z Stashbot 7414 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1302953|Update VE core submodule to master (0930c3a9e) (T406841 T429174 T397501 T424632 T429355)]], [[gerrit:1302952|Update VE core submodule to master (0930c3a9e) (T397501 T424632 T429355)]] (duration: 08m 11s) 2427788 wikitext text/x-wiki qc40z4xgm5h4l6pgli35vtz1ldq6zzs 2427789 2427788 2026-06-16T22:08:49Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2427789 wikitext text/x-wiki numi306t0hnc0lf213rv96de3y4xxiv 2427790 2427789 2026-06-16T22:09:01Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2427790 wikitext text/x-wiki 54eyhtyq1e3bnpukrm9oqkn26wtmtyw 2427791 2427790 2026-06-16T22:30:05Z Stashbot 7414 robh@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-ctrl2006.codfw.wmnet with OS bookworm 2427791 wikitext text/x-wiki 93wcmj1xode845jcn7d4yyhgldd2nia 2427792 2427791 2026-06-16T22:37:15Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host wikikube-ctrl2006.codfw.wmnet 2427792 wikitext text/x-wiki 3ic4x0wodqr7h8a6gmtvdegjgrf306n 2427795 2427792 2026-06-16T22:49:55Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply 2427795 wikitext text/x-wiki ra036pbc6woir8ugx89sm4h9mbah0ri 2427796 2427795 2026-06-16T22:50:06Z Stashbot 7414 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply 2427796 wikitext text/x-wiki gkffpuygimpi40pwkh7x32i99ips3r6 2427797 2427796 2026-06-16T22:50:28Z Stashbot 7414 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host wikikube-ctrl2006.codfw.wmnet 2427797 wikitext text/x-wiki azfie8hlkjnv7ugy7b93pp650obinwq 2427798 2427797 2026-06-16T22:52:26Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host wikikube-ctrl2006.codfw.wmnet 2427798 wikitext text/x-wiki ghoi94kv42svcbylqudisbxh3ujh9lr 2427799 2427798 2026-06-16T22:57:11Z Stashbot 7414 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host wikikube-ctrl2006.codfw.wmnet 2427799 wikitext text/x-wiki jal7gnxg4ftmo8z01jwpxypt9ok0zov 2427800 2427799 2026-06-16T22:57:44Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host wikikube-ctrl2006.codfw.wmnet 2427800 wikitext text/x-wiki ljwndg60qrvqhlgynqy478un9ub18b3 2427801 2427800 2026-06-16T23:01:30Z Stashbot 7414 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host wikikube-ctrl2006.codfw.wmnet 2427801 wikitext text/x-wiki 9bgeypypiio2dh6w2p4tlvyi97buq32 2427802 2427801 2026-06-16T23:02:24Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-restart-haproxy (exit_code=0) rolling restart of HAProxy on A:cp - OpenSSL update () 2427802 wikitext text/x-wiki 72a64dqwjycwguioloiwn0z21ps9nq3 2427803 2427802 2026-06-16T23:03:58Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS trixie 2427803 wikitext text/x-wiki jfdajwplitui2vlakgyjjov9s9vbwvo 2427804 2427803 2026-06-16T23:38:36Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl2006.codfw.wmnet with reason: host reimage 2427804 wikitext text/x-wiki pjgxhdtk7f9ty9ksqx3e1f02zx6vwn1 2427805 2427804 2026-06-16T23:44:36Z Stashbot 7414 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl2006.codfw.wmnet with reason: host reimage 2427805 wikitext text/x-wiki iasz80ycn1vi2il1etbtxwq4upkauwk 2427806 2427805 2026-06-17T00:01:25Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" 2427806 wikitext text/x-wiki t6jsjn6lubspk8jb8uhlni803oocudi 2427807 2427806 2026-06-17T00:01:48Z Stashbot 7414 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" 2427807 wikitext text/x-wiki s35u7vg3rq40mza3u0m9d3ounoigu8e 2427808 2427807 2026-06-17T00:01:50Z Stashbot 7414 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl2006.codfw.wmnet with OS trixie 2427808 wikitext text/x-wiki dzdgythrx2dlixekz2zz4104bnsxexc 2427813 2427808 2026-06-17T02:01:27Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2427813 wikitext text/x-wiki nswme98727sctogzq1v3ie1cunu4ti7 2427814 2427813 2026-06-17T02:08:02Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 35s) 2427814 wikitext text/x-wiki 933pgp0quk5un3u70v7q2se7px4pgda 2427824 2427814 2026-06-17T05:37:11Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2427824 wikitext text/x-wiki 2y3iz0zjrn38zvw1cyzv5jzsudv9dtj 2427825 2427824 2026-06-17T05:37:31Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1037: Upgrading es1037.eqiad.wmnet 2427825 wikitext text/x-wiki ha0ll5hfi7r3i6bvwhpqo3u15z4rkz4 2427826 2427825 2026-06-17T05:37:41Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1037: Upgrading es1037.eqiad.wmnet 2427826 wikitext text/x-wiki 8un1m8c9lqqelp6uwbqbs4qui3t9akc 2427827 2427826 2026-06-17T05:38:39Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1037.eqiad.wmnet with OS trixie 2427827 wikitext text/x-wiki 6emownh45p18u56tqksjyywvy1ywgty 2427828 2427827 2026-06-17T05:54:18Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1037.eqiad.wmnet with reason: host reimage 2427828 wikitext text/x-wiki kenkalmhuktqd6mxmam5eo2df1f1vaf 2427829 2427828 2026-06-17T05:59:13Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1037.eqiad.wmnet with reason: host reimage 2427829 wikitext text/x-wiki 724kzmbppioaxndzsy9ho2b0229hx49 2427830 2427829 2026-06-17T06:16:21Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1037.eqiad.wmnet with OS trixie 2427830 wikitext text/x-wiki bg8irdbc365jt58rdsdnvl5kgooruok 2427831 2427830 2026-06-17T06:28:35Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1037: Migration of es1037.eqiad.wmnet completed 2427831 wikitext text/x-wiki 36enmgzern0i5vuj1wxbtjdrbt80tn0 2427832 2427831 2026-06-17T06:46:02Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Haproxy provenance maps in HP; UX changes - oblivian@cumin1003" 2427832 wikitext text/x-wiki t4pfaqm2zk5u08fhscibp2u8xhpiawl 2427833 2427832 2026-06-17T06:46:05Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Haproxy provenance maps in HP; UX changes - oblivian@cumin1003 2427833 wikitext text/x-wiki qi3tj2402i4ztewz6asyhzsjkh4xa0z 2427834 2427833 2026-06-17T06:46:54Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Haproxy provenance maps in HP; UX changes - oblivian@cumin1003 2427834 wikitext text/x-wiki rm9uplhm2qk753muhszzz524jo787zn 2427835 2427834 2026-06-17T06:46:56Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Haproxy provenance maps in HP; UX changes - oblivian@cumin1003" 2427835 wikitext text/x-wiki h7ry60x5l0fl6okmpkfpw9967fwkval 2427836 2427835 2026-06-17T06:52:19Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "revert deployment - oblivian@cumin1003" 2427836 wikitext text/x-wiki 85vjbs392mznmafcq5xb8z8817k5s0q 2427837 2427836 2026-06-17T06:52:21Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: revert deployment - oblivian@cumin1003 2427837 wikitext text/x-wiki 25353ka1f2erx34yamg3oqn5xa4qfzi 2427838 2427837 2026-06-17T06:53:13Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: revert deployment - oblivian@cumin1003 2427838 wikitext text/x-wiki 960ouwtzucfsdomco9ghjxjounnirhl 2427839 2427838 2026-06-17T06:53:15Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "revert deployment - oblivian@cumin1003" 2427839 wikitext text/x-wiki kcvmykpo2g69ljjo2ctep5ldoys4mu9 2427842 2427839 2026-06-17T07:14:06Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1037: Migration of es1037.eqiad.wmnet completed 2427842 wikitext text/x-wiki eszirb1os0dspo2javj2hzw8fgjr0yy 2427843 2427842 2026-06-17T07:14:08Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2427843 wikitext text/x-wiki jdvlmt8dwpd6abemm7pkw8399dxoars 2427844 2427843 2026-06-17T07:15:26Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2427844 wikitext text/x-wiki mufc8e79rbb7bo4x16kzjrrj4twh24s 2427845 2427844 2026-06-17T07:15:46Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1044: Upgrading es1044.eqiad.wmnet 2427845 wikitext text/x-wiki 5boo0ixphtbjg4pqlpzl1inn97h879p 2427847 2427845 2026-06-17T07:16:17Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1044: Upgrading es1044.eqiad.wmnet 2427847 wikitext text/x-wiki r1dk5frjzedmdqpsd3nksjk7wz9k3qn 2427848 2427847 2026-06-17T07:17:27Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1044.eqiad.wmnet with OS trixie 2427848 wikitext text/x-wiki m7yzwx7hzrdpgtxmpfm8u9jnc197yyu 2427849 2427848 2026-06-17T07:21:42Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Haproxy provenance maps in HP; UX changes (attempt 3) - oblivian@cumin1003" 2427849 wikitext text/x-wiki nls70yh1v835bgu1aurzvfi303lj2xq 2427850 2427849 2026-06-17T07:21:44Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Haproxy provenance maps in HP; UX changes (attempt 3) - oblivian@cumin1003 2427850 wikitext text/x-wiki pbzva0yequohmqialcin0lhwqj3i6uu 2427851 2427850 2026-06-17T07:21:51Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2427851 wikitext text/x-wiki hwikyenepolk3lgeiqb0pe42rzflucq 2427852 2427851 2026-06-17T07:22:32Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Haproxy provenance maps in HP; UX changes (attempt 3) - oblivian@cumin1003 2427852 wikitext text/x-wiki k5xwmissk8r6nnx414jr47sxpsrlokd 2427853 2427852 2026-06-17T07:22:34Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Haproxy provenance maps in HP; UX changes (attempt 3) - oblivian@cumin1003" 2427853 wikitext text/x-wiki 41kwsm74sxw2ddf7twlqjrzoxi3yuqq 2427854 2427853 2026-06-17T07:22:58Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2427854 wikitext text/x-wiki qgqpu8bb3trbfw5tvt9de8u99otplqi 2427855 2427854 2026-06-17T07:23:24Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host conf2007.codfw.wmnet with OS trixie 2427855 wikitext text/x-wiki 3k4cc7eh1xfkygvieo11ayfkm9todrb 2427856 2427855 2026-06-17T07:23:54Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . 2427856 wikitext text/x-wiki rwjb4iijyjn3ccsqllhuh3o48mklwj7 2427857 2427856 2026-06-17T07:30:44Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host conf2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427857 wikitext text/x-wiki j2ov411gcdw3vqcr6qy09cfouzlu3di 2427858 2427857 2026-06-17T07:32:30Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1044.eqiad.wmnet with reason: host reimage 2427858 wikitext text/x-wiki ka67frch3uvoudnoghvk43uqq65196t 2427860 2427858 2026-06-17T07:39:42Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on conf2007.codfw.wmnet with reason: host reimage 2427860 wikitext text/x-wiki jp61kpjtnt8ks7xm00ur0f3v1u3e485 2427861 2427860 2026-06-17T07:40:09Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1044.eqiad.wmnet with reason: host reimage 2427861 wikitext text/x-wiki gm1pck3uijxwizo9942i0hg0h7zapo3 2427862 2427861 2026-06-17T07:41:34Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host conf2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427862 wikitext text/x-wiki 0uq5r1or9oyk8m5hevyt8vmqb71yt2a 2427863 2427862 2026-06-17T07:42:14Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host conf2008.codfw.wmnet with OS trixie 2427863 wikitext text/x-wiki 115p520e283vc6iyt644abxhsdd0xpj 2427864 2427863 2026-06-17T07:43:26Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2427864 wikitext text/x-wiki flebs6zk6mxoik3w52ovnb8oyzswz94 2427865 2427864 2026-06-17T07:43:34Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply 2427865 wikitext text/x-wiki qr1ofaopkg5we4f5zdwjwseofvl1klo 2427866 2427865 2026-06-17T07:44:38Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on conf2007.codfw.wmnet with reason: host reimage 2427866 wikitext text/x-wiki 19vnjx725c8fz03ehzau4xgghl7g0kk 2427867 2427866 2026-06-17T07:49:52Z Stashbot 7414 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1303296|Squashed diff to master]], [[gerrit:1303295|Squashed diff to master]] 2427867 wikitext text/x-wiki c9nag6tu2a4k5jdofxz7izmc47yb2lf 2427868 2427867 2026-06-17T07:50:14Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host conf2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427868 wikitext text/x-wiki q08je7xy9jax3xphtj640wdr76oqvev 2427869 2427868 2026-06-17T07:53:56Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host conf2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427869 wikitext text/x-wiki f9tcm9k2ff3sovdj1otmvxojvl4wx7m 2427872 2427869 2026-06-17T07:57:53Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1044.eqiad.wmnet with OS trixie 2427872 wikitext text/x-wiki 9load7dku54iriicr9zix5o04tiv7bh 2427873 2427872 2026-06-17T07:58:06Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host conf2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427873 wikitext text/x-wiki rfjqwg4hzuauxpo4a76l3v7n1wqxkr8 2427874 2427873 2026-06-17T07:59:58Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2427874 wikitext text/x-wiki 71af4ixyhygzq4foylijqoxc2gyehi7 2427875 2427874 2026-06-17T08:00:07Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on conf2008.codfw.wmnet with reason: host reimage 2427875 wikitext text/x-wiki aqszj4ls5ijg2bwmkf3nl8v3q2gp8qa 2427876 2427875 2026-06-17T08:00:13Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1044: repool after upgrade 2427876 wikitext text/x-wiki 8d5wfg83fhyl03lvo0esair8tgobna8 2427877 2427876 2026-06-17T08:01:32Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003" 2427877 wikitext text/x-wiki psraeat3ocuhpkgl6f2c75l5a662vyt 2427878 2427877 2026-06-17T08:01:34Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-wdqs1001.eqiad.wmnet with OS bookworm 2427878 wikitext text/x-wiki m1ce7v5n9tvg7t2feual3gufpf29y2m 2427879 2427878 2026-06-17T08:03:07Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2427879 wikitext text/x-wiki 21xzm8k9x9i8fnrgoakcicvylqko49u 2427880 2427879 2026-06-17T08:04:11Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2427880 wikitext text/x-wiki r06eibulxg5gmginao03ueikruwbhia 2427881 2427880 2026-06-17T08:04:13Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host conf2007.codfw.wmnet with OS trixie 2427881 wikitext text/x-wiki 5b0kvlp1x9nemtyg4v82b3g32i8a09a 2427882 2427881 2026-06-17T08:04:35Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on conf2008.codfw.wmnet with reason: host reimage 2427882 wikitext text/x-wiki fcx5q7h87vr6c6jgzxk66sz00ztlrmj 2427883 2427882 2026-06-17T08:04:52Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host conf2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427883 wikitext text/x-wiki q4xtvlu4erzl5c77uwk7i896pm9e9oe 2427884 2427883 2026-06-17T08:06:08Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host conf2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427884 wikitext text/x-wiki jdzo2jjw4hm4bsvcydrrp7nmcme7zw2 2427885 2427884 2026-06-17T08:07:30Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs2001.codfw.wmnet with OS bookworm 2427885 wikitext text/x-wiki fsy8t5ahqq2np90ofcp9y4mndgd430r 2427886 2427885 2026-06-17T08:09:28Z Stashbot 7414 mlitn@deploy1003: mlitn: Backport for [[gerrit:1303296|Squashed diff to master]], [[gerrit:1303295|Squashed diff to master]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2427886 wikitext text/x-wiki btfuq10495r5une1e7vhili6uwx7v8f 2427887 2427886 2026-06-17T08:12:03Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host conf2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2427887 wikitext text/x-wiki nl7jaalacghbwsqsp06goyx8wdxun0c 2427888 2427887 2026-06-17T08:12:22Z Stashbot 7414 mlitn@deploy1003: mlitn: Continuing with deployment 2427888 wikitext text/x-wiki 811zgwapeq1nhoe6pbh40dc5s8hn3yv 2427889 2427888 2026-06-17T08:14:01Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host conf2009.codfw.wmnet with OS trixie 2427889 wikitext text/x-wiki leeso2ro0ip5rlc0xmkools7i8dkmpz 2427890 2427889 2026-06-17T08:17:49Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs2001.codfw.wmnet with OS bookworm 2427890 wikitext text/x-wiki j1qzg32dyvlg14l5rtwc7c54e1sovra 2427891 2427890 2026-06-17T08:22:57Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2427891 wikitext text/x-wiki nclyk2brmbjfk3ob8nfnfi0p07o6jxh 2427892 2427891 2026-06-17T08:23:14Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2427892 wikitext text/x-wiki t2pd5rxgv4w22wb9a35fawndzxswwvn 2427893 2427892 2026-06-17T08:23:15Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host conf2008.codfw.wmnet with OS trixie 2427893 wikitext text/x-wiki 32tcm6vbshd0jgottwd5qw91sdzabhe 2427894 2427893 2026-06-17T08:25:26Z Stashbot 7414 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1303296|Squashed diff to master]], [[gerrit:1303295|Squashed diff to master]] (duration: 35m 34s) 2427894 wikitext text/x-wiki hfe1o2ogxhx4j2b248x4gao16d50q3a 2427895 2427894 2026-06-17T08:31:12Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on conf2009.codfw.wmnet with reason: host reimage 2427895 wikitext text/x-wiki fz64ba3uvfgv9u1wkvo3svhz98bvyq3 2427897 2427895 2026-06-17T08:35:59Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on conf2009.codfw.wmnet with reason: host reimage 2427897 wikitext text/x-wiki ecilybloivdzicodhpl18av5u937l3d 2427898 2427897 2026-06-17T08:36:38Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2427898 wikitext text/x-wiki q7ecml50t9a713zrtlol7v0hmenhcac 2427899 2427898 2026-06-17T08:38:09Z Stashbot 7414 moritzm: installing apache2 security updates 2427899 wikitext text/x-wiki j94n1eyi4kwi0ulrsfdo6ibgz35weev 2427900 2427899 2026-06-17T08:38:39Z Stashbot 7414 jelto: "Imported helm3 3.19.5-1 to bullseye-wikimedia, bookworm-wikimedia and trixie-wikimedia - T427403" 2427900 wikitext text/x-wiki hr4kosh0w9wk8z4ky5wss0yp0u6ax2q 2427901 2427900 2026-06-17T08:45:37Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1044: repool after upgrade 2427901 wikitext text/x-wiki svegcwribzz9c92cua2vrpzlikw43iw 2427902 2427901 2026-06-17T08:46:23Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 30 hosts with reason: Primary switchover s1 T429190 2427902 wikitext text/x-wiki g5g3hax4kryghojsjy8pv8qvhda56w9 2427903 2427902 2026-06-17T08:46:25Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs2001.codfw.wmnet with OS bookworm 2427903 wikitext text/x-wiki aujge0ks177c5u5ejpivmjbea7gzm7f 2427904 2427903 2026-06-17T08:46:47Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2212 with weight 0 T429190', diff saved to https://phabricator.wikimedia.org/P94215 and previous config saved to /var/cache/conftool/dbconfig/20260617-084642-cwilliams.json 2427904 wikitext text/x-wiki it58rttxa680sajrjnj7b0gi8r9i6ls 2427905 2427904 2026-06-17T08:48:47Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2427905 wikitext text/x-wiki 6zhrj4qt4yoxem7f9hpi29uc6swizfx 2427906 2427905 2026-06-17T08:49:38Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2427906 wikitext text/x-wiki 3m2n4s8sjvghh42kzmed4rq8p6hzu9i 2427907 2427906 2026-06-17T08:51:22Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2427907 wikitext text/x-wiki cclsyaea1fdulteemb5rtxcz11es6t7 2427908 2427907 2026-06-17T08:51:56Z Stashbot 7414 cezmunsta: Starting s1 codfw failover from db2203 to db2212 - T429190 2427908 wikitext text/x-wiki no9sbpee16e0fv5dq4ak7l3c88n4t5k 2427909 2427908 2026-06-17T08:53:14Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2212 to s1 primary T429190', diff saved to https://phabricator.wikimedia.org/P94217 and previous config saved to /var/cache/conftool/dbconfig/20260617-085310-cwilliams.json 2427909 wikitext text/x-wiki 2lx271jzk9joqpoeuwv1tnzxnltgpy8 2427911 2427909 2026-06-17T08:55:04Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2427911 wikitext text/x-wiki 1ypf5pw2olj49cvkzjl3itzbxkctar9 2427912 2427911 2026-06-17T08:55:29Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" 2427912 wikitext text/x-wiki 611gkydwagloctkc1vl694ibp15w8et 2427913 2427912 2026-06-17T08:55:31Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host conf2009.codfw.wmnet with OS trixie 2427913 wikitext text/x-wiki kouhv6cr4lnhzl4it07o8stwr8xs08d 2427914 2427913 2026-06-17T08:56:21Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2203 T429190', diff saved to https://phabricator.wikimedia.org/P94218 and previous config saved to /var/cache/conftool/dbconfig/20260617-085615-cwilliams.json 2427914 wikitext text/x-wiki 6ejdrcg15q69p1ujanxg2yecchx7641 2427915 2427914 2026-06-17T08:57:03Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs2001.codfw.wmnet with OS bookworm 2427915 wikitext text/x-wiki mp2v28ae9onciu0m81c947f8fzfvy42 2427917 2427915 2026-06-17T08:59:45Z Stashbot 7414 joal@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply 2427917 wikitext text/x-wiki ecjfzbhoxhqsr7fx5osts56sje05jah 2427918 2427917 2026-06-17T09:00:03Z Stashbot 7414 joal@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply 2427918 wikitext text/x-wiki 34q5wowy19x9mmvkaai6xvma50bqczq 2427919 2427918 2026-06-17T09:01:51Z Stashbot 7414 joal@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply 2427919 wikitext text/x-wiki lh95s5bkrgqzncw1lzoqj8509zc2q77 2427920 2427919 2026-06-17T09:02:04Z Stashbot 7414 joal@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply 2427920 wikitext text/x-wiki mbglmuthhpytdw1d2qc24qvwjpank61 2427921 2427920 2026-06-17T09:02:24Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es2046 to es5 codfw primary T428572', diff saved to https://phabricator.wikimedia.org/P94219 and previous config saved to /var/cache/conftool/dbconfig/20260617-090221-marostegui.json 2427921 wikitext text/x-wiki dvwdnygjfc7tszyal016as7t57x0381 2427923 2427921 2026-06-17T09:04:59Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2427923 wikitext text/x-wiki ahxvqjeun47d2yqhc74cslv6h2a2yfg 2427924 2427923 2026-06-17T09:05:20Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2044: Upgrading es2044.codfw.wmnet 2427924 wikitext text/x-wiki qipv0zrg99vm575w08uxmi5tr2ejigf 2427925 2427924 2026-06-17T09:06:00Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool es2044: Upgrading es2044.codfw.wmnet 2427925 wikitext text/x-wiki 99i1nynzx2u1h6h2oovhg1w0z4g7zh7 2427926 2427925 2026-06-17T09:06:09Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2427926 wikitext text/x-wiki 1eckoerz1v5j6gnyccpsp3s1pjm33nj 2427927 2427926 2026-06-17T09:06:25Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2045: repool after maintenance es2045 2427927 wikitext text/x-wiki 9pjulql2slzr288m8wl8z5d7zwmnrap 2427928 2427927 2026-06-17T09:07:47Z Stashbot 7414 elukey: add basic Kafka ACLs for anonymous to logging-codfw - T425528 (I'll add rollback steps in the task if needed) 2427928 wikitext text/x-wiki 6k9rdgj3g3ixxtyikriau2jl4h5e152 2427929 2427928 2026-06-17T09:09:27Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2427929 wikitext text/x-wiki 9d7545sdvl6gwq21hmyk5a9dhofk251 2427930 2427929 2026-06-17T09:09:38Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2203: Upgrading db2203.codfw.wmnet 2427930 wikitext text/x-wiki 65cuo2c4p5wzgmlcntv26e7w0newby9 2427931 2427930 2026-06-17T09:09:49Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2203: Upgrading db2203.codfw.wmnet 2427931 wikitext text/x-wiki ir4lsq0k74dvbb88f0utqzlsslan1pj 2427932 2427931 2026-06-17T09:11:33Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2203.codfw.wmnet with OS trixie 2427932 wikitext text/x-wiki d8abxr7u3dac8qxwibsey18ao91v8i3 2427933 2427932 2026-06-17T09:26:33Z Stashbot 7414 jynus: testing x1 backups @ cumin2003 T427897 2427933 wikitext text/x-wiki j5chv54qpesggqx38tm1qp5hq4k7pzg 2427934 2427933 2026-06-17T09:27:44Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2203.codfw.wmnet with reason: host reimage 2427934 wikitext text/x-wiki 09jg3kzgeo03lo0l6djlxy2pkugwlj0 2427935 2427934 2026-06-17T09:29:17Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Set es6 eqiad as read-only for maintenance - T429436', diff saved to https://phabricator.wikimedia.org/P94222 and previous config saved to /var/cache/conftool/dbconfig/20260617-092913-marostegui.json 2427935 wikitext text/x-wiki h0vnjkwcvefq7i97uf9odaep93psg0s 2427936 2427935 2026-06-17T09:29:30Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1002.eqiad.wmnet with OS bookworm 2427936 wikitext text/x-wiki laxcngboh4r3rmkky7ychfwkwamzy6h 2427937 2427936 2026-06-17T09:29:33Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 8 hosts with reason: Primary switchover es6 T429436 2427937 wikitext text/x-wiki nmx4ghit2cyq4m4ieu6setyxz3gv6pb 2427938 2427937 2026-06-17T09:29:43Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Set es1037 with weight 0 T429436', diff saved to https://phabricator.wikimedia.org/P94223 and previous config saved to /var/cache/conftool/dbconfig/20260617-092940-marostegui.json 2427938 wikitext text/x-wiki hx00d99e9lnbh4qutq3o677z3cg65nz 2427940 2427938 2026-06-17T09:32:08Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1303356|hCaptcha: Remove config for VE and DT enable (T428883)]], [[gerrit:1303354|Drop $wgDiscussionToolsHCaptchaRequiredForAllEdits (T428883)]] 2427940 wikitext text/x-wiki 94r55oh8momtzxdh74e92x6lav9nuu6 2427941 2427940 2026-06-17T09:32:20Z Stashbot 7414 marostegui: Starting es6 eqiad failover from es1038 to es1037 - T429436 2427941 wikitext text/x-wiki netepy5ju4stj9k4cjvio36q1ht17se 2427942 2427941 2026-06-17T09:33:13Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Promote es1037 to es6 primary T429436', diff saved to https://phabricator.wikimedia.org/P94224 and previous config saved to /var/cache/conftool/dbconfig/20260617-093310-marostegui.json 2427942 wikitext text/x-wiki on6vcfpndgbfo3w9nxm05oygd533pfw 2427943 2427942 2026-06-17T09:34:32Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2203.codfw.wmnet with reason: host reimage 2427943 wikitext text/x-wiki b10q5tixoen69op1r4jqk1j68n8e5du 2427944 2427943 2026-06-17T09:35:17Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Depool es1038 T429436', diff saved to https://phabricator.wikimedia.org/P94225 and previous config saved to /var/cache/conftool/dbconfig/20260617-093513-marostegui.json 2427944 wikitext text/x-wiki j1rt6yag6qcdyi09vqkghuw2epr5dg0 2427945 2427944 2026-06-17T09:35:35Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2427945 wikitext text/x-wiki rklyg65c5vud4p7ucjddqarvdwr5kim 2427946 2427945 2026-06-17T09:36:02Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Set es6 eqiad back to read-write - T429436', diff saved to https://phabricator.wikimedia.org/P94226 and previous config saved to /var/cache/conftool/dbconfig/20260617-093559-marostegui.json 2427946 wikitext text/x-wiki suc2v987iummph8b0hnuiypdpgdu8vc 2427947 2427946 2026-06-17T09:37:20Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2427947 wikitext text/x-wiki msfdmp4ttmv9x35n00wn1z8134o6fd9 2427948 2427947 2026-06-17T09:37:44Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2427948 wikitext text/x-wiki hf5m5tan7uxbh5g94kvfhyb0bdxe4dc 2427949 2427948 2026-06-17T09:37:53Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1038: Upgrading es1038.eqiad.wmnet 2427949 wikitext text/x-wiki cqck7w3d53np8i2rqkpnjgkw4e9kfwn 2427950 2427949 2026-06-17T09:38:04Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1303356|hCaptcha: Remove config for VE and DT enable (T428883)]], [[gerrit:1303354|Drop $wgDiscussionToolsHCaptchaRequiredForAllEdits (T428883)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2427950 wikitext text/x-wiki lf4rnxt6mxawsvxxwmg6cqqampziqgj 2427951 2427950 2026-06-17T09:38:06Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1038: Upgrading es1038.eqiad.wmnet 2427951 wikitext text/x-wiki 4xmjpgt49lvwwv9e4pn9rmgc0irjp5s 2427952 2427951 2026-06-17T09:38:54Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1038.eqiad.wmnet with OS trixie 2427952 wikitext text/x-wiki kfhucvt9rt2jlcly5u7b2dwu6u8pkus 2427953 2427952 2026-06-17T09:39:33Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs1002.eqiad.wmnet with OS bookworm 2427953 wikitext text/x-wiki gjqkqzk0irdkmicfkw42b4y0u4b14fu 2427954 2427953 2026-06-17T09:41:05Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment 2427954 wikitext text/x-wiki khx11udy2qeuyrglzlf32wrbpr9s3t6 2427958 2427954 2026-06-17T09:47:40Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1303356|hCaptcha: Remove config for VE and DT enable (T428883)]], [[gerrit:1303354|Drop $wgDiscussionToolsHCaptchaRequiredForAllEdits (T428883)]] (duration: 15m 32s) 2427958 wikitext text/x-wiki 4bh9dxbjzg5pmj0lbis8m8n81qt8dl1 2427960 2427958 2026-06-17T09:48:25Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1002.eqiad.wmnet with OS bookworm 2427960 wikitext text/x-wiki 5mxqcc3sf92unvxudjckgld8nqqgv85 2427962 2427960 2026-06-17T09:51:51Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2045: repool after maintenance es2045 2427962 wikitext text/x-wiki qfu76k1q05n1ce6jbqu1fx20s10krw9 2427963 2427962 2026-06-17T09:52:11Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2203.codfw.wmnet with OS trixie 2427963 wikitext text/x-wiki qbqoe7omglkk50maxa3ef8g0iitwtub 2427964 2427963 2026-06-17T09:54:39Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es1038.eqiad.wmnet with reason: host reimage 2427964 wikitext text/x-wiki 6s5l6xbandhjp2h756o79pdqt15dily 2427965 2427964 2026-06-17T09:58:23Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1038.eqiad.wmnet with reason: host reimage 2427965 wikitext text/x-wiki ojl1v9vsckr6vmey5btv2hfll3a3l05 2427966 2427965 2026-06-17T10:00:12Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-wdqs1002.eqiad.wmnet with reason: host reimage 2427966 wikitext text/x-wiki 044nwxz7kr7ubv6k068574s0fgjmufe 2427967 2427966 2026-06-17T10:02:38Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2203: Migration of db2203.codfw.wmnet completed 2427967 wikitext text/x-wiki km3xum39eev79siyitckgxn613iujmr 2427968 2427967 2026-06-17T10:04:45Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-wdqs1002.eqiad.wmnet with reason: host reimage 2427968 wikitext text/x-wiki 0sqfh1gaa8149o7p2rxc3v6sj9it6rv 2427969 2427968 2026-06-17T10:09:57Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2427969 wikitext text/x-wiki iffp3x9ox5x6es6d70riruq1xqbwgld 2427970 2427969 2026-06-17T10:10:38Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2427970 wikitext text/x-wiki qdg7x570y21uukt4q8h39vkgd9yijmp 2427971 2427970 2026-06-17T10:10:49Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs2001.codfw.wmnet with OS bookworm 2427971 wikitext text/x-wiki 7l4sbti6hghvuobx4d0b08br68h0yqe 2427972 2427971 2026-06-17T10:12:37Z Stashbot 7414 claime: cumin -x 'A:swift-fe' "disable-puppet 'Disabling puppet for ratelimit deploy - cgoubert'" 2427972 wikitext text/x-wiki aazcxxlrpn4er6vb2voxd91h0otxfny 2427973 2427972 2026-06-17T10:15:46Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1038.eqiad.wmnet with OS trixie 2427973 wikitext text/x-wiki lgnnr5mk46788kfcua4cdsk7nzn779b 2427974 2427973 2026-06-17T10:17:56Z Stashbot 7414 claime: cumin -x 'A:swift-fe' "enable-puppet 'Disabling puppet for ratelimit deploy - cgoubert'" 2427974 wikitext text/x-wiki qv542vbuwlg5xzh48wvr3x4at9e0mgu 2427975 2427974 2026-06-17T10:21:28Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs2001.codfw.wmnet with OS bookworm 2427975 wikitext text/x-wiki 22jm8f8yg752pz0wpddji2jfbfdaf5h 2427976 2427975 2026-06-17T10:22:24Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003" 2427976 wikitext text/x-wiki f1ud7qdxmjl2cfqn3hrf542f7k3ub1o 2427977 2427976 2026-06-17T10:22:41Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es1038: Migration of es1038.eqiad.wmnet completed 2427977 wikitext text/x-wiki s528amxx7rqu1c5dwx1vevj6id5jc1i 2427978 2427977 2026-06-17T10:28:33Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003" 2427978 wikitext text/x-wiki o8jgovz8ne4su25ms8z6k43mfisu3vz 2427979 2427978 2026-06-17T10:28:35Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-wdqs1002.eqiad.wmnet with OS bookworm 2427979 wikitext text/x-wiki cdicy20bdt6zeh19i3n18un4xvndgbe 2427980 2427979 2026-06-17T10:28:50Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs2002.codfw.wmnet with OS bookworm 2427980 wikitext text/x-wiki ake1n7tf5qowgmkcozhisbz102qsm7p 2427981 2427980 2026-06-17T10:29:50Z Stashbot 7414 moritzm: installing git-lfs security updates 2427981 wikitext text/x-wiki b2jqtf3l1fl8igoekelapd968nm5t6r 2427982 2427981 2026-06-17T10:31:52Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1003.eqiad.wmnet with OS bookworm 2427982 wikitext text/x-wiki oets1tvhui13zu7gi1gsrikui3gn3vk 2427983 2427982 2026-06-17T10:34:26Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/ratelimit: apply 2427983 wikitext text/x-wiki im5hrn2uakkcnf7vqvipro9dwpvxn8d 2427984 2427983 2026-06-17T10:34:34Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/ratelimit: apply 2427984 wikitext text/x-wiki knbrg3z7r0c4lzz1vsjlhpkd21trrjc 2427985 2427984 2026-06-17T10:34:52Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply 2427985 wikitext text/x-wiki e399zvvmebxzqs29fip7e81is6ckdxn 2427986 2427985 2026-06-17T10:35:13Z Stashbot 7414 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply 2427986 wikitext text/x-wiki aw8ykmfy4dkcvfcfdi1y0i9dbl9kecb 2427987 2427986 2026-06-17T10:35:19Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply 2427987 wikitext text/x-wiki sgkgqrt7za158jl3jaip3p4ju203igd 2427988 2427987 2026-06-17T10:35:31Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2427988 wikitext text/x-wiki ax2qc83ggspn4i5nngaouahcw365pgi 2427989 2427988 2026-06-17T10:35:39Z Stashbot 7414 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply 2427989 wikitext text/x-wiki o0vfmj8bvxo9xx8ett51bkgr8amak1s 2427990 2427989 2026-06-17T10:35:41Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es2044: Upgrading es2044.codfw.wmnet 2427990 wikitext text/x-wiki e2adz8z7lka4qksdda78m2oa696bbt0 2427991 2427990 2026-06-17T10:36:03Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2044: Upgrading es2044.codfw.wmnet 2427991 wikitext text/x-wiki or4pjg3dd877bzfsadjkalfgkbticwv 2427992 2427991 2026-06-17T10:37:07Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es2044.codfw.wmnet with OS trixie 2427992 wikitext text/x-wiki mht0lgs4thc2clha6yos6njntg4soms 2427993 2427992 2026-06-17T10:38:39Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs2002.codfw.wmnet with OS bookworm 2427993 wikitext text/x-wiki a9900usgi2brynugm9gm80xs2urgqv1 2427994 2427993 2026-06-17T10:43:39Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-wdqs1003.eqiad.wmnet with reason: host reimage 2427994 wikitext text/x-wiki bjwb6sozdp95sdz4r5a5w5uec0y7eah 2427995 2427994 2026-06-17T10:48:09Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2203: Migration of db2203.codfw.wmnet completed 2427995 wikitext text/x-wiki mr9c9jroh8awt6vpig2wyzjybvatgli 2427996 2427995 2026-06-17T10:48:11Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2427996 wikitext text/x-wiki 6kk03680w8pytx9018rkcok6mpjqjhi 2427997 2427996 2026-06-17T10:50:36Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-wdqs1003.eqiad.wmnet with reason: host reimage 2427997 wikitext text/x-wiki ihislppkryhrp4xirwg7cxhssnjz7k9 2427998 2427997 2026-06-17T10:53:24Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on es2044.codfw.wmnet with reason: host reimage 2427998 wikitext text/x-wiki i5lvhufykkn0xvuyz6r0ner9qzhpzw1 2427999 2427998 2026-06-17T10:59:07Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2044.codfw.wmnet with reason: host reimage 2427999 wikitext text/x-wiki e5fwh9xfxizkav3u1azoksrkmlk5sjf 2428000 2427999 2026-06-17T10:59:30Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade 2428000 wikitext text/x-wiki bts3l7q600jfromi1az53wcxgrbq1gt 2428001 2428000 2026-06-17T10:59:50Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1163: Upgrading db1163.eqiad.wmnet 2428001 wikitext text/x-wiki ieigp6zm1mytqrim6iurwng71ty849q 2428002 2428001 2026-06-17T10:59:57Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.dhcp for host dse-k8s-wdqs2001.codfw.wmnet 2428002 wikitext text/x-wiki cu14u5ldf6epoh19jzu6qpn3s4oei7r 2428003 2428002 2026-06-17T11:00:21Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1163: Upgrading db1163.eqiad.wmnet 2428003 wikitext text/x-wiki 16d2qn18ai5ymk7xmyj4kk8jmxjenr4 2428004 2428003 2026-06-17T11:01:33Z Stashbot 7414 moritzm: The Debian mirror on mirrors.wikimedia.org has been disabled T416707 2428004 wikitext text/x-wiki 3raojrb85syquwulg0ueatpzdigiqe9 2428006 2428004 2026-06-17T11:02:40Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply 2428006 wikitext text/x-wiki gwz1jxsltty6th1aclzb9cc9opjfi3s 2428007 2428006 2026-06-17T11:02:56Z Stashbot 7414 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply 2428007 wikitext text/x-wiki ocso9ghs3e3j8fy2kfyectpr96gsq3z 2428008 2428007 2026-06-17T11:04:43Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1163.eqiad.wmnet with OS trixie 2428008 wikitext text/x-wiki juy3byy9zdjjuf0r1lx7y6g9i3dn276 2428009 2428008 2026-06-17T11:08:12Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1038: Migration of es1038.eqiad.wmnet completed 2428009 wikitext text/x-wiki 2phtlcpw9llke4z72dp990w4iewxee4 2428010 2428009 2026-06-17T11:08:15Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2428010 wikitext text/x-wiki escg8s3cz5e68zajtl1vvryke704wxs 2428011 2428010 2026-06-17T11:08:31Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003" 2428011 wikitext text/x-wiki 4g1usflwoockbfe1xrtu7mwnfmnpgcx 2428012 2428011 2026-06-17T11:09:22Z Stashbot 7414 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply 2428012 wikitext text/x-wiki rn054g97yb4t5dpbhjcnuff0vqke10x 2428013 2428012 2026-06-17T11:09:50Z Stashbot 7414 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply 2428013 wikitext text/x-wiki 572covwlrl6w89vu6cleb9aiyqthns3 2428014 2428013 2026-06-17T11:10:10Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2428014 wikitext text/x-wiki o87js4o6k75hsim6z48mtrvnq23ma9z 2428015 2428014 2026-06-17T11:11:20Z Stashbot 7414 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply 2428015 wikitext text/x-wiki bbv2jnu73vm9ppmk0zgr4pj129n5d2e 2428016 2428015 2026-06-17T11:11:47Z Stashbot 7414 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply 2428016 wikitext text/x-wiki p7r0ozgpcgtkoe24wyhemn99enm2q7o 2428017 2428016 2026-06-17T11:12:01Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003" 2428017 wikitext text/x-wiki pzx7yilnmdhurhur5h9t9n38m6rmrfg 2428018 2428017 2026-06-17T11:12:02Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-wdqs1003.eqiad.wmnet with OS bookworm 2428018 wikitext text/x-wiki 63v30yuwy4yr5529k11ij81nd5s1wao 2428019 2428018 2026-06-17T11:16:40Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2044.codfw.wmnet with OS trixie 2428019 wikitext text/x-wiki tjkar6xdacgmm81dg055whod11ivsiz 2428020 2428019 2026-06-17T11:17:29Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) 2428020 wikitext text/x-wiki hfhhz1b835mkg8oep6gwskb8e74mqy1 2428021 2428020 2026-06-17T11:17:55Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool es2044: repool after maintenance es2044 2428021 wikitext text/x-wiki 7242cxix84mfynbpkvu2fq73m4c86kq 2428022 2428021 2026-06-17T11:18:05Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2428022 wikitext text/x-wiki 4899k8n92lt9c6teh0u3hg4vrdvbx1j 2428023 2428022 2026-06-17T11:18:27Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1163.eqiad.wmnet with reason: host reimage 2428023 wikitext text/x-wiki 1hifgdrnijs789rwm3rr6pyg8y41gyo 2428024 2428023 2026-06-17T11:19:47Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1190.eqiad.wmnet with reason: upgrading 2428024 wikitext text/x-wiki c3ustj0myw70bq03n2k1w82ve232k5a 2428025 2428024 2026-06-17T11:21:15Z Stashbot 7414 marostegui@cumin1003: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on db1171.eqiad.wmnet with reason: upgrading 2428025 wikitext text/x-wiki qr4c8gx68bazhv8shias7pr5lum3xjf 2428026 2428025 2026-06-17T11:22:11Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-wdqs2001.codfw.wmnet 2428026 wikitext text/x-wiki re3ldj8ayyyu1kdt29px90ky0ncb048 2428027 2428026 2026-06-17T11:22:12Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1172.eqiad.wmnet with reason: upgrading 2428027 wikitext text/x-wiki o5xrrtwb8e8n2muc00e9igm6mn3s12p 2428028 2428027 2026-06-17T11:23:33Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1163.eqiad.wmnet with reason: host reimage 2428028 wikitext text/x-wiki 4gpqwx4s0f4ylesap1dmbx69rbf0w8t 2428029 2428028 2026-06-17T11:23:42Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs2002.codfw.wmnet with OS bookworm 2428029 wikitext text/x-wiki 4t21ih0ka85hadbn1jxhzzmklqjeyp1 2428031 2428029 2026-06-17T11:26:38Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1191.eqiad.wmnet with reason: upgrading 2428031 wikitext text/x-wiki owmzozvxkqv9y8qo1h8xw9vpf5e6fnf 2428032 2428031 2026-06-17T11:27:43Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2428032 wikitext text/x-wiki 1fibyvee7v2le3wh4a3r50ohj8empp8 2428033 2428032 2026-06-17T11:35:56Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-wdqs2002.codfw.wmnet with reason: host reimage 2428033 wikitext text/x-wiki ankfym7vvdmr09sxvy5xht3izwxqpsm 2428034 2428033 2026-06-17T11:40:16Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-wdqs2002.codfw.wmnet with reason: host reimage 2428034 wikitext text/x-wiki g9hlqx47tan8ixk7ccvs5rojnr072sl 2428035 2428034 2026-06-17T11:40:22Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1163.eqiad.wmnet with OS trixie 2428035 wikitext text/x-wiki c57qv8599vsx4p2h2zew7p7uwenk5cl 2428036 2428035 2026-06-17T11:42:37Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.major-upgrade 2428036 wikitext text/x-wiki 77ccq7p3ljodpkzs6h4bsg2q0i98gr0 2428037 2428036 2026-06-17T11:42:57Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool es1045: Upgrading es1045.eqiad.wmnet 2428037 wikitext text/x-wiki 532u1f8lu2ehmaq44m6mjxfj57iuhqe 2428038 2428037 2026-06-17T11:43:47Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1045: Upgrading es1045.eqiad.wmnet 2428038 wikitext text/x-wiki julj2ofsue647w7xl6hn3k48sebj2qt 2428039 2428038 2026-06-17T11:44:41Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host es1045.eqiad.wmnet with OS trixie 2428039 wikitext text/x-wiki i4q5txi3wji00mxmdt734ayo17177e0 2428040 2428039 2026-06-17T11:51:10Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1163: Migration of db1163.eqiad.wmnet completed 2428040 wikitext text/x-wiki 577m3ifmy21sjtzqjdq679ctrxa5odg 2428041 2428040 2026-06-17T11:51:21Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs2002.codfw.wmnet with OS bookworm 2428041 wikitext text/x-wiki ncroyagac25myvu4dbgcj67kc2qvkiv 2428042 2428041 2026-06-17T11:54:39Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2428042 wikitext text/x-wiki dhvo5fzocmr19ud7f37wbrnbfg8u6bi 2428043 2428042 2026-06-17T11:55:06Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2428043 wikitext text/x-wiki cd5m8954zer73pxrbr2r0lmx1u5rzj7 2428044 2428043 2026-06-17T11:55:09Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2428044 wikitext text/x-wiki cvfvgf9a7eucbrun7c50p8rrrzr7gj1 2428045 2428044 2026-06-17T11:55:36Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2428045 wikitext text/x-wiki 5u1e3gr43ouomi0lf8417d27wjebccr 0 17290 2427580 2427192 2026-06-16T15:31:14Z Stashbot 7414 dancy: Turning off deployment-db11 and deployment-db14 2427580 wikitext text/x-wiki 3n52mry83pnq3u58t3bxz1lzg5o4z85 2427591 2427580 2026-06-16T15:48:22Z Stashbot 7414 dancy: Resizing deployment-deploy04.deployment-prep from g4.cores4.ram8.disk20 to g4.cores8.ram16.disk20 (T429364) 2427591 wikitext text/x-wiki euognkz12fd6aa8u54ywpaopo3pqp4v 2427597 2427591 2026-06-16T16:00:46Z Stashbot 7414 dancy: sudo keyholder arm on deployment-deploy04.deployment-prep 2427597 wikitext text/x-wiki ksqa9a912ychgc1efixoa8nd9g3z44c 0 22503 2427823 2427196 2026-06-17T03:50:12Z Jon (WMF) 42946 /* Currently reserved values */ 2427823 wikitext text/x-wiki 17hznll7hk86khwqbhrfwmi3rysswy8 0 23732 2427840 2335867 2026-06-17T07:07:38Z MArostegui (WMF) 8620 /* Copy data from a backup source to a host */ 2427840 wikitext text/x-wiki gfg0sjfhzi5frrudht90nml0mf3ls6k 2427841 2427840 2026-06-17T07:10:02Z Jynus 38174 /* Backups quick cheatsheet */ 2427841 wikitext text/x-wiki kyyem1kw70p1fhf0gcbtqaj1tqxxrzq 498 30942 2427503 2427180 2026-06-16T13:18:45Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudservices.safe_reboot on hosts matched by 'P{O:wmcs::openstack::codfw1dev::services}' 2427503 wikitext text/x-wiki 80alyt3r4ndik6meadm1yclc5ge8cz9 2427511 2427503 2026-06-16T13:24:21Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudservices.safe_reboot (exit_code=0) on hosts matched by 'P{O:wmcs::openstack::codfw1dev::services}' 2427511 wikitext text/x-wiki mzqno953f953uzx131kroxk0plleu5g 2427567 2427511 2026-06-16T14:55:56Z Stashbot 7414 andrewbogott: "systemctl restart ceph-osd@289.service" as 289 shows as down 2427567 wikitext text/x-wiki pkxizat77vv67q8cnn58v2wsm766ybw 2427569 2427567 2026-06-16T14:57:58Z Stashbot 7414 andrewbogott: "systemctl restart ceph-osd@281.service" as 281 shows as down 2427569 wikitext text/x-wiki 4ajvv0fn3dc2hejt9lhme27h07iurie 2427575 2427569 2026-06-16T15:13:51Z Stashbot 7414 andrewbogott: systemctl restart ceph-osd@58.service due to slow ops 2427575 wikitext text/x-wiki n5d2dvxlguj5v01m9kv4mtwjfroa9cp 2427579 2427575 2026-06-16T15:31:01Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet' (T429361) 2427579 wikitext text/x-wiki 3fw2jcwtej628zt17zz59j33fonx6ur 2427582 2427579 2026-06-16T15:33:04Z Stashbot 7414 andrewbogott: restarting quite a few other ceph-osd services in an attempt to quiet some slow op alerts 2427582 wikitext text/x-wiki cds067bous07yiws4hbv5jsv7u7ebtp 2427588 2427582 2026-06-16T15:40:57Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet' (T429361) 2427588 wikitext text/x-wiki 8p1izr3fc0l8fienkm73tfqbrn1eckx 2427589 2427588 2026-06-16T15:41:51Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet' (T429361) 2427589 wikitext text/x-wiki 8bvmzkaqe9iqr3lsfzvzce27jq1b18k 2427593 2427589 2026-06-16T15:52:23Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet' (T429361) 2427593 wikitext text/x-wiki m9k7wpqlqal5pltiw7wa82spw10pqqg 2427594 2427593 2026-06-16T15:52:34Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2011-dev.codfw.wmnet' (T429361) 2427594 wikitext text/x-wiki 1n2kvoudnht3u93ggt1c0vi3c1sgzss 2427595 2427594 2026-06-16T15:52:38Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2011-dev.codfw.wmnet' (T429361) 2427595 wikitext text/x-wiki byfg6bawuijnomdcaknor9zqz2nrf1d 2427596 2427595 2026-06-16T15:52:42Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2010-dev.codfw.wmnet' (T429361) 2427596 wikitext text/x-wiki 0cukupf40y5m1q0v3eviqj1yv777rhw 2427598 2427596 2026-06-16T16:01:47Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2010-dev.codfw.wmnet' (T429361) 2427598 wikitext text/x-wiki e76b160aocsimbecghbbf5gpjihnip8 2427606 2427598 2026-06-16T16:12:13Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet' (T429361) 2427606 wikitext text/x-wiki 6w6wbabkuixhwfpsrsx9qyagar02npc 2427608 2427606 2026-06-16T16:22:57Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1011.eqiad.wmnet' (T429361) 2427608 wikitext text/x-wiki 7r0cas223blmebxbidvdss7xqxndoty 2427609 2427608 2026-06-16T16:23:27Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' (T429361) 2427609 wikitext text/x-wiki hgqez2r06mqc4fenpafvgdn6dk9ko9h 2427614 2427609 2026-06-16T16:35:58Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' (T429361) 2427614 wikitext text/x-wiki hotu51pu0nsmr7hlz5si4ya7hihkrfc 2427615 2427614 2026-06-16T16:36:14Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' (T429361) 2427615 wikitext text/x-wiki pdpdwhzautovfnqp4u1onrvrlekfp1u 2427616 2427615 2026-06-16T16:47:03Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' (T429361) 2427616 wikitext text/x-wiki tn1ckelet4a40rp87zjl4c4t077y17m 2427617 2427616 2026-06-16T16:47:41Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.upgrade_osds (T428385) 2427617 wikitext text/x-wiki t5t91nhusehmfzhkuk1ge99t5vmeqov 2427618 2427617 2026-06-16T16:47:46Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.upgrade_osds (exit_code=99) (T428385) 2427618 wikitext text/x-wiki 7hh3beua8ub1q1o2m3wdhyo3zpku6r7 2427622 2427618 2026-06-16T17:04:58Z Stashbot 7414 andrewbogott: rebooting cloudcephosd1037 and cloudcephosd1038 because of missing volumes. The volumes reappeared after boot. 2427622 wikitext text/x-wiki cg6zjdymp1vejfijsdjlcpdcw4hth1v 2427643 2427622 2026-06-16T17:51:30Z Stashbot 7414 andrewbogott: ceph tell osd.126,127,129,131 compact 2427643 wikitext text/x-wiki 0sk6cm0se9efg4k2reb9u5r8fbgenhz 0 442952 2427510 2424557 2026-06-16T13:22:57Z DCaro (WMF) 18271 Updated my own stars and such 2427510 wikitext text/x-wiki 33mm5s1jv6oab5etxiiasxakwobk64r 498 443628 2427377 2427019 2026-06-16T12:04:14Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770908 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9) 2427377 wikitext text/x-wiki hpw1r6lzujyf1sfudnzkuhzmm9olp1b 2427620 2427377 2026-06-16T17:01:05Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634022645 (https://github.com/cluebotng/component-configs/commits/8a9410bd29ead90dcd183405a7c6ef35c0d898af) 2427620 wikitext text/x-wiki 7v2j1vbblta0c4evtw628urlmi96hk5 2427625 2427620 2026-06-16T17:13:30Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634737553 (https://github.com/cluebotng/component-configs/commits/a1821e6d783f3a9822825a8325623592bd1754ea) 2427625 wikitext text/x-wiki k120a7khct844iyix34dcspsjfh8xmm 2427633 2427625 2026-06-16T17:32:52Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27635849976 (https://github.com/cluebotng/component-configs/commits/893b9730fafbbd5762e1473e55ad6dbe8cb9db24) 2427633 wikitext text/x-wiki ahveliem5pxv9e7plltcihgo7gdjmk6 2427675 2427633 2026-06-16T18:48:11Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27639941201 (https://github.com/cluebotng/component-configs/commits/893b9730fafbbd5762e1473e55ad6dbe8cb9db24) 2427675 wikitext text/x-wiki 3guu0awrw80havdyx5rqgx84iatjru1 0 444008 2427485 2426167 2026-06-16T13:00:00Z BJensen-WMF 48942 /* Testing and deploying config */ Specify apache config location. 2427485 wikitext text/x-wiki n69mb3c2df80mg0t6qxaeau7ys520k0 498 446211 2427365 2425668 2026-06-16T12:00:24Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770905 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9) 2427365 wikitext text/x-wiki 1jgfmobxliidj6ntj2afq6q8g3i2vwa 498 447152 2427576 2405536 2026-06-16T15:15:21Z Stashbot 7414 dcausse: drop toolhub_* indices from the main cirrussearch opensearch cluster (T423993) 2427576 wikitext text/x-wiki s9r99aeh4tdfsws1gfkja1stvpoas4a 10 447759 2427818 2324782 2026-06-17T02:53:28Z FGT0043 56614 /* */ 2427818 wikitext text/x-wiki 9dhyu8f228dirhwdd8ti5xcgu1dess7 2427819 2427818 2026-06-17T03:17:30Z PieWriter 50962 Undid edits by [[Special:Contribs/FGT0043|FGT0043]] ([[User talk:FGT0043|talk]]) to last version by JCrespo (WMF): test edits, please use the sandbox 2427819 wikitext text/x-wiki et6xojjs6qcagk1dv1382ni509ujwts 0 447760 2427817 2420450 2026-06-17T02:52:01Z FGT0043 56614 2427817 wikitext text/x-wiki hj3z3a75bocay1e26lzd1mcxbwtx1kd 2427846 2427817 2026-06-17T07:16:03Z PieWriter 50962 Reverted 1 edit by [[Special:Contributions/FGT0043|FGT0043]] ([[User talk:FGT0043|talk]]): Vandalism (TwinkleGlobal) 2427846 wikitext text/x-wiki a9lz7n07bwrup03z36713k56occ6qsd 0 448996 2427401 2423997 2026-06-16T12:15:11Z DAdedoyin (WMF) 39150 2427401 wikitext text/x-wiki 7m38cll394e3ap5r526namq75i9pz7s 0 449160 2427809 2427331 2026-06-17T00:02:11Z Dexbot 30554 Bot: Updating the report 2427809 wikitext text/x-wiki tvsv2mcq06avwwbztuf5bavdmmh0yo2 2427910 2427809 2026-06-17T08:54:24Z Dexbot 30554 Bot: Updating the report 2427910 wikitext text/x-wiki fsyonrlt2fvwoswvaizad0lr587pisb 2427922 2427910 2026-06-17T09:03:26Z Dexbot 30554 Bot: Updating the report 2427922 wikitext text/x-wiki li08dq665o9ni19yc4g48hzajg7h3d0 2427939 2427922 2026-06-17T09:30:32Z Dexbot 30554 Bot: Updating the report 2427939 wikitext text/x-wiki 83bp6s3y4we2faulzsx8ez9h2uzwnj3 0 451075 2427704 2427164 2026-06-16T20:04:33Z DAdedoyin (WMF) 39150 /* Current */ 2427704 wikitext text/x-wiki 1xejmiqt65t4g65lytvv4nuc8camotf 498 451572 2428005 2379435 2026-06-17T11:02:35Z Stashbot 7414 volans@cloudcumin1001: START - Cookbook wmcs.vps.delete_project for project netops-clab in eqiad1 (T429447) 2428005 wikitext text/x-wiki g94ioxe3ruc3q40jfru9wdutusezn57 498 452890 2427372 2427084 2026-06-16T12:02:14Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770870 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9) 2427372 wikitext text/x-wiki 0zwnganld9emv44v2wji1orbcoi6m7v 2427621 2427372 2026-06-16T17:01:14Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634025367 (https://github.com/cluebotng/component-configs/commits/8a9410bd29ead90dcd183405a7c6ef35c0d898af) 2427621 wikitext text/x-wiki h3t4802wepjhg4sbpzutevipqlupyyz 2427626 2427621 2026-06-16T17:14:39Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634737527 (https://github.com/cluebotng/component-configs/commits/a1821e6d783f3a9822825a8325623592bd1754ea) 2427626 wikitext text/x-wiki 4j21tgeb68xc7zz9d4wx4fytahc62m1 2427634 2427626 2026-06-16T17:34:07Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27635849959 (https://github.com/cluebotng/component-configs/commits/893b9730fafbbd5762e1473e55ad6dbe8cb9db24) 2427634 wikitext text/x-wiki 4fmo6jt2rfyp8gfsi7v9kcqcl1kmy6s 116 453906 2427957 2427191 2026-06-17T09:45:30Z Gitlabaccountapprovalbot 37332 nathanveritas was rejected. 2427957 wikitext text/x-wiki n6uwwejkjvjqjyqe67bps8n2zife1rb 2428030 2427957 2026-06-17T11:24:19Z Gitlabaccountapprovalbot 37332 adinathq8x was rejected. 2428030 wikitext text/x-wiki 7wpg7du6ne7ptgb97x4s9i2gx2uyuyb 0 455989 2427734 2419925 2026-06-16T20:44:16Z MPopov (WMF) 5891 /* Test */ 2427734 wikitext text/x-wiki q0fftjcuf088nikohlbyoc7s1p44kc5 2427744 2427734 2026-06-16T21:06:21Z MPopov (WMF) 5891 /* Test */ 2427744 wikitext text/x-wiki 130a0616bpbmd0lk1wnin71luainjk8 0 456130 2427955 2334493 2026-06-17T09:43:17Z FNegri-WMF 32595 Add link to web interface 2427955 wikitext text/x-wiki oivs9ckztxdnevew8b3um4prrvhztm4 2427956 2427955 2026-06-17T09:43:34Z FNegri-WMF 32595 /* Promote packages to the main repository */ Remove link to resolved issue. 2427956 wikitext text/x-wiki c6ne8y9ooenrhsm4101r4kvzydz8hiv 0 457955 2427816 2414345 2026-06-17T02:50:56Z ADenisse-WMF 31462 /* 1. Install Yubikey management software */ 2427816 wikitext text/x-wiki 2b3j9vty3hpl6mzb30ihq0c9hql9gis 498 458787 2427413 2427362 2026-06-16T12:18:37Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27616804682 (https://github.com/cluebotng/component-configs/commits/af2bc530f42e1d932c23f8bea9c8d3686c343d10) 2427413 wikitext text/x-wiki sz10mq8xdpai5wgfdgcof9apogwuiu9 0 459044 2427687 2418567 2026-06-16T19:14:22Z BKing (WMF) 29017 /* Deployment Environments (codfw, eqiad, cloudelastic, relforge) */ 2427687 wikitext text/x-wiki fmiemghncikphd9l7xthc5b3savafmj 498 459243 2427371 2427361 2026-06-16T12:02:13Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615852687 (https://github.com/cluebotng/component-configs/commits/c9746412542e654e3ae7337a570727eb5d7195d6) 2427371 wikitext text/x-wiki 16tx5asjpsuw50oqhe63qurt42087vh 2427417 2427371 2026-06-16T12:20:02Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27616668922 (https://github.com/cluebotng/component-configs/commits/1bfb0487d9d8b3b799e0d9a30269ca314046feba) 2427417 wikitext text/x-wiki jpycz3d6umclezxzhqq4v5eqc9z6f87 2427420 2427417 2026-06-16T12:23:57Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27616804749 (https://github.com/cluebotng/component-configs/commits/af2bc530f42e1d932c23f8bea9c8d3686c343d10) 2427420 wikitext text/x-wiki gr7vdnqg1685kgs9fzjnyx5sbc6b0no 2427425 2427420 2026-06-16T12:30:44Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27617061512 (https://github.com/cluebotng/component-configs/commits/4505fa21f50c25710eaf644eecdc0233f726c946) 2427425 wikitext text/x-wiki q7wg0abpujounagmtkmfpym2lyw59dw 2427426 2427425 2026-06-16T12:34:26Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27617439342 (https://github.com/cluebotng/component-configs/commits/25d55050b7cd85801d40ba66cf804d6d4a3b51bd) 2427426 wikitext text/x-wiki 7k20st3ejgyea6965jdkf01isyi08xk 2427480 2427426 2026-06-16T12:51:36Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27618465342 (https://github.com/cluebotng/component-configs/commits/9b3fcbbd479041d764ab53f0a74027e2df6df4f6) 2427480 wikitext text/x-wiki g6r35t5l5xw60j79irjwwoh8x26yz1b 498 459244 2427364 2425669 2026-06-16T12:00:21Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615771226 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9) 2427364 wikitext text/x-wiki 7ftzhzahpnfh9o3ffokss72932ty8fl 2427619 2427364 2026-06-16T16:58:28Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634024663 (https://github.com/cluebotng/component-configs/commits/8a9410bd29ead90dcd183405a7c6ef35c0d898af) 2427619 wikitext text/x-wiki r938ez53vjhcmdi1i2q10dymh5mqxis 2427623 2427619 2026-06-16T17:10:34Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634738411 (https://github.com/cluebotng/component-configs/commits/a1821e6d783f3a9822825a8325623592bd1754ea) 2427623 wikitext text/x-wiki tqsdjrvbzdeqhgv63enm26yeh7a68r6 2427631 2427623 2026-06-16T17:30:03Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27635849962 (https://github.com/cluebotng/component-configs/commits/893b9730fafbbd5762e1473e55ad6dbe8cb9db24) 2427631 wikitext text/x-wiki ouq2v8k62bvmhcvwld1vzzhi0w0o4lh 498 459255 2427366 2425666 2026-06-16T12:00:24Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770932 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9) 2427366 wikitext text/x-wiki 32xv3tgsvr8i7hzxo5p9mo8vuhns8k8 498 459263 2427367 2425663 2026-06-16T12:00:40Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770917 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9) 2427367 wikitext text/x-wiki 82o1qkmx1m5x37ek3stczokt9uognx5 498 459362 2427363 2425513 2026-06-16T12:00:01Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770896 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9) 2427363 wikitext text/x-wiki ncbm6rvnjomqfcz9hbl1stu9i5mi9es 2427409 2427363 2026-06-16T12:17:23Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27616804754 (https://github.com/cluebotng/component-configs/commits/af2bc530f42e1d932c23f8bea9c8d3686c343d10) 2427409 wikitext text/x-wiki qr5s82wnubupnxnr731yhtx0d3jp8dj 0 459560 2427556 2422324 2026-06-16T14:45:22Z DAdedoyin (WMF) 39150 Test 2427556 wikitext text/x-wiki 23h7r6vglzs2pg4vek1mnlirk1jsn3n 0 459699 2427729 2427198 2026-06-16T20:31:45Z Stashbot 7414 ejegg: standalone (ipn listener) SmashPig upgraded from 611947be to 9cd51fd1 2427729 wikitext text/x-wiki 9u7ldcneirxu3pan9nep5d68d4dmcrw 2427811 2427729 2026-06-17T01:22:30Z Stashbot 7414 ejegg: payments-wiki upgraded from 1f5f40f9 to 873882a5 2427811 wikitext text/x-wiki 9pqj67z9qz75i33xcdj7t6i4mi001xs 0 460161 2427427 2427153 2026-06-16T12:36:26Z Udehb-WMF 40856 /* Architecture */ 2427427 wikitext text/x-wiki k3ehu0hbsz4vbojxcjzf8w5hwmlohno 0 460177 2427770 2417430 2026-06-16T21:36:55Z Effie Mouzeli (WMF) 12880 2427770 wikitext text/x-wiki cpx310zadigidehwqjcwlmagweb3ty6 0 460178 2427740 2424670 2026-06-16T20:54:44Z Effie Mouzeli (WMF) 12880 scrollbot update 2427740 wikitext text/x-wiki 1c6r9gw4uikvhpnpxk7950qp3gfh4dc 2427742 2427740 2026-06-16T21:02:41Z Effie Mouzeli (WMF) 12880 2427742 wikitext text/x-wiki 2zwf3htlp3bzhmw8h9dy7yprc8lbh1r 2427752 2427742 2026-06-16T21:19:37Z Effie Mouzeli (WMF) 12880 scrollbot update 2427752 wikitext text/x-wiki iylrefsx8l5m7i5q4501epy4ml4yi41 14 460180 2427706 2417001 2026-06-16T20:09:23Z Effie Mouzeli (WMF) 12880 Effie Mouzeli (WMF) moved page [[Category:SCROLLs]] to [[Category:Scrolls]] without leaving a redirect: use lowercase 2417001 wikitext text/x-wiki 1uyr6jbis31k9f4l753jmdkvcmcof81 0 460191 2427748 2421060 2026-06-16T21:16:26Z Effie Mouzeli (WMF) 12880 scrollbot update 2427748 wikitext text/x-wiki 8d7b7fazznkew21hn95rp5scc2wwvhs 2427763 2427748 2026-06-16T21:30:25Z Effie Mouzeli (WMF) 12880 scrollbot update 2427763 wikitext text/x-wiki m38xmjxyny0jibzr3qd27bug5tka4us 0 460192 2427736 2424406 2026-06-16T20:48:58Z Effie Mouzeli (WMF) 12880 scrollbot update 2427736 wikitext text/x-wiki ta8ez11lt30c05eqa3x4yxnu51wi3jf 0 460194 2427723 2426810 2026-06-16T20:28:07Z Effie Mouzeli (WMF) 12880 scrollbot update 2427723 wikitext text/x-wiki 008ibe2puka57brlt00axungqfbs1l0 14 460200 2427741 2417013 2026-06-16T21:01:30Z Effie Mouzeli (WMF) 12880 2427741 wikitext text/x-wiki g6t01ia0v5pzs8ogxeg3pumdfof8jr6 14 460201 2427708 2417018 2026-06-16T20:12:01Z Effie Mouzeli (WMF) 12880 2427708 wikitext text/x-wiki g6t01ia0v5pzs8ogxeg3pumdfof8jr6 14 460208 2427749 2417076 2026-06-16T21:17:24Z Effie Mouzeli (WMF) 12880 2427749 wikitext text/x-wiki g6t01ia0v5pzs8ogxeg3pumdfof8jr6 0 460209 2427738 2417560 2026-06-16T20:54:14Z Effie Mouzeli (WMF) 12880 scrollbot update 2427738 wikitext text/x-wiki 5zqa67krxjzjfszdmsd7k4bzud2m5vk 0 460210 2427722 2417140 2026-06-16T20:27:44Z Effie Mouzeli (WMF) 12880 scrollbot update 2427722 wikitext text/x-wiki 718lraa30iylgaa3wxt2tbupch2ll8k 0 460234 2427743 2421968 2026-06-16T21:03:21Z MPopov (WMF) 5891 2427743 wikitext text/x-wiki ls8zdaq8sbos3qz7a588uol0ophep11 2 460317 2427656 2425254 2026-06-16T18:09:17Z ASarabadani (WMF) 39030 2427656 wikitext text/x-wiki 64emy9abb75qohq7zsz1zyq81cy8ll4 0 460318 2427537 2426742 2026-06-16T13:51:14Z Atsuko-WMF 54082 /* Verification */ additional link for Special:Translate 2427537 wikitext text/x-wiki e67xgthar4w2o31d7vgfwpzhdryu7mt 2 460340 2427560 2427313 2026-06-16T14:50:23Z Phuedx (WMF) 47850 Snapshot 2427560 wikitext text/x-wiki t3kle5gd7ulvtdy5aym1ujixpkpuqz9 0 460341 2427483 2026-06-16T12:53:42Z BTullis (WMF) 25295 Added a new request to review the topolvm chart 2427483 wikitext text/x-wiki cy0cfyqrv5zx3m62xf064bp7ge9kysq 12 460342 2427574 2026-06-16T15:08:46Z Tejinderk.2004 56571 Added basic page skeleton on Managing and sharing files in Toolforge , yet to add a few sections and condense original sources 2427574 wikitext text/x-wiki jd6l4dvcvpgl5zdp2390hbbod71a7gm 2427645 2427574 2026-06-16T17:55:48Z APaskulin (WMF) 14880 Adjust headings levels to start with h2 since on wiki pages the title is the h1, and remove the "Introduction" heading to create a lead section 2427645 wikitext text/x-wiki ozsc9mmmw1kpljtiy9c89k0620o6t42 2427916 2427645 2026-06-17T08:58:48Z Tejinderk.2004 56571 removed duplicate text in the table 2427916 wikitext text/x-wiki m1t5d28x3g9esp6b2mss99c5r7h3r8x 0 460343 2427607 2026-06-16T16:15:49Z Effie Mouzeli (WMF) 12880 Created page with "<includeonly>{{#switch: {{lc:{{{1|requested}}}}} | in-progress | in progress | inprogress | progress = <div style="background: #fef6e7; border: 1px solid #a2a9b1; border-left: 6px solid #ac6600; border-radius: 4px; padding: 10px 16px; margin: 0 0 1.2em 0;">'''🛠️ SCROLL stage: in progress.''' The team is working through the checklist items.</div>[[Category:SCROLL in progress]] | in-review | in review | inreview | review | reviews = <div style="background: #eaf3ff; bo..." 2427607 wikitext text/x-wiki nzdz5lfx7dakspxrx1vnozk1fqdy4in 2427705 2427607 2026-06-16T20:06:35Z Effie Mouzeli (WMF) 12880 scrollbot update 2427705 wikitext text/x-wiki kd6tw4oxkn27m399x7fl27v7uq5co22 2427718 2427705 2026-06-16T20:26:01Z Effie Mouzeli (WMF) 12880 Replaced content with "nothing" 2427718 wikitext text/x-wiki 1uyr6jbis31k9f4l753jmdkvcmcof81 0 460344 2427610 2026-06-16T16:25:17Z Effie Mouzeli (WMF) 12880 Created page with "TBA" 2427610 wikitext text/x-wiki kd1620ukml0ssm2lahetsf8y997ay3c 2427611 2427610 2026-06-16T16:25:22Z Effie Mouzeli (WMF) 12880 scrollbot update 2427611 wikitext text/x-wiki nylsztglr27ta9sn2sitf5g97m3dyyb 2427612 2427611 2026-06-16T16:26:07Z Effie Mouzeli (WMF) 12880 scrollbot update 2427612 wikitext text/x-wiki jfdard6yojr5yzcqcdklfhp6urj16zk 2427793 2427612 2026-06-16T22:37:38Z Effie Mouzeli (WMF) 12880 scrollbot update 2427793 wikitext text/x-wiki 0ajr9oqeucrvimttrnrxygbrqruz2y3 2427794 2427793 2026-06-16T22:38:41Z Effie Mouzeli (WMF) 12880 scrollbot update 2427794 wikitext text/x-wiki 6hyz8rasvotuxm4iywibpd7ohn6hcpd 3 460345 2427709 2026-06-16T20:17:33Z Poro26 40508 Created page with "<nowiki>{{Welcome}}</nowiki> ~~~~" 2427709 wikitext text/x-wiki fgvyhgjz7p50wgy2xg1koz9li9st5x3 2427712 2427709 2026-06-16T20:19:13Z Poro26 40508 Blanked the page 2427712 wikitext text/x-wiki phoiac9h4m842xq45sp7s6u21eteeq1 2427768 2427712 2026-06-16T21:35:12Z Poro26 40508 /* Message de bienvenue */ new section 2427768 wikitext text/x-wiki g47mmmq6ljuqdn52v69pxwpm9l2cyzr 10 460346 2427714 2026-06-16T20:20:14Z Effie Mouzeli (WMF) 12880 Created page with "<includeonly>{{#switch: {{lc:{{{1|requested}}}}} | in-progress | in progress | inprogress | progress = <div style="background: #fef6e7; border: 1px solid #a2a9b1; border-left: 6px solid #ac6600; border-radius: 4px; padding: 10px 16px; margin: 0 0 1.2em 0;">'''🛠️ SCROLL stage: in progress.''' The team is working through the checklist items.</div>[[Category:SCROLL in progress]] | in-review | in review | inreview | review | reviews = <div style="background: #eaf3ff; bo..." 2427714 wikitext text/x-wiki kd6tw4oxkn27m399x7fl27v7uq5co22 2427716 2427714 2026-06-16T20:23:44Z Effie Mouzeli (WMF) 12880 scrollbot update 2427716 wikitext text/x-wiki os3i9k66sej6uvxvs93lzsszthad3x0 2427751 2427716 2026-06-16T21:19:21Z Effie Mouzeli (WMF) 12880 scrollbot update 2427751 wikitext text/x-wiki femylyhfpsg0sxcvj62708cw1h21xv8 14 460347 2427760 2026-06-16T21:26:28Z Effie Mouzeli (WMF) 12880 Created page with "[[Category: Scrolls]]" 2427760 wikitext text/x-wiki 8lw1bghvg3qxjdbp7z086wqfn878icf 14 460348 2427769 2026-06-16T21:35:15Z Effie Mouzeli (WMF) 12880 Created page with "tba" 2427769 wikitext text/x-wiki tniwr98iwgfkad3x93czd6gjzngw0dl 3 460349 2427771 2026-06-16T21:38:10Z Poro26 40508 /* Message de bienvenue */ new section 2427771 wikitext text/x-wiki rbvktr4q94angqz6k806zodh6wbpd7i 2427779 2427771 2026-06-16T21:47:40Z Tways2017 56604 /* Message de bienvenue */ Reply 2427779 wikitext text/x-wiki n4g3qrjkd9l66hwi1e2zp8gturu1e3w