Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.46.0-wmf.24 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 0 4108 2402344 2402117 2026-04-15T13:13:05Z ScheduleDeploymentBot 37566 Add [[gerrit:1140748]] to Thursday, April 16 UTC afternoon backport window 2402344 wikitext text/x-wiki n99goppvc2j88yt6huweworegf0t4tj 2402413 2402344 2026-04-15T14:46:24Z ScheduleDeploymentBot 37566 Add [[gerrit:1267116]] to Thursday, April 16 UTC late backport window 2402413 wikitext text/x-wiki icmqouxhcpgtofwyr07x3bhxm1ra4xa 2402544 2402413 2026-04-15T17:46:36Z ScheduleDeploymentBot 37566 Add [[gerrit:1271832]] to Wednesday, April 15 UTC late backport window 2402544 wikitext text/x-wiki 7q4dowhhohzvpxdhzafxbs22j30h907 2402546 2402544 2026-04-15T17:47:08Z ScheduleDeploymentBot 37566 Add [[gerrit:1271833]] to Wednesday, April 15 UTC late backport window 2402546 wikitext text/x-wiki m3e9fogpbm1x57mzhrbv30jclhbru4y 2402548 2402546 2026-04-15T17:47:33Z ScheduleDeploymentBot 37566 Add [[gerrit:1271831]] to Wednesday, April 15 UTC late backport window 2402548 wikitext text/x-wiki 4b4yhfzh5wf9ci8muncap0g0l8157bb 2402550 2402548 2026-04-15T17:47:50Z ScheduleDeploymentBot 37566 Add [[gerrit:1271830]] to Wednesday, April 15 UTC late backport window 2402550 wikitext text/x-wiki qkzg11ousuo47yvvm7lb74c5df6jgjg 2402569 2402550 2026-04-15T18:00:27Z ScheduleDeploymentBot 37566 Add [[gerrit:1271839]] to Wednesday, April 15 UTC late backport window 2402569 wikitext text/x-wiki 6jmh0zyyfnznx2nk4cryhfddk94ogjr 2402570 2402569 2026-04-15T18:00:56Z ScheduleDeploymentBot 37566 Add [[gerrit:1268300]] to Wednesday, April 15 UTC late backport window 2402570 wikitext text/x-wiki 7uc47o8vzzl4saxl3yes3iqh0gadq90 2402591 2402570 2026-04-15T19:55:08Z ScheduleDeploymentBot 37566 Add [[gerrit:1270583]] to Wednesday, April 15 UTC late backport window 2402591 wikitext text/x-wiki c6znq54kvdyh8fr135sygk4pyo9jqj4 2402597 2402591 2026-04-15T20:01:29Z ScheduleDeploymentBot 37566 Add [[gerrit:1271886]] to Wednesday, April 15 UTC late backport window 2402597 wikitext text/x-wiki iy20ftxcvvb2b9y7w7fp3c8c9douoer 2402617 2402597 2026-04-15T20:34:27Z Jdforrester (WMF) 468 /* {{Deployment_day|date=2026-04-15}} */ Five done. 2402617 wikitext text/x-wiki dkc5rb5g1bpdy98oifh57q2qyi3tnps 2402626 2402617 2026-04-15T20:48:48Z ScheduleDeploymentBot 37566 Add [[gerrit:1271889]] to Wednesday, April 15 UTC late backport window 2402626 wikitext text/x-wiki chm5b7s88o3b8i37q59e25yidfe462d 2402627 2402626 2026-04-15T20:48:57Z ScheduleDeploymentBot 37566 Add [[gerrit:1271910]] to Wednesday, April 15 UTC late backport window 2402627 wikitext text/x-wiki oi8lnurmz048x9iwy5c6lpqbqpvnfw4 2402628 2402627 2026-04-15T20:49:06Z ScheduleDeploymentBot 37566 Add [[gerrit:1271911]] to Wednesday, April 15 UTC late backport window 2402628 wikitext text/x-wiki n1rw3avs3n6u03x0t22dvhgmftdffah 2402633 2402628 2026-04-15T20:58:21Z Jdforrester (WMF) 468 /* {{Deployment_day|date=2026-04-15}} */ Three done. 2402633 wikitext text/x-wiki f01t4go98zqmekh2weqrac13fmknau8 2402725 2402633 2026-04-16T04:57:01Z ScheduleDeploymentBot 37566 Add [[gerrit:1270567]] to Thursday, April 16 UTC morning backport window 2402725 wikitext text/x-wiki 8v0eqhucflabv9prz1llvd7yx1v26le 2402752 2402725 2026-04-16T06:57:18Z ScheduleDeploymentBot 37566 Add [[gerrit:1272424]] to Thursday, April 16 UTC morning backport window 2402752 wikitext text/x-wiki ei4785x3zamielmh7dmv8zo0i0mlsf6 2402789 2402752 2026-04-16T08:00:29Z ScheduleDeploymentBot 37566 Add [[gerrit:1272527]] to Thursday, April 16 UTC afternoon backport window 2402789 wikitext text/x-wiki 04bomh6s2ptiq4kbv720tj4a7n9kw9y 2402885 2402789 2026-04-16T10:53:29Z ScheduleDeploymentBot 37566 Add [[gerrit:1272447]] to Thursday, April 16 UTC afternoon backport window 2402885 wikitext text/x-wiki hglmnkzvmvjmmobhk43f9aoz01n0vyw 2402892 2402885 2026-04-16T11:03:50Z A826 34037 /* {{Deployment_day|date=2026-04-16}} */ 2402892 wikitext text/x-wiki ckbz5nlt2u7mzinzu0wfazo4aq0ooxd 0 7919 2402249 2402248 2026-04-15T12:00:46Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2069.codfw.wmnet with OS trixie 2402249 wikitext text/x-wiki o3mghcexxaoi76tded0c60t8c7plprp 2402250 2402249 2026-04-15T12:01:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T419961)', diff saved to https://phabricator.wikimedia.org/P90766 and previous config saved to /var/cache/conftool/dbconfig/20260415-120117-fceratto.json 2402250 wikitext text/x-wiki 5biiub6gp8mppea50thmt69va02evfy 2402251 2402250 2026-04-15T12:01:31Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance 2402251 wikitext text/x-wiki 5rd12kxvdtgzuo0025klcs00ft7kg2x 2402252 2402251 2026-04-15T12:01:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2169 (T419961)', diff saved to https://phabricator.wikimedia.org/P90767 and previous config saved to /var/cache/conftool/dbconfig/20260415-120138-fceratto.json 2402252 wikitext text/x-wiki smadqjevimu3ze43pj1e21v6vlp87mm 2402253 2402252 2026-04-15T12:03:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T419635)', diff saved to https://phabricator.wikimedia.org/P90768 and previous config saved to /var/cache/conftool/dbconfig/20260415-120305-fceratto.json 2402253 wikitext text/x-wiki nf90nndmyuwgfz8iiyv66yh2i2k9hbd 2402254 2402253 2026-04-15T12:03:25Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1226.eqiad.wmnet with reason: Maintenance 2402254 wikitext text/x-wiki lo53lfskpd5el8yn8ypk58jk11lfrd0 2402255 2402254 2026-04-15T12:03:35Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1226 (T419635)', diff saved to https://phabricator.wikimedia.org/P90769 and previous config saved to /var/cache/conftool/dbconfig/20260415-120331-fceratto.json 2402255 wikitext text/x-wiki 04q8wgy7pccpvnc27ee51pmiygqibuq 2402256 2402255 2026-04-15T12:04:17Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2069.codfw.wmnet with OS bullseye 2402256 wikitext text/x-wiki tjrn6gjhdil5t88acwsdr4he5lkmqi7 2402257 2402256 2026-04-15T12:07:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T419635)', diff saved to https://phabricator.wikimedia.org/P90770 and previous config saved to /var/cache/conftool/dbconfig/20260415-120739-fceratto.json 2402257 wikitext text/x-wiki 2opp6tnvhth1e96c5nyz629xqtecoie 2402258 2402257 2026-04-15T12:08:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T419961)', diff saved to https://phabricator.wikimedia.org/P90771 and previous config saved to /var/cache/conftool/dbconfig/20260415-120851-fceratto.json 2402258 wikitext text/x-wiki r8s64ib6jo51ixmvgh9yq54nwjq22b9 2402259 2402258 2026-04-15T12:11:30Z Stashbot 7414 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply 2402259 wikitext text/x-wiki qwmwbx929sx3oxi70eyl8s8d4b8hdn1 2402260 2402259 2026-04-15T12:11:54Z Stashbot 7414 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply 2402260 wikitext text/x-wiki shh7xlf5mde96s3yxylrow7665b4gpc 2402261 2402260 2026-04-15T12:17:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P90772 and previous config saved to /var/cache/conftool/dbconfig/20260415-121748-fceratto.json 2402261 wikitext text/x-wiki 1pk57os0qg8iro4jci2k2sbhlxp4f4n 2402262 2402261 2026-04-15T12:19:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P90773 and previous config saved to /var/cache/conftool/dbconfig/20260415-121859-fceratto.json 2402262 wikitext text/x-wiki 1i7ervrpd30j055dmo5ma6jkzijwjy8 2402263 2402262 2026-04-15T12:21:41Z Stashbot 7414 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply 2402263 wikitext text/x-wiki a77u825sz95gnjxe7ekg8uj410dt092 2402264 2402263 2026-04-15T12:22:13Z Stashbot 7414 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply 2402264 wikitext text/x-wiki dh1vgs9p73jbd04gu710hi9banu2shk 2402265 2402264 2026-04-15T12:22:18Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2069.codfw.wmnet with reason: host reimage 2402265 wikitext text/x-wiki ilq4nm4lgzb466kgwj2newa3i7fk3jq 2402266 2402265 2026-04-15T12:22:37Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402266 wikitext text/x-wiki 82e5s3yz6a389dc9dyixw0s0xt3v91a 2402267 2402266 2026-04-15T12:22:58Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402267 wikitext text/x-wiki kbf17aun2htunpgrh9mmnzx97jzgxmu 2402268 2402267 2026-04-15T12:23:13Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402268 wikitext text/x-wiki bxhmb3zk321upt4w18mnsd3revcl59w 2402269 2402268 2026-04-15T12:25:10Z Stashbot 7414 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply 2402269 wikitext text/x-wiki bw5jelprhilw54ayd5c7h0td9caln47 2402270 2402269 2026-04-15T12:25:32Z Stashbot 7414 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) 2402270 wikitext text/x-wiki 76nrjih723wcej9btbh04mzmy9u1rr1 2402271 2402270 2026-04-15T12:25:39Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402271 wikitext text/x-wiki iscislgjip0l4f7ki3kk1904evnh3ye 2402272 2402271 2026-04-15T12:25:44Z Stashbot 7414 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply 2402272 wikitext text/x-wiki f4k8bcbvqmeb6bigi1s179asitq53xz 2402273 2402272 2026-04-15T12:25:47Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402273 wikitext text/x-wiki d06f61p1u3f3x01zwis6jg347wg4nba 2402274 2402273 2026-04-15T12:26:15Z Stashbot 7414 kart_: Updated cxserver to 2026-04-14-071531-production 2402274 wikitext text/x-wiki nqozyeeqdddgkjfejol1rh3r8ze82m9 2402275 2402274 2026-04-15T12:27:22Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402275 wikitext text/x-wiki 0ubczdsta2tjft073fwgb9159y4tvhu 2402276 2402275 2026-04-15T12:27:34Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402276 wikitext text/x-wiki ij4f98vok8te57lhz1waxjd48jpwwv8 2402277 2402276 2026-04-15T12:27:57Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P90774 and previous config saved to /var/cache/conftool/dbconfig/20260415-122756-fceratto.json 2402277 wikitext text/x-wiki 03lqydud1v6xm5qnibwubimubkudako 2402278 2402277 2026-04-15T12:28:51Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2069.codfw.wmnet with reason: host reimage 2402278 wikitext text/x-wiki 7dbeh8mmtui8la4hwu7e0vlxpbtdli5 2402279 2402278 2026-04-15T12:29:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P90775 and previous config saved to /var/cache/conftool/dbconfig/20260415-122907-fceratto.json 2402279 wikitext text/x-wiki cnmk408rpqy0s0qtmryvurhxgxyiywd 2402280 2402279 2026-04-15T12:29:33Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402280 wikitext text/x-wiki 3s0hdmyd3gcjwdi12b185opjpsr2x39 2402281 2402280 2026-04-15T12:30:04Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402281 wikitext text/x-wiki c9ks6l65b0zwsjxofiykflat7hom94l 2402282 2402281 2026-04-15T12:30:10Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402282 wikitext text/x-wiki 9hlx0f6bv1iquawqtww52zhu1sa3x9b 2402283 2402282 2026-04-15T12:30:19Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402283 wikitext text/x-wiki ecxna8xx4823feznjvayh0vm7z8j10q 2402284 2402283 2026-04-15T12:31:04Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402284 wikitext text/x-wiki 5vbm1gys49ks3gsy02p6fl5rie3n5cb 2402285 2402284 2026-04-15T12:31:16Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402285 wikitext text/x-wiki pizec0kqqd6xbge4dbvpub4rfwek1dr 2402286 2402285 2026-04-15T12:32:03Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402286 wikitext text/x-wiki n48ehwt2b1symz5flbkt6u3mhuyvjzy 2402287 2402286 2026-04-15T12:32:11Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402287 wikitext text/x-wiki r4pilcv9hicsy9af1qumhkxedczdjke 2402288 2402287 2026-04-15T12:33:51Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402288 wikitext text/x-wiki 9o88iqagu9osn1va2eps0q52m9d9s7o 2402289 2402288 2026-04-15T12:34:05Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402289 wikitext text/x-wiki ot3phpupof4m9ku9xozjwzjmyook6e3 2402290 2402289 2026-04-15T12:34:17Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402290 wikitext text/x-wiki pcbcpu4c0gaydd3lr2y1sadxvaxzwad 2402291 2402290 2026-04-15T12:34:26Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402291 wikitext text/x-wiki 35filq1j8fhvuacd0at2x2cc9j2ni0o 2402292 2402291 2026-04-15T12:34:54Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1271697|VisualEditor hCaptcha: Clear challenge container for new render (T423294)]] 2402292 wikitext text/x-wiki l2ol01m2v7hamwb2njupfznqq5gok4f 2402293 2402292 2026-04-15T12:34:57Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402293 wikitext text/x-wiki 7k1yvyb8kvf4lca9g8fsrnkn74o2r7u 2402294 2402293 2026-04-15T12:36:18Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402294 wikitext text/x-wiki pp2bc5op7vxqhmxr6eda4n8id0vi5oz 2402295 2402294 2026-04-15T12:36:50Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1271697|VisualEditor hCaptcha: Clear challenge container for new render (T423294)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2402295 wikitext text/x-wiki b1wk1dz9n4mwy984zkbcbm446ei3aws 2402296 2402295 2026-04-15T12:37:09Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402296 wikitext text/x-wiki bdpmbuubv9dfvgzew1a4cva9n15nzud 2402297 2402296 2026-04-15T12:37:17Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402297 wikitext text/x-wiki c41w58e5v1j4jzwugyqzw0spspaxuef 2402298 2402297 2026-04-15T12:37:45Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402298 wikitext text/x-wiki 444sq0x7yt3ba4zjnw5m1m8y88vboh0 2402299 2402298 2026-04-15T12:37:53Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402299 wikitext text/x-wiki 3yvisc6c2uy390z25r6ledpd8p4kkm9 2402300 2402299 2026-04-15T12:38:06Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T419635)', diff saved to https://phabricator.wikimedia.org/P90776 and previous config saved to /var/cache/conftool/dbconfig/20260415-123803-fceratto.json 2402300 wikitext text/x-wiki 0ny52vkpdcwqz0b4wj79gpnbgehto55 2402301 2402300 2026-04-15T12:38:14Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2218 (T410589)', diff saved to https://phabricator.wikimedia.org/P90777 and previous config saved to /var/cache/conftool/dbconfig/20260415-123811-ladsgroup.json 2402301 wikitext text/x-wiki emtvn05e8n2ejl7qnsxsttgryk57o2c 2402302 2402301 2026-04-15T12:38:45Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402302 wikitext text/x-wiki ac2wt0idlhkvssfprnrqfqf4xbjip30 2402303 2402302 2026-04-15T12:38:56Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402303 wikitext text/x-wiki npqf7egspy1096hm2gcbqvoqk0dxt7l 2402304 2402303 2026-04-15T12:39:15Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz: Continuing with sync 2402304 wikitext text/x-wiki 9pvy1pvbw315766pdgyq809qdjdylzy 2402305 2402304 2026-04-15T12:39:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T419961)', diff saved to https://phabricator.wikimedia.org/P90778 and previous config saved to /var/cache/conftool/dbconfig/20260415-123915-fceratto.json 2402305 wikitext text/x-wiki 74gnxbngjbqao00evmedx1einchezmi 2402306 2402305 2026-04-15T12:39:30Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2193.codfw.wmnet with reason: Maintenance 2402306 wikitext text/x-wiki qnj5uc9eek0jc0zwieeq21i2k2rp1jl 2402307 2402306 2026-04-15T12:39:39Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2193 (T419961)', diff saved to https://phabricator.wikimedia.org/P90779 and previous config saved to /var/cache/conftool/dbconfig/20260415-123937-fceratto.json 2402307 wikitext text/x-wiki t2i48aq4sot3ofez60wovarh19ss21p 2402308 2402307 2026-04-15T12:40:38Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402308 wikitext text/x-wiki ebdyfunnhygi7gs685s3l902d76t5sx 2402309 2402308 2026-04-15T12:40:45Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402309 wikitext text/x-wiki ibmcn1s5hsizvvwqtr7kercdqqqk1ii 2402310 2402309 2026-04-15T12:41:00Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402310 wikitext text/x-wiki pa72jui4mx41gy6x67b8adtxr57e53h 2402311 2402310 2026-04-15T12:41:15Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402311 wikitext text/x-wiki 3w26c2vuh5n7px5ot1dm8y3er5a5ntv 2402312 2402311 2026-04-15T12:41:43Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402312 wikitext text/x-wiki 343dstc9norrpkd95i4vsvzpsm83ruy 2402313 2402312 2026-04-15T12:41:53Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402313 wikitext text/x-wiki gq2mrhz994w296pydtwpg37k8menoip 2402314 2402313 2026-04-15T12:43:05Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1271697|VisualEditor hCaptcha: Clear challenge container for new render (T423294)]] (duration: 08m 11s) 2402314 wikitext text/x-wiki syo0br9ju4c7ne12s0gu26dvqkp4cdy 2402315 2402314 2026-04-15T12:43:36Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402315 wikitext text/x-wiki cu9t35gxf6rjddoi1upuk0hj04d8u2n 2402316 2402315 2026-04-15T12:43:49Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402316 wikitext text/x-wiki obvn3kt142wwvy88xn03vhaaal0qeo5 2402317 2402316 2026-04-15T12:44:10Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402317 wikitext text/x-wiki 681ab0l501urw6gecx0miq30x3j3670 2402318 2402317 2026-04-15T12:44:20Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402318 wikitext text/x-wiki gevasdgu600qh2a8o5rn05j65qhx8sj 2402319 2402318 2026-04-15T12:44:36Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402319 wikitext text/x-wiki gg3xkze79s5u6wlawehrtoygds3h1sq 2402320 2402319 2026-04-15T12:44:46Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402320 wikitext text/x-wiki czzzr7kfxlo51d4tydlblsbq3ttpkiv 2402321 2402320 2026-04-15T12:45:02Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402321 wikitext text/x-wiki sn0k2bbbn29wjjyt5hfcvs1ytvuwn0w 2402322 2402321 2026-04-15T12:45:13Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402322 wikitext text/x-wiki ag2pyrgad2zadytt9fgvh7t9bh9m1ff 2402323 2402322 2026-04-15T12:45:20Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402323 wikitext text/x-wiki od5qcbaokbf0408bceh0tr3yuq4dab0 2402324 2402323 2026-04-15T12:45:28Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402324 wikitext text/x-wiki qi6rhptiok5umjok6u0mi5yfy9f3pie 2402325 2402324 2026-04-15T12:45:55Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.update-replication 2402325 wikitext text/x-wiki c1l3vc2td6q8wjttxgde0lg3bv88cw7 2402326 2402325 2026-04-15T12:46:04Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) 2402326 wikitext text/x-wiki hasgbc351ogmj2pcfzshqfg6d2y5cd1 2402327 2402326 2026-04-15T12:46:35Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T419961)', diff saved to https://phabricator.wikimedia.org/P90780 and previous config saved to /var/cache/conftool/dbconfig/20260415-124633-fceratto.json 2402327 wikitext text/x-wiki 4gg3vxh2qdjvq4n8afhz9kpa05yd8n9 2402328 2402327 2026-04-15T12:48:19Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2069.codfw.wmnet with OS bullseye 2402328 wikitext text/x-wiki 28cr394gylzotwuq0q6icianiwfeoas 2402329 2402328 2026-04-15T12:48:21Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P90781 and previous config saved to /var/cache/conftool/dbconfig/20260415-124819-ladsgroup.json 2402329 wikitext text/x-wiki ne1c5r0gl3v19nf6byf6nqiy0vpg4s4 2402330 2402329 2026-04-15T12:56:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P90782 and previous config saved to /var/cache/conftool/dbconfig/20260415-125640-fceratto.json 2402330 wikitext text/x-wiki apy3f1gpddodik05we0ob8dnql262go 2402331 2402330 2026-04-15T12:58:30Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P90783 and previous config saved to /var/cache/conftool/dbconfig/20260415-125828-ladsgroup.json 2402331 wikitext text/x-wiki 9igiqq8lck5mjwlurw8fmtru0mimp5k 2402332 2402331 2026-04-15T13:00:12Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2068.codfw.wmnet with OS bullseye 2402332 wikitext text/x-wiki 3rmo2svsn7vkib1h437et3ir1im6olp 2402333 2402332 2026-04-15T13:04:38Z Stashbot 7414 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1270991|Register ArticleGuidance extension and enable in labs (T423295)]] 2402333 wikitext text/x-wiki 2zpu7la5auhlmp13ijmthx815nci5rw 2402334 2402333 2026-04-15T13:06:29Z Stashbot 7414 kartik@deploy1003: sbisson, kartik: Backport for [[gerrit:1270991|Register ArticleGuidance extension and enable in labs (T423295)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2402334 wikitext text/x-wiki 3udcfmtyjns9jjramfj29ai086f2ljb 2402335 2402334 2026-04-15T13:06:51Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P90784 and previous config saved to /var/cache/conftool/dbconfig/20260415-130649-fceratto.json 2402335 wikitext text/x-wiki 91ipu7s3lyb9chw87zylbdamsflzc6j 2402336 2402335 2026-04-15T13:07:21Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance 2402336 wikitext text/x-wiki p8fyyjjtft6rdoi68nn9jchp5lfycmo 2402337 2402336 2026-04-15T13:07:31Z Stashbot 7414 fceratto@cumin1003: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance 2402337 wikitext text/x-wiki 316t3x0kor4urj72r4ot79iapz9rwdj 2402338 2402337 2026-04-15T13:08:23Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance 2402338 wikitext text/x-wiki e080npnj98wrk69trdyrn72guwbavaf 2402339 2402338 2026-04-15T13:08:25Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw 2402339 wikitext text/x-wiki 24ze8j9q59u8cxhh0sjyouzsgv11vw4 2402340 2402339 2026-04-15T13:08:39Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2218 (T410589)', diff saved to https://phabricator.wikimedia.org/P90785 and previous config saved to /var/cache/conftool/dbconfig/20260415-130836-ladsgroup.json 2402340 wikitext text/x-wiki jfaazj228ogcpozaf12bv1zmlml5m6x 2402341 2402340 2026-04-15T13:08:43Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance 2402341 wikitext text/x-wiki cv6opztkhncnkryu6ymlravt7ykh5c2 2402342 2402341 2026-04-15T13:08:52Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2221 (T410589)', diff saved to https://phabricator.wikimedia.org/P90786 and previous config saved to /var/cache/conftool/dbconfig/20260415-130849-ladsgroup.json 2402342 wikitext text/x-wiki kljosy18w5fgj1smvitp02macvflv8u 2402343 2402342 2026-04-15T13:12:48Z Stashbot 7414 kartik@deploy1003: sbisson, kartik: Continuing with sync 2402343 wikitext text/x-wiki 4g8s9iedns2cuitfe9z3yllbwg3k46p 2402347 2402343 2026-04-15T13:16:40Z Stashbot 7414 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1270991|Register ArticleGuidance extension and enable in labs (T423295)]] (duration: 12m 02s) 2402347 wikitext text/x-wiki oqjtpi5e88ipsa8oxh3049usjfuq9z6 2402348 2402347 2026-04-15T13:16:59Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T419961)', diff saved to https://phabricator.wikimedia.org/P90787 and previous config saved to /var/cache/conftool/dbconfig/20260415-131657-fceratto.json 2402348 wikitext text/x-wiki lz6a1xqd7peh9c2bszqe4b1re7oopkp 2402349 2402348 2026-04-15T13:17:23Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance 2402349 wikitext text/x-wiki qdgxoj8buajivik9gnk5yvl3chhtjx1 2402350 2402349 2026-04-15T13:17:47Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply 2402350 wikitext text/x-wiki h9slyndxqaoup48itemjqytyig3xp5o 2402351 2402350 2026-04-15T13:18:04Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply 2402351 wikitext text/x-wiki 6fb0byinmgbdhxro36aedyvx9bz11z8 2402352 2402351 2026-04-15T13:19:17Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2068.codfw.wmnet with reason: host reimage 2402352 wikitext text/x-wiki 41kkcq1ausj73t92rjkgd3ml0rvf33i 2402353 2402352 2026-04-15T13:21:34Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.remove-downtime for aqs1026.eqiad.wmnet 2402353 wikitext text/x-wiki n16ub7xg134gx5xfzbe1xpmcfu45cju 2402354 2402353 2026-04-15T13:21:36Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for aqs1026.eqiad.wmnet 2402354 wikitext text/x-wiki 2didj3wvtl4ty250w7gz6iuxllwnw1a 2402355 2402354 2026-04-15T13:25:03Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2068.codfw.wmnet with reason: host reimage 2402355 wikitext text/x-wiki qn1405vn2w44pkhwtqclbze0agx66jz 2402356 2402355 2026-04-15T13:28:30Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2006.codfw.wmnet 2402356 wikitext text/x-wiki 1s80osgg5j2t1i8dwxca7ne3s7ns4ww 2402357 2402356 2026-04-15T13:29:04Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2006.codfw.wmnet 2402357 wikitext text/x-wiki 3isgvnvh7axx5sngjt3bm4qvysu7esv 2402358 2402357 2026-04-15T13:34:33Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2006.codfw.wmnet 2402358 wikitext text/x-wiki 719cudjiq8mewm7zw02qfb4hrt2036i 2402359 2402358 2026-04-15T13:34:35Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2006.codfw.wmnet 2402359 wikitext text/x-wiki 9t8zqk23qx1pp06g3hmgyf2x3qegpw2 2402360 2402359 2026-04-15T13:34:41Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2007.codfw.wmnet 2402360 wikitext text/x-wiki r1p1s0usfrg94p1kzzyry66zwj83243 2402361 2402360 2026-04-15T13:39:43Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2007.codfw.wmnet 2402361 wikitext text/x-wiki a19v5f2z80cso5i3m2pzhml5ucf9fph 2402362 2402361 2026-04-15T13:44:24Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2068.codfw.wmnet with OS bullseye 2402362 wikitext text/x-wiki bzpnl1d1hc15zn6tfxcm9fg14ghh2xu 2402363 2402362 2026-04-15T13:44:53Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2007.codfw.wmnet 2402363 wikitext text/x-wiki 2p6vo7ao9aahr6xiquffbd28w070cof 2402364 2402363 2026-04-15T13:44:55Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2007.codfw.wmnet 2402364 wikitext text/x-wiki cu1ld5297vlljlvdtbnzh7z4fj843wz 2402365 2402364 2026-04-15T13:45:01Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2008.codfw.wmnet 2402365 wikitext text/x-wiki 16vt2tstrnoja9f26r2tn1gflogsg98 2402366 2402365 2026-04-15T13:45:36Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2008.codfw.wmnet 2402366 wikitext text/x-wiki cnxiu3d60n3vtwxeuvs4u618q61hzri 2402367 2402366 2026-04-15T13:46:58Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance 2402367 wikitext text/x-wiki jtexca738b7idsh36gv204cihgg665h 2402368 2402367 2026-04-15T13:47:06Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2157 (T419961)', diff saved to https://phabricator.wikimedia.org/P90788 and previous config saved to /var/cache/conftool/dbconfig/20260415-134704-fceratto.json 2402368 wikitext text/x-wiki bzeqmfiz0dv2n5oc12eq9ttv5cf5hu2 2402370 2402368 2026-04-15T13:50:46Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2008.codfw.wmnet 2402370 wikitext text/x-wiki mting0dcanj2gh8pfgxpfl9dl1vbf0f 2402371 2402370 2026-04-15T13:50:48Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2008.codfw.wmnet 2402371 wikitext text/x-wiki jh9pqofc5xd76lk8i25tjwz82kjxj7o 2402372 2402371 2026-04-15T13:50:53Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2009.codfw.wmnet 2402372 wikitext text/x-wiki 52ynf6kf0178z8pct2xa1fkvekj2e2t 2402373 2402372 2026-04-15T13:51:28Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2009.codfw.wmnet 2402373 wikitext text/x-wiki cxloi0zdcwztjiawt1nub2978g229ak 2402374 2402373 2026-04-15T13:51:51Z Stashbot 7414 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1271206|lbwiki: Set minimum requirement of 10 edits for wgAutoConfirmCount (T423102)]] 2402374 wikitext text/x-wiki 38rww2y32b87csrp0f3osg2cahn90ja 2402376 2402374 2026-04-15T13:53:42Z Stashbot 7414 samtar@deploy1003: samtar, codenamenoreste: Backport for [[gerrit:1271206|lbwiki: Set minimum requirement of 10 edits for wgAutoConfirmCount (T423102)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2402376 wikitext text/x-wiki 7znti3t9kkubp2g1rn6oq5hl309nxl0 2402377 2402376 2026-04-15T13:55:20Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T419961)', diff saved to https://phabricator.wikimedia.org/P90789 and previous config saved to /var/cache/conftool/dbconfig/20260415-135519-fceratto.json 2402377 wikitext text/x-wiki 7gpclsvfrhbmpxvrd5uyt6a1155ipv0 2402378 2402377 2026-04-15T13:55:37Z Stashbot 7414 samtar@deploy1003: samtar, codenamenoreste: Continuing with sync 2402378 wikitext text/x-wiki hsdeqjj4b89uobd4cepr9wzjvixsef6 2402379 2402378 2026-04-15T13:56:39Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2009.codfw.wmnet 2402379 wikitext text/x-wiki 70dh288gkpiqba7hxyua8steuzmpj9w 2402380 2402379 2026-04-15T13:56:41Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2009.codfw.wmnet 2402380 wikitext text/x-wiki 9vlakmjjwd88c6lcfw4vmedhs35m2un 2402381 2402380 2026-04-15T13:56:43Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw 2402381 wikitext text/x-wiki btphrtoxpgodrujuvw42oqoa8vi2x8f 2402382 2402381 2026-04-15T14:04:39Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2402382 wikitext text/x-wiki 8asjohf4butdwlh7fpcoba9w117mz96 2402383 2402382 2026-04-15T14:04:43Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2402383 wikitext text/x-wiki 0wp7rq56yfjq3fb9ox18mxgp7re8cip 2402384 2402383 2026-04-15T14:05:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P90790 and previous config saved to /var/cache/conftool/dbconfig/20260415-140527-fceratto.json 2402384 wikitext text/x-wiki 3eqenoapav12xhoiolw71zsaegd8gvk 2402385 2402384 2026-04-15T14:06:04Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2402385 wikitext text/x-wiki nfnkwn16e9rqi7vtvec4p6xto6yym8a 2402386 2402385 2026-04-15T14:06:48Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2402386 wikitext text/x-wiki m2u5wrzhjie1wru1hs432umcjr8am2u 2402387 2402386 2026-04-15T14:06:50Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2402387 wikitext text/x-wiki 7htvo9w29ies60jezh6wiw5o1gdplfs 2402389 2402387 2026-04-15T14:15:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P90792 and previous config saved to /var/cache/conftool/dbconfig/20260415-141535-fceratto.json 2402389 wikitext text/x-wiki 84dwv0qs9hnbbu6woimfszr4joquozp 2402393 2402389 2026-04-15T14:25:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T419961)', diff saved to https://phabricator.wikimedia.org/P90794 and previous config saved to /var/cache/conftool/dbconfig/20260415-142543-fceratto.json 2402393 wikitext text/x-wiki cch2tyrhnal1i10vk91x98sycei2zjg 2402394 2402393 2026-04-15T14:26:08Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: Maintenance 2402394 wikitext text/x-wiki 2kb1440n5hnoqxf33424s8iaqeycgbs 2402395 2402394 2026-04-15T14:26:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2171 (T419961)', diff saved to https://phabricator.wikimedia.org/P90795 and previous config saved to /var/cache/conftool/dbconfig/20260415-142615-fceratto.json 2402395 wikitext text/x-wiki qwpp9jeuib96zjxvbamqbtjcvn0ksbl 2402397 2402395 2026-04-15T14:33:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T419961)', diff saved to https://phabricator.wikimedia.org/P90796 and previous config saved to /var/cache/conftool/dbconfig/20260415-143319-fceratto.json 2402397 wikitext text/x-wiki qvxyvnxyq6xgq21cqdfuuxge9w00y3n 2402398 2402397 2026-04-15T14:36:27Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply 2402398 wikitext text/x-wiki bfcymm06wi5i9jiuzr6vrnp60jdcqpl 2402399 2402398 2026-04-15T14:36:44Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply 2402399 wikitext text/x-wiki cytak8vbp3y2ls46cewpyauss670r95 2402400 2402399 2026-04-15T14:39:12Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2280.codfw.wmnet 2402400 wikitext text/x-wiki h89qrv08j1pmga52jpoani3w6wsrv90 2402401 2402400 2026-04-15T14:40:31Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2402401 wikitext text/x-wiki hblryziz2w3hqi19lwoue7ml9pbwmtw 2402402 2402401 2026-04-15T14:40:35Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2402402 wikitext text/x-wiki 1vsvxsut5nqvlhjb7g1vwro4h1k8vuv 2402403 2402402 2026-04-15T14:41:51Z Stashbot 7414 cgoubert@cumin1003: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker2280.codfw.wmnet 2402403 wikitext text/x-wiki hor31uhxuj3fnr2upuz3xem81jbbwqf 2402404 2402403 2026-04-15T14:42:01Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply 2402404 wikitext text/x-wiki qr5otthduspvsssaqwo6btw54fmca12 2402405 2402404 2026-04-15T14:42:29Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply 2402405 wikitext text/x-wiki 3todiqzt2pv5vvlyuphttfhy3kxpedu 2402406 2402405 2026-04-15T14:42:49Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply 2402406 wikitext text/x-wiki jmanouhtved2rwcn050q5bnjtd1a1xv 2402407 2402406 2026-04-15T14:43:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P90797 and previous config saved to /var/cache/conftool/dbconfig/20260415-144327-fceratto.json 2402407 wikitext text/x-wiki oy1gzvzuyt2pdbawf2m7foe1xii7h7i 2402408 2402407 2026-04-15T14:43:34Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply 2402408 wikitext text/x-wiki ac4dfjg64zz8qs0xnwu9w461a3qgic3 2402410 2402408 2026-04-15T14:45:58Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2280.codfw.wmnet 2402410 wikitext text/x-wiki dctdfvt09qnfdw5ssibgbzvpyf6dgd4 2402412 2402410 2026-04-15T14:46:02Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2280.codfw.wmnet 2402412 wikitext text/x-wiki t0lemo69u5edzysb85oxukqwjmuy7qy 2402414 2402412 2026-04-15T14:47:23Z Stashbot 7414 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1271748|Revert "lbwiki: Set minimum requirement of 10 edits for wgAutoConfirmCount"]] 2402414 wikitext text/x-wiki s9qaf4zxyk5i26tpdx1dhnnvtiad0lp 2402415 2402414 2026-04-15T14:49:19Z Stashbot 7414 samtar@deploy1003: samtar: Backport for [[gerrit:1271748|Revert "lbwiki: Set minimum requirement of 10 edits for wgAutoConfirmCount"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2402415 wikitext text/x-wiki f0p3p84cnnqxdjec9x0om7gsz5cxpmh 2402416 2402415 2026-04-15T14:49:42Z Stashbot 7414 samtar@deploy1003: samtar: Continuing with sync 2402416 wikitext text/x-wiki 2l32trifvvaw772j9rxl1v6nhdxn070 2402417 2402416 2026-04-15T14:51:57Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1273.eqiad.wmnet 2402417 wikitext text/x-wiki 7fzfn21xnh8m0ymypgue5sg0qgkhzyk 2402418 2402417 2026-04-15T14:52:09Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1273.eqiad.wmnet 2402418 wikitext text/x-wiki dpko0f9o70fs4x5fpujt3q1oqhn9hev 2402419 2402418 2026-04-15T14:53:35Z Stashbot 7414 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1271748|Revert "lbwiki: Set minimum requirement of 10 edits for wgAutoConfirmCount"]] (duration: 06m 12s) 2402419 wikitext text/x-wiki n78n41azolzhkpogyfw1trgxkzq4cgc 2402420 2402419 2026-04-15T14:53:38Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P90798 and previous config saved to /var/cache/conftool/dbconfig/20260415-145335-fceratto.json 2402420 wikitext text/x-wiki ssva9dxa5hx8cevpryfgfa5hbnubeiq 2402421 2402420 2026-04-15T14:55:20Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1012.eqiad.wmnet with reason: maintenance 2402421 wikitext text/x-wiki bnuonsfay97xxdv305k41nz5ojybbmp 2402423 2402421 2026-04-15T14:56:38Z Stashbot 7414 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. 2402423 wikitext text/x-wiki 8v5z0c4vdqagdj4ozb5owspzc780u5k 2402424 2402423 2026-04-15T14:56:48Z Stashbot 7414 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. 2402424 wikitext text/x-wiki hqaw80wymgzm7gihp64exd9v4h4bgz2 2402425 2402424 2026-04-15T14:57:10Z Stashbot 7414 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. 2402425 wikitext text/x-wiki 4kiweo8m7gt2q9hsevw73wwtkjiw9f0 2402426 2402425 2026-04-15T14:57:26Z Stashbot 7414 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. 2402426 wikitext text/x-wiki m2vd8y6u6pzhw8p9i8n3uy9mexylx54 2402427 2402426 2026-04-15T14:57:28Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2402427 wikitext text/x-wiki ffc21cyzb1sxe9gw72iv0cntkau7dwv 2402428 2402427 2026-04-15T14:59:12Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1190.eqiad.wmnet with reason: Maintenance 2402428 wikitext text/x-wiki 69u2t0yto18n201pnrtyvx5zr7ptz0z 2402429 2402428 2026-04-15T14:59:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1190 (T419635)', diff saved to https://phabricator.wikimedia.org/P90799 and previous config saved to /var/cache/conftool/dbconfig/20260415-145918-fceratto.json 2402429 wikitext text/x-wiki 0uyupobes9l0nvmrrqb1hu495tnrnsj 2402431 2402429 2026-04-15T15:00:00Z Stashbot 7414 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1271760|Revert^2 "lbwiki: Set minimum requirement of 10 edits for wgAutoConfirmCount"]] 2402431 wikitext text/x-wiki n876cqpsh85xc2epqowthiz32mkrumb 2402432 2402431 2026-04-15T15:01:33Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2402432 wikitext text/x-wiki 8gwo6lucrmi9tmb7h3cryt1fpx5jh9q 2402433 2402432 2026-04-15T15:01:37Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2402433 wikitext text/x-wiki cs4e3o9czhf2m8dw1q4gh0p08k3lehu 2402434 2402433 2026-04-15T15:02:20Z Stashbot 7414 samtar@deploy1003: samtar: Backport for [[gerrit:1271760|Revert^2 "lbwiki: Set minimum requirement of 10 edits for wgAutoConfirmCount"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2402434 wikitext text/x-wiki bkq3lug3qzn29jibqxho8vt0a1jeo0n 2402436 2402434 2026-04-15T15:02:58Z Stashbot 7414 samtar@deploy1003: samtar: Continuing with sync 2402436 wikitext text/x-wiki pwiz9cd6a8ljnno1i2pznjy4a7s0oo5 2402437 2402436 2026-04-15T15:03:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T419961)', diff saved to https://phabricator.wikimedia.org/P90800 and previous config saved to /var/cache/conftool/dbconfig/20260415-150344-fceratto.json 2402437 wikitext text/x-wiki gv3ethtjzvvmbtceieq841mar68qv6r 2402438 2402437 2026-04-15T15:04:08Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2178.codfw.wmnet with reason: Maintenance 2402438 wikitext text/x-wiki l8hrfcrf8tkbjpj6k0ibwjnse2j66fu 2402439 2402438 2026-04-15T15:04:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2178 (T419961)', diff saved to https://phabricator.wikimedia.org/P90801 and previous config saved to /var/cache/conftool/dbconfig/20260415-150415-fceratto.json 2402439 wikitext text/x-wiki mhvbqsw08a4dz0kh3f6s97fmr3i20nj 2402440 2402439 2026-04-15T15:06:55Z Stashbot 7414 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1271760|Revert^2 "lbwiki: Set minimum requirement of 10 edits for wgAutoConfirmCount"]] (duration: 06m 54s) 2402440 wikitext text/x-wiki m6zqs26icjaach6iqxb6wj8x9iktpwf 2402441 2402440 2026-04-15T15:08:04Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2402441 wikitext text/x-wiki dy33am3jck4b1rnx7zbi14egbjhfbzq 2402442 2402441 2026-04-15T15:11:16Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T419961)', diff saved to https://phabricator.wikimedia.org/P90802 and previous config saved to /var/cache/conftool/dbconfig/20260415-151114-fceratto.json 2402442 wikitext text/x-wiki 1fewof1kp6wes7mp6c1go3vroawj3sv 2402443 2402442 2026-04-15T15:11:34Z Stashbot 7414 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1271742|ProductionServices: remove poolcounter1006.eqiad (T420171)]] 2402443 wikitext text/x-wiki 469ja1od6uj0etynsy6knp5a7ajxvze 2402444 2402443 2026-04-15T15:12:01Z Stashbot 7414 moritzm: installing inetutils security updates 2402444 wikitext text/x-wiki nibhs1zfd7ih3svfrz77e6dua8ozjin 2402445 2402444 2026-04-15T15:13:27Z Stashbot 7414 blake@deploy1003: blake: Backport for [[gerrit:1271742|ProductionServices: remove poolcounter1006.eqiad (T420171)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2402445 wikitext text/x-wiki gcdbah9fr4n9zanxysujrxn2rill3v6 2402446 2402445 2026-04-15T15:14:45Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw 2402446 wikitext text/x-wiki ppwyyjw7xzwx124t75rpzysqwdnffsf 2402447 2402446 2026-04-15T15:14:47Z Stashbot 7414 blake@deploy1003: blake: Continuing with sync 2402447 wikitext text/x-wiki s4iniby80s197q9kmnkig5rphoctv5z 2402448 2402447 2026-04-15T15:18:33Z Stashbot 7414 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1271742|ProductionServices: remove poolcounter1006.eqiad (T420171)]] (duration: 06m 59s) 2402448 wikitext text/x-wiki ib0gugyshi55e4nsqzcutzimq0dlpyu 2402449 2402448 2026-04-15T15:18:58Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet 2402449 wikitext text/x-wiki 8dkdbz9jldonx5m7f2rxq6xk00kaboq 2402450 2402449 2026-04-15T15:19:03Z Stashbot 7414 moritzm: installing Dovecot security updates on mx-out* 2402450 wikitext text/x-wiki 76a0ha8m47wjvb4mhvezkzw1vselrse 2402451 2402450 2026-04-15T15:21:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P90803 and previous config saved to /var/cache/conftool/dbconfig/20260415-152122-fceratto.json 2402451 wikitext text/x-wiki cbu8ewuohmkc1on49114ccigqkacu2p 2402452 2402451 2026-04-15T15:22:55Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet 2402452 wikitext text/x-wiki s2xvqtfqvw5xm8n128ryez5j7peiz2m 2402454 2402452 2026-04-15T15:24:32Z Stashbot 7414 Emperor: update & restart envoy on apus frontends T423065 T382824 2402454 wikitext text/x-wiki d2p30sbr8pk1jp6dz1vpfum1fkdx62e 2402455 2402454 2026-04-15T15:24:50Z Stashbot 7414 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1271766|ProductionServices: remove poolcounter1007.eqiad, add 1006 (T420171)]] 2402455 wikitext text/x-wiki 72rghlubt9duwcprgaza0wkhn66uxyt 2402456 2402455 2026-04-15T15:24:56Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw 2402456 wikitext text/x-wiki ezxjh2z78f3ki8ryuhne6nyzql4l9u4 2402457 2402456 2026-04-15T15:26:40Z Stashbot 7414 Emperor: update & restart envoy on apus frontends T410975 T419637 2402457 wikitext text/x-wiki cjn6ckkqvu2rqqzbl0itmwni4xscu6u 2402458 2402457 2026-04-15T15:26:44Z Stashbot 7414 blake@deploy1003: blake: Backport for [[gerrit:1271766|ProductionServices: remove poolcounter1007.eqiad, add 1006 (T420171)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2402458 wikitext text/x-wiki qeh4ief9adcb5hv5e8veh1ejvtmgrvh 2402460 2402458 2026-04-15T15:27:23Z Stashbot 7414 blake@deploy1003: blake: Continuing with sync 2402460 wikitext text/x-wiki o9bvwtexwjgqw12yzejn6jf3kn32qho 2402461 2402460 2026-04-15T15:30:14Z Stashbot 7414 Emperor: update & restart envoy on thanos frontends T410975 T419637 2402461 wikitext text/x-wiki fmlx6xpsc5obc9jtejo714jgaiiew2d 2402462 2402461 2026-04-15T15:30:46Z Stashbot 7414 Emperor: update & restart envoy on ms swift frontends T410975 T419637 2402462 wikitext text/x-wiki 310epewjb1d4fvmmu2v913o3g38q1us 2402463 2402462 2026-04-15T15:31:09Z Stashbot 7414 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1271766|ProductionServices: remove poolcounter1007.eqiad, add 1006 (T420171)]] (duration: 06m 19s) 2402463 wikitext text/x-wiki k7r2lbz89wb0go9rngliufhy9lebsgs 2402464 2402463 2026-04-15T15:31:22Z Stashbot 7414 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet 2402464 wikitext text/x-wiki 8mr6grfalkbyqeaqc4jxeladwkeey4o 2402465 2402464 2026-04-15T15:31:32Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P90804 and previous config saved to /var/cache/conftool/dbconfig/20260415-153130-fceratto.json 2402465 wikitext text/x-wiki rpaw51itc365aq4qkqh2nb5q7id7k5p 2402467 2402465 2026-04-15T15:35:09Z Stashbot 7414 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet 2402467 wikitext text/x-wiki ebupaahu1qbxldcxcycx7q9trb1mkew 2402468 2402467 2026-04-15T15:37:15Z Stashbot 7414 blake@deploy1003: Started scap sync-world: Backport for [[gerrit:1271772|ProductionServices: re-add poolcounter1007.eqiad. (T420171)]] 2402468 wikitext text/x-wiki jwbbr69ohiwgzj4klc5d49o9bl27y30 2402469 2402468 2026-04-15T15:39:11Z Stashbot 7414 blake@deploy1003: blake: Backport for [[gerrit:1271772|ProductionServices: re-add poolcounter1007.eqiad. (T420171)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2402469 wikitext text/x-wiki pd8hjcpoc9bqbgw0rmhjbuqdic3fumy 2402470 2402469 2026-04-15T15:39:41Z Stashbot 7414 blake@deploy1003: blake: Continuing with sync 2402470 wikitext text/x-wiki rdv3hib74ctpws28bgqti6n5r0f2g9q 2402471 2402470 2026-04-15T15:41:42Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T419961)', diff saved to https://phabricator.wikimedia.org/P90805 and previous config saved to /var/cache/conftool/dbconfig/20260415-154138-fceratto.json 2402471 wikitext text/x-wiki kggfyjd0uozhbr837itkngih5msh0p0 2402472 2402471 2026-04-15T15:42:03Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2192.codfw.wmnet with reason: Maintenance 2402472 wikitext text/x-wiki qercbdiks4lxq8h0vop33giwwnoff54 2402473 2402472 2026-04-15T15:42:12Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2192 (T419961)', diff saved to https://phabricator.wikimedia.org/P90806 and previous config saved to /var/cache/conftool/dbconfig/20260415-154210-fceratto.json 2402473 wikitext text/x-wiki nyetshgx2blt3xfb82qwz118q3vxo9v 2402474 2402473 2026-04-15T15:43:24Z Stashbot 7414 blake@deploy1003: Finished scap sync-world: Backport for [[gerrit:1271772|ProductionServices: re-add poolcounter1007.eqiad. (T420171)]] (duration: 06m 09s) 2402474 wikitext text/x-wiki 3xxmc7tvng7h52oote9pvt73yab2vbt 2402475 2402474 2026-04-15T15:49:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T419961)', diff saved to https://phabricator.wikimedia.org/P90807 and previous config saved to /var/cache/conftool/dbconfig/20260415-154911-fceratto.json 2402475 wikitext text/x-wiki dy8p140o6oi3hqyfedh9k7re8iyi2uy 2402476 2402475 2026-04-15T15:56:34Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for backup1012.eqiad.wmnet 2402476 wikitext text/x-wiki dxtdwumtm8uw7lh78uzexyiftoz52bm 2402477 2402476 2026-04-15T15:56:36Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for backup1012.eqiad.wmnet 2402477 wikitext text/x-wiki m80z6crupvhr5f1m43jnowrahaynh8n 2402478 2402477 2026-04-15T15:59:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P90809 and previous config saved to /var/cache/conftool/dbconfig/20260415-155920-fceratto.json 2402478 wikitext text/x-wiki d5icdtu7vogu0cnm6wtk1h9p6jrbecx 2402479 2402478 2026-04-15T16:09:30Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P90810 and previous config saved to /var/cache/conftool/dbconfig/20260415-160928-fceratto.json 2402479 wikitext text/x-wiki 0ukir4hqogxyt413zxbrzb1pc7cdihr 2402480 2402479 2026-04-15T16:19:38Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T419961)', diff saved to https://phabricator.wikimedia.org/P90813 and previous config saved to /var/cache/conftool/dbconfig/20260415-161936-fceratto.json 2402480 wikitext text/x-wiki 4wgh50lhco62w926wz044oswkzu2cxv 2402481 2402480 2026-04-15T16:20:00Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2201.codfw.wmnet with reason: Maintenance 2402481 wikitext text/x-wiki e5dei3m956117v03fge1skd2qxh0ifh 2402482 2402481 2026-04-15T16:25:06Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2211.codfw.wmnet with reason: Maintenance 2402482 wikitext text/x-wiki qyjromn8m8nquxqynmuh30s1bf7uasr 2402483 2402482 2026-04-15T16:25:15Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2211 (T419961)', diff saved to https://phabricator.wikimedia.org/P90814 and previous config saved to /var/cache/conftool/dbconfig/20260415-162513-fceratto.json 2402483 wikitext text/x-wiki 3a071pvmxrxv1pvuwrgkh7akbui5ym4 2402484 2402483 2026-04-15T16:32:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T419961)', diff saved to https://phabricator.wikimedia.org/P90815 and previous config saved to /var/cache/conftool/dbconfig/20260415-163215-fceratto.json 2402484 wikitext text/x-wiki dkvt8o0o0zfbn6wxypm3z91omn27q9d 2402486 2402484 2026-04-15T16:42:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P90816 and previous config saved to /var/cache/conftool/dbconfig/20260415-164223-fceratto.json 2402486 wikitext text/x-wiki bu90m59girh804nklthozo3346em34w 2402487 2402486 2026-04-15T16:52:33Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P90817 and previous config saved to /var/cache/conftool/dbconfig/20260415-165231-fceratto.json 2402487 wikitext text/x-wiki 6ovw8dpf2oj4w4wan3eeayyr7uo7hot 2402488 2402487 2026-04-15T16:53:20Z Stashbot 7414 kamila@deploy1003: Started scap sync-world: Backport for [[gerrit:1270472|Revert "Temporarily add shellbox-icu to $wgShellboxUrls" (T422546)]], [[gerrit:1270470|Revert "Enable $wgTempCategoryCollations for testwiki." (T422546)]] 2402488 wikitext text/x-wiki iczfwdg1j3kyn554dsf9m596opy33lz 2402489 2402488 2026-04-15T16:55:13Z Stashbot 7414 kamila@deploy1003: kamila: Backport for [[gerrit:1270472|Revert "Temporarily add shellbox-icu to $wgShellboxUrls" (T422546)]], [[gerrit:1270470|Revert "Enable $wgTempCategoryCollations for testwiki." (T422546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2402489 wikitext text/x-wiki 24ild4iq7kaqcdot8xtxtkc3cb0rzvd 2402490 2402489 2026-04-15T17:02:41Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T419961)', diff saved to https://phabricator.wikimedia.org/P90818 and previous config saved to /var/cache/conftool/dbconfig/20260415-170239-fceratto.json 2402490 wikitext text/x-wiki kuhqmunkn1eb06az7tua2tq743ku1nz 2402491 2402490 2026-04-15T17:03:03Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2223.codfw.wmnet with reason: Maintenance 2402491 wikitext text/x-wiki cax0926t2r00j1pm63rm00cykl7jn07 2402492 2402491 2026-04-15T17:03:12Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2223 (T419961)', diff saved to https://phabricator.wikimedia.org/P90819 and previous config saved to /var/cache/conftool/dbconfig/20260415-170310-fceratto.json 2402492 wikitext text/x-wiki lvr4f1fe8e941bdcj6ky9tjbovmz4en 2402494 2402492 2026-04-15T17:05:05Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T419635)', diff saved to https://phabricator.wikimedia.org/P90820 and previous config saved to /var/cache/conftool/dbconfig/20260415-170501-fceratto.json 2402494 wikitext text/x-wiki oj9pgy62g5piw3c6932auxvtaangoqa 2402495 2402494 2026-04-15T17:05:42Z Stashbot 7414 kamila@deploy1003: kamila: Continuing with sync 2402495 wikitext text/x-wiki ptq4wqq3subt1jaeoxb8kpmw9rv4k14 2402496 2402495 2026-04-15T17:09:30Z Stashbot 7414 kamila@deploy1003: Finished scap sync-world: Backport for [[gerrit:1270472|Revert "Temporarily add shellbox-icu to $wgShellboxUrls" (T422546)]], [[gerrit:1270470|Revert "Enable $wgTempCategoryCollations for testwiki." (T422546)]] (duration: 16m 10s) 2402496 wikitext text/x-wiki s53bpnjcfvkseyxqxf93lgjabss9jqh 2402497 2402496 2026-04-15T17:10:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 (T419961)', diff saved to https://phabricator.wikimedia.org/P90821 and previous config saved to /var/cache/conftool/dbconfig/20260415-171011-fceratto.json 2402497 wikitext text/x-wiki 2y49oh7n7r6ydrdscmzi2whytqfxofb 2402498 2402497 2026-04-15T17:11:50Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1236 (T410589)', diff saved to https://phabricator.wikimedia.org/P90822 and previous config saved to /var/cache/conftool/dbconfig/20260415-171147-ladsgroup.json 2402498 wikitext text/x-wiki kzpu6q246wp49kz97tlkllwlwm7qo2g 2402499 2402498 2026-04-15T17:12:27Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1115.eqiad.wmnet with OS trixie 2402499 wikitext text/x-wiki 914cqrj5n9k0oruxp80e52ty089pwvu 2402500 2402499 2026-04-15T17:15:11Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P90823 and previous config saved to /var/cache/conftool/dbconfig/20260415-171509-fceratto.json 2402500 wikitext text/x-wiki ru8093fapifudj709twrqdcfmpndsah 2402501 2402500 2026-04-15T17:20:20Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P90824 and previous config saved to /var/cache/conftool/dbconfig/20260415-172019-fceratto.json 2402501 wikitext text/x-wiki 5s4oe3h2trmed4tip5tyhzjn8hikomi 2402502 2402501 2026-04-15T17:21:57Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P90825 and previous config saved to /var/cache/conftool/dbconfig/20260415-172155-ladsgroup.json 2402502 wikitext text/x-wiki kb56y06q799ojs2mrtas8mpr78gm65k 2402503 2402502 2026-04-15T17:25:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P90826 and previous config saved to /var/cache/conftool/dbconfig/20260415-172517-fceratto.json 2402503 wikitext text/x-wiki lahqbys2rcwtp21xeqks86jth449o5j 2402504 2402503 2026-04-15T17:29:00Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1115.eqiad.wmnet with reason: host reimage 2402504 wikitext text/x-wiki 9li6k7gr0b8w4hdr47ux0eenwkvgjex 2402505 2402504 2026-04-15T17:30:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P90827 and previous config saved to /var/cache/conftool/dbconfig/20260415-173027-fceratto.json 2402505 wikitext text/x-wiki 27lx4ive2gt6etcwx5kqruitf7vsggx 2402506 2402505 2026-04-15T17:32:05Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P90828 and previous config saved to /var/cache/conftool/dbconfig/20260415-173203-ladsgroup.json 2402506 wikitext text/x-wiki 9xuh8rreh1oj56qk7vrt3o5lkjkshmv 2402508 2402506 2026-04-15T17:34:52Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1115.eqiad.wmnet with reason: host reimage 2402508 wikitext text/x-wiki qt4cfki0jv4gm0m82f0f7774qif5jc9 2402509 2402508 2026-04-15T17:35:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T419635)', diff saved to https://phabricator.wikimedia.org/P90829 and previous config saved to /var/cache/conftool/dbconfig/20260415-173525-fceratto.json 2402509 wikitext text/x-wiki aml6jut3n7p9iligzogjv01f2e9kmce 2402510 2402509 2026-04-15T17:35:55Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1199.eqiad.wmnet with reason: Maintenance 2402510 wikitext text/x-wiki qokyjc0bj5sombhwxnfeqhpumlhch76 2402511 2402510 2026-04-15T17:36:05Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1199 (T419635)', diff saved to https://phabricator.wikimedia.org/P90830 and previous config saved to /var/cache/conftool/dbconfig/20260415-173602-fceratto.json 2402511 wikitext text/x-wiki g0syf91pbqpsxufcvju90lymaxukxu9 2402512 2402511 2026-04-15T17:36:29Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/data-gateway: apply 2402512 wikitext text/x-wiki qy4wnqhjpikedxmwatnk5isux6wk7bw 2402513 2402512 2026-04-15T17:36:39Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply 2402513 wikitext text/x-wiki c5tmjbfy0b9dz4zibzewmfik8n6kxnx 2402514 2402513 2026-04-15T17:37:09Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/device-analytics: apply 2402514 wikitext text/x-wiki pzin9i1wrnykge2a53lnyr4pinhk4vo 2402515 2402514 2026-04-15T17:37:17Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply 2402515 wikitext text/x-wiki bljgm23ldn396fm8e72biu3tykbqnid 2402516 2402515 2026-04-15T17:37:48Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/edit-analytics: apply 2402516 wikitext text/x-wiki lg5z4sqam5gg9roz0jjq8xh5xn1y3fl 2402517 2402516 2026-04-15T17:37:56Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply 2402517 wikitext text/x-wiki ktmzalr34wo11vidz06ylkt5yip8vtg 2402518 2402517 2026-04-15T17:38:27Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/editor-analytics: apply 2402518 wikitext text/x-wiki rvxw938msk3a86rvw0cseawlla1wtzh 2402519 2402518 2026-04-15T17:38:35Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply 2402519 wikitext text/x-wiki b5b198vdsf5x6g4scl4re0iyw1w9hjq 2402521 2402519 2026-04-15T17:39:11Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/geo-analytics: apply 2402521 wikitext text/x-wiki s29xxouplgxnh1id15ez4l27on957tk 2402522 2402521 2026-04-15T17:39:17Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply 2402522 wikitext text/x-wiki dhtfboia5zzolr1ftlghflyfkj5i0l8 2402523 2402522 2026-04-15T17:39:48Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/media-analytics: apply 2402523 wikitext text/x-wiki lbxvrxuhik9qqhy2w90h4yljxjy53yi 2402524 2402523 2026-04-15T17:40:01Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/media-analytics: apply 2402524 wikitext text/x-wiki nmkoqk0fnr8ss1cf2x3791ynzykmeaa 2402525 2402524 2026-04-15T17:40:31Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/page-analytics: apply 2402525 wikitext text/x-wiki dqdfbu8rwsp8s9xuzmyj8do9g3gqim4 2402526 2402525 2026-04-15T17:40:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 (T419961)', diff saved to https://phabricator.wikimedia.org/P90831 and previous config saved to /var/cache/conftool/dbconfig/20260415-174035-fceratto.json 2402526 wikitext text/x-wiki cdtxvinp63vrlq6ohclcdj7p1ydcl5j 2402527 2402526 2026-04-15T17:40:46Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/page-analytics: apply 2402527 wikitext text/x-wiki 8sqxm66dkkbdiz6jbmgvdp2qffmj7ud 2402528 2402527 2026-04-15T17:41:00Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance 2402528 wikitext text/x-wiki bith8x79b0jedm7gy2n4gj5ht3q7kxu 2402529 2402528 2026-04-15T17:41:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2228 (T419961)', diff saved to https://phabricator.wikimedia.org/P90832 and previous config saved to /var/cache/conftool/dbconfig/20260415-174107-fceratto.json 2402529 wikitext text/x-wiki 5fcdxh2omfot5r6m0febksd868lhcfz 2402530 2402529 2026-04-15T17:42:14Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1236 (T410589)', diff saved to https://phabricator.wikimedia.org/P90833 and previous config saved to /var/cache/conftool/dbconfig/20260415-174212-ladsgroup.json 2402530 wikitext text/x-wiki 0l8dk3ms6xmmdjx2ik602hpszyo2m8f 2402531 2402530 2026-04-15T17:42:30Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance 2402531 wikitext text/x-wiki 28bbw63lwnarsmji8r362yjndljqwlj 2402532 2402531 2026-04-15T17:42:39Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db1253 (T410589)', diff saved to https://phabricator.wikimedia.org/P90834 and previous config saved to /var/cache/conftool/dbconfig/20260415-174236-ladsgroup.json 2402532 wikitext text/x-wiki l9bt8rr8xgcx7yim0z3j6rwof6btng9 2402533 2402532 2026-04-15T17:43:04Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/data-gateway: apply 2402533 wikitext text/x-wiki ldbj21q8fetz6jfky3zsxl6xurpmwd1 2402534 2402533 2026-04-15T17:43:23Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/data-gateway: apply 2402534 wikitext text/x-wiki m7t3m5jr2z35pbf5ce13hl4bulqrnya 2402535 2402534 2026-04-15T17:43:54Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/device-analytics: apply 2402535 wikitext text/x-wiki bxsndv3wxl085ddwpb4cxd9pt8pslb6 2402536 2402535 2026-04-15T17:44:10Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply 2402536 wikitext text/x-wiki 15jfai8m7lbl5rblqf4abi9q1flct2q 2402538 2402536 2026-04-15T17:44:41Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply 2402538 wikitext text/x-wiki bbceq43zh7bc8qr1eaayh3c9ukfgmct 2402539 2402538 2026-04-15T17:44:57Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply 2402539 wikitext text/x-wiki 04bh8jinsmnyfry7rg91b7iebg0rkah 2402540 2402539 2026-04-15T17:45:29Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/editor-analytics: apply 2402540 wikitext text/x-wiki q5vt6skga5kmhb2yik0y5bostn0cnms 2402541 2402540 2026-04-15T17:45:43Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply 2402541 wikitext text/x-wiki lv2u1qze0sc84iviuuchmx42ziradri 2402542 2402541 2026-04-15T17:46:14Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/geo-analytics: apply 2402542 wikitext text/x-wiki 81q1vlkpfggwgf434fmmdfu7sbajf5i 2402543 2402542 2026-04-15T17:46:28Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply 2402543 wikitext text/x-wiki 65kez1zhofrrfrwp25amulmkvyox85m 2402545 2402543 2026-04-15T17:46:59Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/media-analytics: apply 2402545 wikitext text/x-wiki 1hxtdfeoumxd7c3m6m99n5eyr79ttiy 2402547 2402545 2026-04-15T17:47:14Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply 2402547 wikitext text/x-wiki 67mzwo9aywqu8ai1ih9e0cxc8t8052r 2402549 2402547 2026-04-15T17:47:45Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/page-analytics: apply 2402549 wikitext text/x-wiki pejy4rux5rwp6zhvrsvk3ue5ty5cwwj 2402551 2402549 2026-04-15T17:48:03Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/page-analytics: apply 2402551 wikitext text/x-wiki e8kc2lyys6hnxesvmek80nn603wvxjz 2402552 2402551 2026-04-15T17:48:10Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 (T419961)', diff saved to https://phabricator.wikimedia.org/P90835 and previous config saved to /var/cache/conftool/dbconfig/20260415-174808-fceratto.json 2402552 wikitext text/x-wiki awov4cbbh57qvhxlmzcromxne1dxa03 2402553 2402552 2026-04-15T17:53:44Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/data-gateway: apply 2402553 wikitext text/x-wiki nmlaauxpq1yn0inpmog1l8xl6awwmp8 2402554 2402553 2026-04-15T17:53:58Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply 2402554 wikitext text/x-wiki ka705w5kww8nxuqnefh6e4ry5xn229z 2402555 2402554 2026-04-15T17:54:29Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply 2402555 wikitext text/x-wiki f0poor0cbo7t5s7mgj59hzxfff31ie5 2402556 2402555 2026-04-15T17:54:44Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply 2402556 wikitext text/x-wiki 4spe04in0db0nx6qhekteaavv1vlidj 2402557 2402556 2026-04-15T17:55:16Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply 2402557 wikitext text/x-wiki b0sdr5ielrx24xc3trbm45yko5eqj5n 2402558 2402557 2026-04-15T17:55:29Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply 2402558 wikitext text/x-wiki e1zhh8hvtr0o01pksam1vm3140ou4d1 2402559 2402558 2026-04-15T17:56:02Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply 2402559 wikitext text/x-wiki 7y5a8x2s3c60afbxt0kngefb7wqzkgc 2402560 2402559 2026-04-15T17:56:13Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply 2402560 wikitext text/x-wiki pxvewaswsvzbwi17yo8m2hdkpk45el1 2402561 2402560 2026-04-15T17:56:45Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply 2402561 wikitext text/x-wiki bpg1s9za2n2raog14h629nifw4gqtmc 2402562 2402561 2026-04-15T17:56:58Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply 2402562 wikitext text/x-wiki ch0po1in01v12cobbpfylerd0805w0m 2402563 2402562 2026-04-15T17:57:09Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - brett@cumin2002" 2402563 wikitext text/x-wiki lq90woqo4fqv4nbdawbff2co23x8gro 2402564 2402563 2026-04-15T17:57:29Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/media-analytics: apply 2402564 wikitext text/x-wiki 9312bbx8htgu3c5z0k02q4jjx33k7hp 2402565 2402564 2026-04-15T17:57:43Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply 2402565 wikitext text/x-wiki 622lde6qsa7rmkv48l6er13uzxdt2u7 2402566 2402565 2026-04-15T17:58:14Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/page-analytics: apply 2402566 wikitext text/x-wiki 3ujl8x4lant5722ozuvp5qzbeac9qx4 2402567 2402566 2026-04-15T17:58:18Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P90836 and previous config saved to /var/cache/conftool/dbconfig/20260415-175817-fceratto.json 2402567 wikitext text/x-wiki kbevu6393nrj5nuqtf2f98wlfajzyqw 2402568 2402567 2026-04-15T17:58:29Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply 2402568 wikitext text/x-wiki isrn9rgg559sogdtjpncgjg78pmvma5 2402571 2402568 2026-04-15T18:01:28Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - brett@cumin2002" 2402571 wikitext text/x-wiki 55nnmrgsmzpvuu8rv0f7gqzvcz285f4 2402572 2402571 2026-04-15T18:01:30Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1115.eqiad.wmnet with OS trixie 2402572 wikitext text/x-wiki pf9laggifgacgvoyqbom61br7rme6pb 2402573 2402572 2026-04-15T18:08:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P90837 and previous config saved to /var/cache/conftool/dbconfig/20260415-180825-fceratto.json 2402573 wikitext text/x-wiki dt67s8ezub3uzfnxlhkg4t2ujtwko1j 2402574 2402573 2026-04-15T18:08:48Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1115.* 2402574 wikitext text/x-wiki qx2tvasx2g779zjlmi2gagbl3g4xr9k 2402575 2402574 2026-04-15T18:15:15Z Stashbot 7414 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.46.0-wmf.24 refs T420482 2402575 wikitext text/x-wiki dy75wzeicokajxl4822pmbkxjf4ns00 2402576 2402575 2026-04-15T18:18:35Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 (T419961)', diff saved to https://phabricator.wikimedia.org/P90839 and previous config saved to /var/cache/conftool/dbconfig/20260415-181833-fceratto.json 2402576 wikitext text/x-wiki 4x0isw6yxalgkwf9gexgofxrkeeailv 2402577 2402576 2026-04-15T18:27:01Z Stashbot 7414 eevans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on aqs1027.eqiad.wmnet with reason: Bootstrapping — T412830 2402577 wikitext text/x-wiki 36keamhtu7enmd4ot288o1g3ubgx0c4 2402578 2402577 2026-04-15T18:43:24Z Stashbot 7414 dduvall: rolling back due to steady `Term with languageCode "en" not found` errors (cc T420482) 2402578 wikitext text/x-wiki p21v6kjc4gmet14q5ztsdt331q6a5gr 2402580 2402578 2026-04-15T18:50:32Z Stashbot 7414 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.46.0-wmf.24 refs T420482 2402580 wikitext text/x-wiki rtr3x0oczl0s37g1y7ic9qdh7o7t7na 2402581 2402580 2026-04-15T19:37:02Z Stashbot 7414 topranks: add GRE tunnel to cr1-drmrs towards cr2-eqiad 2402581 wikitext text/x-wiki 4lbdm8g4k9j1b6fkajexirxvg1ifqih 2402582 2402581 2026-04-15T19:38:07Z Stashbot 7414 topranks: add GRE tunnel to cr2-eqiad towards cr1-drmrs 2402582 wikitext text/x-wiki hiyoqymcxrhhatwkosgbj8oewi1j5lm 2402584 2402582 2026-04-15T19:45:51Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T419635)', diff saved to https://phabricator.wikimedia.org/P90840 and previous config saved to /var/cache/conftool/dbconfig/20260415-194548-fceratto.json 2402584 wikitext text/x-wiki tvnsz1mnfjktdm8fzs5b7jsz5rfycaf 2402592 2402584 2026-04-15T19:55:15Z Stashbot 7414 topranks: add static routes on cr1-drmrs and cr2-eqiad for arelion GRE far-side IPv4 addresses 2402592 wikitext text/x-wiki fo5su5l1ir4c14lhxzxw5ttm0ueilx6 2402593 2402592 2026-04-15T19:55:44Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2402593 wikitext text/x-wiki 6uzw3e7bmd6vf2kewz8vmet9oadk9rk 2402594 2402593 2026-04-15T19:55:58Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P90841 and previous config saved to /var/cache/conftool/dbconfig/20260415-195556-fceratto.json 2402594 wikitext text/x-wiki c55k38s737vvdhfctgvtkqrkkrsagk9 2402596 2402594 2026-04-15T20:00:42Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add ipv4 dns names for eqiad-drmrs gre tunnel - cmooney@cumin1003" 2402596 wikitext text/x-wiki shfy02uutfuo5c1c5lsn4j30qe29zy5 2402598 2402596 2026-04-15T20:05:01Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add ipv4 dns names for eqiad-drmrs gre tunnel - cmooney@cumin1003" 2402598 wikitext text/x-wiki oqfdj2ynfhruj27aoav382ap6sabhzo 2402599 2402598 2026-04-15T20:05:03Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2402599 wikitext text/x-wiki p4jvrctphjkvihfm3g1cagxieyzck0y 2402601 2402599 2026-04-15T20:06:06Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P90842 and previous config saved to /var/cache/conftool/dbconfig/20260415-200605-fceratto.json 2402601 wikitext text/x-wiki 9kfuizeuv9q4fr061aol71b77igca76 2402602 2402601 2026-04-15T20:16:16Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T419635)', diff saved to https://phabricator.wikimedia.org/P90843 and previous config saved to /var/cache/conftool/dbconfig/20260415-201613-fceratto.json 2402602 wikitext text/x-wiki jzztrvp8kzu67k0kew8wbg8hnsgspsi 2402603 2402602 2026-04-15T20:16:33Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1221.eqiad.wmnet with reason: Maintenance 2402603 wikitext text/x-wiki aqqwkyuzryr4oal6ratyeeh7viaamo7 2402604 2402603 2026-04-15T20:16:54Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1024-1025].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance 2402604 wikitext text/x-wiki qqfjh3kojig6x01etqzom0r90eq1c2j 2402605 2402604 2026-04-15T20:17:03Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1221 (T419635)', diff saved to https://phabricator.wikimedia.org/P90844 and previous config saved to /var/cache/conftool/dbconfig/20260415-201700-fceratto.json 2402605 wikitext text/x-wiki lqac0jrsevkes6ocyyqq68z7ldwj3ss 2402606 2402605 2026-04-15T20:19:54Z Stashbot 7414 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply 2402606 wikitext text/x-wiki 73ntepo4jzmoufn1p0dsvghjizwppgn 2402607 2402606 2026-04-15T20:20:09Z Stashbot 7414 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply 2402607 wikitext text/x-wiki bliuaedredf6s7nfci04hau04e094j2 2402608 2402607 2026-04-15T20:23:14Z Stashbot 7414 mstyles@deploy1003: Started scap sync-world: Backport for [[gerrit:1271886|Force Reauth (T419621)]], [[gerrit:1271830|Rename Test Kitchen Experiment (T420007)]], [[gerrit:1271831|Rename Test Kitchen Experiment (T420007)]], [[gerrit:1271833|Rename Test Kitchen Experiment (T420007)]], [[gerrit:1271832|Rename Test Kitchen Experiment (T420007)]] 2402608 wikitext text/x-wiki smjzlxm4uy10bfytds9xj28ojsn1qao 2402609 2402608 2026-04-15T20:25:12Z Stashbot 7414 mstyles@deploy1003: mstyles: Backport for [[gerrit:1271886|Force Reauth (T419621)]], [[gerrit:1271830|Rename Test Kitchen Experiment (T420007)]], [[gerrit:1271831|Rename Test Kitchen Experiment (T420007)]], [[gerrit:1271833|Rename Test Kitchen Experiment (T420007)]], [[gerrit:1271832|Rename Test Kitchen Experiment (T420007)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now 2402609 wikitext text/x-wiki 734iep5ar7ddzmpe80r7n7dis1pcnnt 2402610 2402609 2026-04-15T20:26:36Z Stashbot 7414 topranks: enable ospf on GRE cr1-drmrs <-> cr2-eqiad 2402610 wikitext text/x-wiki rv599q4s3ctveu9gmawfacpmdezelvv 2402611 2402610 2026-04-15T20:27:09Z Stashbot 7414 mstyles@deploy1003: mstyles: Continuing with sync 2402611 wikitext text/x-wiki cn9k9eoyb8c9871ur3cfld0w48z8wor 2402612 2402611 2026-04-15T20:30:28Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2402612 wikitext text/x-wiki hw63r5c8i38n6plwkb0ie7e9bx11ua9 2402613 2402612 2026-04-15T20:31:02Z Stashbot 7414 mstyles@deploy1003: Finished scap sync-world: Backport for [[gerrit:1271886|Force Reauth (T419621)]], [[gerrit:1271830|Rename Test Kitchen Experiment (T420007)]], [[gerrit:1271831|Rename Test Kitchen Experiment (T420007)]], [[gerrit:1271833|Rename Test Kitchen Experiment (T420007)]], [[gerrit:1271832|Rename Test Kitchen Experiment (T420007)]] (duration: 07m 48s) 2402613 wikitext text/x-wiki 5of45p0yxwsl84nca3lsdkr04alml8d 2402614 2402613 2026-04-15T20:33:33Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: generate v6 reverse records for 2a02:ec80:600:fe0a::1/64 - cmooney@cumin1003" 2402614 wikitext text/x-wiki lajvffnjrqm8qg5o8pkkpciv30ir68a 2402615 2402614 2026-04-15T20:34:04Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: generate v6 reverse records for 2a02:ec80:600:fe0a::1/64 - cmooney@cumin1003" 2402615 wikitext text/x-wiki 71y7bkt9b4hgax2i4yrhtrzznafshra 2402616 2402615 2026-04-15T20:34:06Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2402616 wikitext text/x-wiki kpplzikce98h38wtuo8r06amzx13t6w 2402618 2402616 2026-04-15T20:35:27Z Stashbot 7414 cmooney@dns2005: START - running authdns-update 2402618 wikitext text/x-wiki trktj1anrpwdp80lmr3gz31v9mzqwuh 2402619 2402618 2026-04-15T20:36:37Z Stashbot 7414 cmooney@dns2005: END - running authdns-update 2402619 wikitext text/x-wiki p0aqenqad1xmjt6pb9cpgbgria40axv 2402621 2402619 2026-04-15T20:37:05Z Stashbot 7414 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1268300|Drop 1.5x logos (T246054)]], [[gerrit:1271839|Enwikinews: disable lingering FlaggedRevs template processing (T423512)]], [[gerrit:1270583|Record file usage from TemplateStyles pages (T413707)]] 2402621 wikitext text/x-wiki 6ybt17ar1nvxfjjb105cgfer7pwtlv9 2402622 2402621 2026-04-15T20:38:57Z Stashbot 7414 jforrester@deploy1003: jforrester, bawolff, pppery: Backport for [[gerrit:1268300|Drop 1.5x logos (T246054)]], [[gerrit:1271839|Enwikinews: disable lingering FlaggedRevs template processing (T423512)]], [[gerrit:1270583|Record file usage from TemplateStyles pages (T413707)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2402622 wikitext text/x-wiki pz79zco5xu2474texzw660d8maraons 2402623 2402622 2026-04-15T20:42:23Z Stashbot 7414 topranks: enable BGP over GRE between cr1-drmrs and cr2-eqiad 2402623 wikitext text/x-wiki stn0bsdgjve8x4aqqey2d57v1g4dmd4 2402624 2402623 2026-04-15T20:42:28Z Stashbot 7414 jforrester@deploy1003: jforrester, bawolff, pppery: Continuing with sync 2402624 wikitext text/x-wiki khbkrshmziiullq0j7oz6rhswjfxgjf 2402625 2402624 2026-04-15T20:46:22Z Stashbot 7414 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1268300|Drop 1.5x logos (T246054)]], [[gerrit:1271839|Enwikinews: disable lingering FlaggedRevs template processing (T423512)]], [[gerrit:1270583|Record file usage from TemplateStyles pages (T413707)]] (duration: 09m 15s) 2402625 wikitext text/x-wiki 1hkl0clh7kxih7x3zm5on7etinraco6 2402629 2402625 2026-04-15T20:52:05Z Stashbot 7414 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1271894|PageRenderingHandler: Handle Wikibase's OutOfBoundsException for "we don't have a label" (T423514)]], [[gerrit:1271905|PageRenderingHandler: Don't run repo-mode stuff in non-repo world (T423515)]] 2402629 wikitext text/x-wiki 50m6vili7fd50oxcczja45x4k4lo8xk 2402630 2402629 2026-04-15T20:54:01Z Stashbot 7414 jforrester@deploy1003: jforrester: Backport for [[gerrit:1271894|PageRenderingHandler: Handle Wikibase's OutOfBoundsException for "we don't have a label" (T423514)]], [[gerrit:1271905|PageRenderingHandler: Don't run repo-mode stuff in non-repo world (T423515)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2402630 wikitext text/x-wiki 95r3l9t1ppo7yf05o9derpdj64pctlp 2402631 2402630 2026-04-15T20:54:19Z Stashbot 7414 jforrester@deploy1003: jforrester: Continuing with sync 2402631 wikitext text/x-wiki 0wfxb16bubvxfknq5qq5trf9wi1qf8s 2402632 2402631 2026-04-15T20:58:13Z Stashbot 7414 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1271894|PageRenderingHandler: Handle Wikibase's OutOfBoundsException for "we don't have a label" (T423514)]], [[gerrit:1271905|PageRenderingHandler: Don't run repo-mode stuff in non-repo world (T423515)]] (duration: 06m 08s) 2402632 wikitext text/x-wiki reiv0zevsr0a158cych00vk4agixslh 2402634 2402632 2026-04-15T21:02:05Z Stashbot 7414 swfrench@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2402634 wikitext text/x-wiki n32qantgwirm992uwgijxux7ewdznhq 2402635 2402634 2026-04-15T21:03:09Z Stashbot 7414 swfrench@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2402635 wikitext text/x-wiki 4m1yrkcf9p321jx3wk80lyvnie7715n 2402636 2402635 2026-04-15T21:03:30Z Stashbot 7414 swfrench@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2402636 wikitext text/x-wiki 38bflcownkq4bpkibmzhyh0rlzv3ytx 2402637 2402636 2026-04-15T21:04:01Z Stashbot 7414 swfrench@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2402637 wikitext text/x-wiki oqchecs37cmhd052yfd3ey0ale0dhxs 2402638 2402637 2026-04-15T21:04:13Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2402638 wikitext text/x-wiki 961bgrfbjh0r5esvlfn3ucc04odtle7 2402639 2402638 2026-04-15T21:04:50Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2402639 wikitext text/x-wiki 8n25tsk6b1cz7vutwnw0j9d5ac8jer9 2402640 2402639 2026-04-15T21:05:00Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2402640 wikitext text/x-wiki 369tpy2jt57q9tay11cpbp7xc7miqlq 2402641 2402640 2026-04-15T21:05:03Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2402641 wikitext text/x-wiki s1498l9qiaqlai6fsd1ypv1xo2daklb 2402642 2402641 2026-04-15T21:05:09Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2402642 wikitext text/x-wiki icci6wv6p6jmiibde93zej0mzse8i7n 2402643 2402642 2026-04-15T21:05:32Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2402643 wikitext text/x-wiki 4f0iein9j1g4xpavfezk5j4h1ui669n 2402644 2402643 2026-04-15T21:05:34Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2402644 wikitext text/x-wiki dfl8tbdwjjf91u9ftwcvtw1pwzln787 2402646 2402644 2026-04-15T21:06:01Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2402646 wikitext text/x-wiki j2kgf0lryrnwvyyuafpae29t8yq5sp6 2402647 2402646 2026-04-15T21:06:16Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2402647 wikitext text/x-wiki 0ndlg1r69txxswzt5f2w67psaapndc1 2402648 2402647 2026-04-15T21:06:19Z Stashbot 7414 bking@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cloudelastic1012.eqiad.wmnet with reason: still fixing Puppet 2402648 wikitext text/x-wiki 4rf0kte7kyn7jgvbdggkalda25bugxt 2402649 2402648 2026-04-15T21:06:22Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2402649 wikitext text/x-wiki tskdm9iauagd70ikio9hvo8ei3v3hhm 2402650 2402649 2026-04-15T21:06:27Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2402650 wikitext text/x-wiki t9ieoki505wb5ortnxyjysl2vxkenog 2402651 2402650 2026-04-15T21:07:06Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2402651 wikitext text/x-wiki 60dckoh37o3xzep0nxid1ntwucjal2z 2402652 2402651 2026-04-15T21:12:26Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2402652 wikitext text/x-wiki 5yq781klgvsgq3fngiv4iy7v91v68od 2402653 2402652 2026-04-15T21:12:46Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2402653 wikitext text/x-wiki j8kt5myxbok2xwr0uod0fj60jyvn8qx 2402654 2402653 2026-04-15T21:13:01Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2402654 wikitext text/x-wiki 8ecmcu9wg4gjlz72zv6tinqiarcdltf 2402655 2402654 2026-04-15T21:13:18Z Stashbot 7414 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.46.0-wmf.24 refs T420482 2402655 wikitext text/x-wiki 5plh1h7w7f0zdhvo1qcw9pyhuf7e22p 2402656 2402655 2026-04-15T21:13:34Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2402656 wikitext text/x-wiki ceqcyiuhl5dey0cjxgc1shkfbi9ike6 2402657 2402656 2026-04-15T21:13:42Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2402657 wikitext text/x-wiki lfypfwyc5y2suncv53h9v1qpgxxv9rk 2402658 2402657 2026-04-15T21:14:19Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2402658 wikitext text/x-wiki fvgu25sgyr0ac97i3qqloq5cyn709ig 2402659 2402658 2026-04-15T21:29:16Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.remove-downtime for aqs1027.eqiad.wmnet 2402659 wikitext text/x-wiki buek3q2mqtu9b29zwqvhglehrt7imht 2402660 2402659 2026-04-15T21:29:18Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for aqs1027.eqiad.wmnet 2402660 wikitext text/x-wiki of3nod1akspf4cer0iq8ahdqs4dxop7 2402664 2402660 2026-04-15T22:06:53Z Stashbot 7414 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1271945|PageRenderingHandler: Don't run repo-mode lang check in non-repo world either (T423515)]] 2402664 wikitext text/x-wiki px9qq0c0qtfotpm0cnwnz9m8d16wo8h 2402665 2402664 2026-04-15T22:08:46Z Stashbot 7414 jforrester@deploy1003: jforrester: Backport for [[gerrit:1271945|PageRenderingHandler: Don't run repo-mode lang check in non-repo world either (T423515)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2402665 wikitext text/x-wiki 1lbxn7dn3gkunac83jbqyjmmlnhn1xv 2402667 2402665 2026-04-15T22:11:49Z Stashbot 7414 jforrester@deploy1003: jforrester: Continuing with sync 2402667 wikitext text/x-wiki 65b5ysmpcd65i6rju5ixl2uzzm1mhc4 2402668 2402667 2026-04-15T22:12:20Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T419635)', diff saved to https://phabricator.wikimedia.org/P90845 and previous config saved to /var/cache/conftool/dbconfig/20260415-221216-fceratto.json 2402668 wikitext text/x-wiki jviq9idh3ucca43764w4vr52i2iw8ao 2402670 2402668 2026-04-15T22:15:42Z Stashbot 7414 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1271945|PageRenderingHandler: Don't run repo-mode lang check in non-repo world either (T423515)]] (duration: 08m 48s) 2402670 wikitext text/x-wiki 6i7ivv3ngpqkuc1rhsafbzr0w6z5vxy 2402671 2402670 2026-04-15T22:22:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P90846 and previous config saved to /var/cache/conftool/dbconfig/20260415-222225-fceratto.json 2402671 wikitext text/x-wiki h0z8auovj316qwkn2n8uqqoyuxujdcf 2402672 2402671 2026-04-15T22:32:34Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P90847 and previous config saved to /var/cache/conftool/dbconfig/20260415-223233-fceratto.json 2402672 wikitext text/x-wiki 0o11372865ktfhbu4yo4a5oyvhwz8m7 2402674 2402672 2026-04-15T22:41:23Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1271910|Pass preferred LanguageConverter variant explicitly instead of implicitly (T415435)]], [[gerrit:1271911|Make variant into a parser option for parsoid language conversion (T415435)]] 2402674 wikitext text/x-wiki qspocwh2vizs9omsp2jzxog7fctsi09 2402675 2402674 2026-04-15T22:42:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T419635)', diff saved to https://phabricator.wikimedia.org/P90848 and previous config saved to /var/cache/conftool/dbconfig/20260415-224241-fceratto.json 2402675 wikitext text/x-wiki 8gjxthjnmbw6ga7ar6ctlb1mxj5zfbg 2402676 2402675 2026-04-15T22:42:59Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1238.eqiad.wmnet with reason: Maintenance 2402676 wikitext text/x-wiki 96vmqwjbxi83x78al58mkbfpz756tcq 2402677 2402676 2026-04-15T22:43:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1238 (T419635)', diff saved to https://phabricator.wikimedia.org/P90849 and previous config saved to /var/cache/conftool/dbconfig/20260415-224305-fceratto.json 2402677 wikitext text/x-wiki iybewhurezfxni9baisy8qs19cwt9w0 2402678 2402677 2026-04-15T22:43:14Z Stashbot 7414 cscott@deploy1003: cscott: Backport for [[gerrit:1271910|Pass preferred LanguageConverter variant explicitly instead of implicitly (T415435)]], [[gerrit:1271911|Make variant into a parser option for parsoid language conversion (T415435)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2402678 wikitext text/x-wiki 5dn9xx7m60ftr0ac7p3slqc3rq96bsa 2402680 2402678 2026-04-15T22:53:35Z Stashbot 7414 cscott@deploy1003: cscott: Continuing with sync 2402680 wikitext text/x-wiki qchzawfm634hf2dzio41a7kn6kvv836 2402681 2402680 2026-04-15T22:57:24Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1271910|Pass preferred LanguageConverter variant explicitly instead of implicitly (T415435)]], [[gerrit:1271911|Make variant into a parser option for parsoid language conversion (T415435)]] (duration: 16m 00s) 2402681 wikitext text/x-wiki jt7ykknpbluw4fvfn35ea7cjxd2vuu9 2402682 2402681 2026-04-15T23:03:01Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1271889|Exclude parser functions from SpecialLintTemplateErrors (T420102)]] 2402682 wikitext text/x-wiki fpimoto7qvq9t1vw6u1j8k57k58ll6t 2402683 2402682 2026-04-15T23:05:14Z Stashbot 7414 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply 2402683 wikitext text/x-wiki 4i2h1tn7ron4ntx1m44lezcm5iweooj 2402684 2402683 2026-04-15T23:05:33Z Stashbot 7414 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply 2402684 wikitext text/x-wiki 8cnp97n1npkone51unrrx1kku53lehk 2402685 2402684 2026-04-15T23:20:03Z Stashbot 7414 cscott@deploy1003: cscott: Backport for [[gerrit:1271889|Exclude parser functions from SpecialLintTemplateErrors (T420102)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2402685 wikitext text/x-wiki 5k8o3vpeugoney1ct7ib99gsibbqcu7 2402686 2402685 2026-04-15T23:23:01Z Stashbot 7414 cscott@deploy1003: cscott: Continuing with sync 2402686 wikitext text/x-wiki 7d39vd5fquzjq4094ugdt55ptfhcqvb 2402692 2402686 2026-04-15T23:35:51Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1271889|Exclude parser functions from SpecialLintTemplateErrors (T420102)]] (duration: 32m 47s) 2402692 wikitext text/x-wiki 8z2nsav7wb2661eio0bwuwquk807ov6 2402696 2402692 2026-04-16T00:31:33Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238 (T419635)', diff saved to https://phabricator.wikimedia.org/P90850 and previous config saved to /var/cache/conftool/dbconfig/20260416-003130-fceratto.json 2402696 wikitext text/x-wiki g90l3vgnpa8190i4eox8p8u7c3xkrrh 2402698 2402696 2026-04-16T00:41:40Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P90851 and previous config saved to /var/cache/conftool/dbconfig/20260416-004138-fceratto.json 2402698 wikitext text/x-wiki th2tfqjsx6gmfl6rq7yshm12jgui2eu 2402699 2402698 2026-04-16T00:51:47Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P90852 and previous config saved to /var/cache/conftool/dbconfig/20260416-005146-fceratto.json 2402699 wikitext text/x-wiki adp1w1dcocatlfismlex1iirid3rgm0 2402700 2402699 2026-04-16T00:57:09Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221 (T410589)', diff saved to https://phabricator.wikimedia.org/P90853 and previous config saved to /var/cache/conftool/dbconfig/20260416-005706-ladsgroup.json 2402700 wikitext text/x-wiki r5wxerpm23vfgpnfeb3gm9gkjgij8zi 2402702 2402700 2026-04-16T01:01:57Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238 (T419635)', diff saved to https://phabricator.wikimedia.org/P90854 and previous config saved to /var/cache/conftool/dbconfig/20260416-010154-fceratto.json 2402702 wikitext text/x-wiki 3l3wdfdgsev3eke5cwb6eun9nwryzz9 2402703 2402702 2026-04-16T01:02:12Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1241.eqiad.wmnet with reason: Maintenance 2402703 wikitext text/x-wiki q7trvmj7b9o2maqijg72krh6x7bxsb1 2402704 2402703 2026-04-16T01:02:21Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1241 (T419635)', diff saved to https://phabricator.wikimedia.org/P90855 and previous config saved to /var/cache/conftool/dbconfig/20260416-010218-fceratto.json 2402704 wikitext text/x-wiki 8r40g3b57ekk2wdevhc6rbhg423zv1f 2402705 2402704 2026-04-16T01:07:16Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P90856 and previous config saved to /var/cache/conftool/dbconfig/20260416-010714-ladsgroup.json 2402705 wikitext text/x-wiki 1xssmqxi3dirs2weslh5nmv8gvxwcuh 2402706 2402705 2026-04-16T01:17:24Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P90857 and previous config saved to /var/cache/conftool/dbconfig/20260416-011722-ladsgroup.json 2402706 wikitext text/x-wiki fm134er8vujnnk6dxllx3q8avtgdrqu 2402707 2402706 2026-04-16T01:27:33Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221 (T410589)', diff saved to https://phabricator.wikimedia.org/P90858 and previous config saved to /var/cache/conftool/dbconfig/20260416-012730-ladsgroup.json 2402707 wikitext text/x-wiki rr88i7rndtvkmtg6p8nvlt9qis1cxo6 2402708 2402707 2026-04-16T01:27:48Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance 2402708 wikitext text/x-wiki 1kyyed1r9hcqgtn7ictok6kqbr0tiqi 2402709 2402708 2026-04-16T01:27:58Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2222 (T410589)', diff saved to https://phabricator.wikimedia.org/P90859 and previous config saved to /var/cache/conftool/dbconfig/20260416-012755-ladsgroup.json 2402709 wikitext text/x-wiki mu2gl9uiy7mqno7bd813fswsvwh3oqg 2402710 2402709 2026-04-16T02:01:21Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2402710 wikitext text/x-wiki 6fkc7h9ivvvgbskze51tqw53oioiaq4 2402711 2402710 2026-04-16T02:07:38Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 16s) 2402711 wikitext text/x-wiki mij6hdz60tc42jgzan1p60a1u2gvwaa 2402712 2402711 2026-04-16T02:22:26Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253 (T410589)', diff saved to https://phabricator.wikimedia.org/P90860 and previous config saved to /var/cache/conftool/dbconfig/20260416-022223-ladsgroup.json 2402712 wikitext text/x-wiki psln9lco1zb58od76i88v593h1qc3ur 2402714 2402712 2026-04-16T02:32:33Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P90861 and previous config saved to /var/cache/conftool/dbconfig/20260416-023231-ladsgroup.json 2402714 wikitext text/x-wiki helgyu556eka3dt9hkcg6czwpjwkl6g 2402716 2402714 2026-04-16T02:42:41Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P90862 and previous config saved to /var/cache/conftool/dbconfig/20260416-024239-ladsgroup.json 2402716 wikitext text/x-wiki pl7gfa0ejp2p4bs37wvn967sv4713lq 2402717 2402716 2026-04-16T02:48:49Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T419635)', diff saved to https://phabricator.wikimedia.org/P90863 and previous config saved to /var/cache/conftool/dbconfig/20260416-024845-fceratto.json 2402717 wikitext text/x-wiki juc6bhs14ljou4yad439urym8k2xviz 2402718 2402717 2026-04-16T02:52:55Z Stashbot 7414 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253 (T410589)', diff saved to https://phabricator.wikimedia.org/P90864 and previous config saved to /var/cache/conftool/dbconfig/20260416-025247-ladsgroup.json 2402718 wikitext text/x-wiki hayqjf4ruaeel79amvyuj01y7n02v50 2402719 2402718 2026-04-16T02:53:05Z Stashbot 7414 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance 2402719 wikitext text/x-wiki ivszf2qf2uaracfce2ste1yq4e9mdcf 2402720 2402719 2026-04-16T02:58:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P90865 and previous config saved to /var/cache/conftool/dbconfig/20260416-025853-fceratto.json 2402720 wikitext text/x-wiki n6z3pow4yg5xoofqmxygk9ihb9w8hi8 2402721 2402720 2026-04-16T03:09:03Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P90866 and previous config saved to /var/cache/conftool/dbconfig/20260416-030902-fceratto.json 2402721 wikitext text/x-wiki q343p138ph1rmqr1rd5bgmvtsqxjter 2402722 2402721 2026-04-16T03:19:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T419635)', diff saved to https://phabricator.wikimedia.org/P90867 and previous config saved to /var/cache/conftool/dbconfig/20260416-031910-fceratto.json 2402722 wikitext text/x-wiki 32faluf4z7aj6e3lp3o22u6c0us4wbb 2402723 2402722 2026-04-16T03:19:29Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance 2402723 wikitext text/x-wiki kdpk7rbijtszar0sb2a18lm41p0pu2r 2402724 2402723 2026-04-16T03:19:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1242 (T419635)', diff saved to https://phabricator.wikimedia.org/P90868 and previous config saved to /var/cache/conftool/dbconfig/20260416-031934-fceratto.json 2402724 wikitext text/x-wiki ajng59f7xc6lequku0iy8nseaenkc07 2402726 2402724 2026-04-16T05:06:13Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T419635)', diff saved to https://phabricator.wikimedia.org/P90869 and previous config saved to /var/cache/conftool/dbconfig/20260416-050609-fceratto.json 2402726 wikitext text/x-wiki e7la6p3u9zpekyqmmcwmcf441y31den 2402727 2402726 2026-04-16T05:16:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P90870 and previous config saved to /var/cache/conftool/dbconfig/20260416-051618-fceratto.json 2402727 wikitext text/x-wiki t07g6i2h9qxfjtjx07qgase4zloepc6 2402728 2402727 2026-04-16T05:22:29Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.decommission for hosts clouddb1019.eqiad.wmnet 2402728 wikitext text/x-wiki cgvg7l3y8ebitwxxiv0aos73zr847i8 2402729 2402728 2026-04-16T05:26:28Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P90871 and previous config saved to /var/cache/conftool/dbconfig/20260416-052626-fceratto.json 2402729 wikitext text/x-wiki lz86anerlrtohum4sueab3sgn0x6nnv 2402730 2402729 2026-04-16T05:27:05Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.dns.netbox 2402730 wikitext text/x-wiki 0x2bq81bj3ybhe956dk9lnghaak28ky 2402731 2402730 2026-04-16T05:30:47Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clouddb1019.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2402731 wikitext text/x-wiki 6etudpcpezxiq42qmb4lyxiaf66tz6u 2402732 2402731 2026-04-16T05:31:02Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clouddb1019.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" 2402732 wikitext text/x-wiki g0rzq2efw4klup5a80hkrda7a8bvx6q 2402733 2402732 2026-04-16T05:31:04Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2402733 wikitext text/x-wiki ccvs4cktefk3z76yw7y14amy9c9z9ru 2402734 2402733 2026-04-16T05:31:05Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts clouddb1019.eqiad.wmnet 2402734 wikitext text/x-wiki krqsl0a38zsg5xf4w7nyd6cy9d81hfh 2402735 2402734 2026-04-16T05:36:38Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T419635)', diff saved to https://phabricator.wikimedia.org/P90872 and previous config saved to /var/cache/conftool/dbconfig/20260416-053635-fceratto.json 2402735 wikitext text/x-wiki kc7c5o1vlrizp6kcbmvp23owhe7pz1p 2402736 2402735 2026-04-16T05:36:52Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1243.eqiad.wmnet with reason: Maintenance 2402736 wikitext text/x-wiki 091ybr4ktf4roywvd964yvpsd4ftxi6 2402737 2402736 2026-04-16T05:37:02Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1243 (T419635)', diff saved to https://phabricator.wikimedia.org/P90873 and previous config saved to /var/cache/conftool/dbconfig/20260416-053659-fceratto.json 2402737 wikitext text/x-wiki rdj38tvoyb21wksrkb6epqhpurctj1g 2402738 2402737 2026-04-16T05:47:42Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2193.codfw.wmnet with reason: Reimage to Trixie 2402738 wikitext text/x-wiki n7zap9dabpxp74tt1x1xtke58ktliuj 2402739 2402738 2026-04-16T05:47:47Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2193: Reimage to Trixie 2402739 wikitext text/x-wiki f8vowymvesmfg9ntistat1nc69mussd 2402740 2402739 2026-04-16T05:48:06Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2193: Reimage to Trixie 2402740 wikitext text/x-wiki a6u8pafh9h1h9ec7krl1uuw3vbin24e 2402741 2402740 2026-04-16T05:49:03Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2193.codfw.wmnet with OS trixie 2402741 wikitext text/x-wiki 8cwgw1fi6jaytbvr6b6n1cwetxz5d5l 2402742 2402741 2026-04-16T06:08:04Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2193.codfw.wmnet with reason: host reimage 2402742 wikitext text/x-wiki 094m08pj9d139iycjah7p7civwr4pax 2402743 2402742 2026-04-16T06:14:48Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2193.codfw.wmnet with reason: host reimage 2402743 wikitext text/x-wiki 2rho3zwdx0yjkio7r8stooln9ioslhx 2402744 2402743 2026-04-16T06:37:34Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2193.codfw.wmnet with OS trixie 2402744 wikitext text/x-wiki 1rgtjfrininphxv5g95fnvsadatdp4s 2402745 2402744 2026-04-16T06:38:00Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2193: after reimage to trixie 2402745 wikitext text/x-wiki 80b8kbjhoe9raf9vtn11egnpie2c1oh 2402746 2402745 2026-04-16T06:40:25Z Stashbot 7414 jayme@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2280.codfw.wmnet 2402746 wikitext text/x-wiki ffzlwkl79xzqz2cqfrtd1i4n5sicytt 2402747 2402746 2026-04-16T06:40:27Z Stashbot 7414 jayme@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2280.codfw.wmnet 2402747 wikitext text/x-wiki rdpc3tz2ommty7uni8xjsvfpev35xyq 2402748 2402747 2026-04-16T06:40:33Z Stashbot 7414 jayme@cumin2002: START - Cookbook sre.hosts.remove-downtime for wikikube-worker2280.codfw.wmnet 2402748 wikitext text/x-wiki f7lt5cftgykhqiqrhqa4yfwvbv4r7nv 2402749 2402748 2026-04-16T06:40:34Z Stashbot 7414 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker2280.codfw.wmnet 2402749 wikitext text/x-wiki 4vf4lh9z7k5czkuwf5rswenzm6v94zd 2402751 2402749 2026-04-16T06:55:19Z Stashbot 7414 moritzm: imported opensearch-madvise 0.2+deb13u1 to component/opensearch2 of trixie-wikimedia T422860 2402751 wikitext text/x-wiki 6jxjlmtlgpn7lxf2ut1uzl4xui4md7a 2402753 2402751 2026-04-16T06:59:22Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad 2402753 wikitext text/x-wiki 89ssh9ts5b36ywi79qxijqxllvvwo27 2402754 2402753 2026-04-16T07:16:36Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet 2402754 wikitext text/x-wiki f37st44f5k5n23f9b5h18itmft7gm6t 2402755 2402754 2026-04-16T07:21:41Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet 2402755 wikitext text/x-wiki gd28386qrcznzd5prbrpimoysx9tk5t 2402756 2402755 2026-04-16T07:23:26Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2193: after reimage to trixie 2402756 wikitext text/x-wiki detqwhe44c699qokbc75hes9mo9bl5n 2402757 2402756 2026-04-16T07:24:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T419635)', diff saved to https://phabricator.wikimedia.org/P90879 and previous config saved to /var/cache/conftool/dbconfig/20260416-072432-fceratto.json 2402757 wikitext text/x-wiki 7m5d5txkm972n35kuv2mieywpg69yh9 2402758 2402757 2026-04-16T07:26:36Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1165.eqiad.wmnet with reason: Maintenance 2402758 wikitext text/x-wiki 7bhkws53s865nk3eygiunh9bmkk7vdg 2402759 2402758 2026-04-16T07:26:44Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb1015.eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance 2402759 wikitext text/x-wiki avwtr7v1yridy0zrognwewn7uj01i7z 2402760 2402759 2026-04-16T07:26:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1165 (T419961)', diff saved to https://phabricator.wikimedia.org/P90880 and previous config saved to /var/cache/conftool/dbconfig/20260416-072650-fceratto.json 2402760 wikitext text/x-wiki q65f6bgazn5wjqe60kbveuv25tl5ln1 2402761 2402760 2026-04-16T07:26:55Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet 2402761 wikitext text/x-wiki h69f0jrppyb922pyb6ykljmofbbauvp 2402762 2402761 2026-04-16T07:26:58Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet 2402762 wikitext text/x-wiki 72uxlv83olm618xu8ks05h8vx5chjmp 2402763 2402762 2026-04-16T07:27:03Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet 2402763 wikitext text/x-wiki n7079ervukwif6wteg91a09t40szdj6 2402764 2402763 2026-04-16T07:27:41Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet 2402764 wikitext text/x-wiki ckmykgjkggmg11si2ml7aehbgbc3nbd 2402766 2402764 2026-04-16T07:32:54Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet 2402766 wikitext text/x-wiki kmafayicmj5j3k54nktfffq5ohe6xf9 2402767 2402766 2026-04-16T07:32:56Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet 2402767 wikitext text/x-wiki 99fnp0eb0wcj2ggnl8jcvabldulj2l0 2402768 2402767 2026-04-16T07:33:02Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet 2402768 wikitext text/x-wiki g1x3y3ccgh3iiequ169dvzr9s5764iz 2402769 2402768 2026-04-16T07:33:40Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet 2402769 wikitext text/x-wiki 7dnekg4kzqikvkixtk371v69ietwt83 2402770 2402769 2026-04-16T07:33:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T419961)', diff saved to https://phabricator.wikimedia.org/P90881 and previous config saved to /var/cache/conftool/dbconfig/20260416-073354-fceratto.json 2402770 wikitext text/x-wiki ke6nelyihsgm7zujhb1cnp8y49xzium 2402771 2402770 2026-04-16T07:34:42Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P90882 and previous config saved to /var/cache/conftool/dbconfig/20260416-073440-fceratto.json 2402771 wikitext text/x-wiki 2ek4lwopxwk0ieez1jwtkk8q0don6we 2402772 2402771 2026-04-16T07:39:07Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet 2402772 wikitext text/x-wiki 4z18mpz298oeoopeqpw09fmu1o5kqcu 2402773 2402772 2026-04-16T07:39:09Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1008.eqiad.wmnet 2402773 wikitext text/x-wiki 6ne1pr9vauf2fcxk45hmsr8892ij2x7 2402774 2402773 2026-04-16T07:39:15Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1009.eqiad.wmnet 2402774 wikitext text/x-wiki 0e5dv4f1y49czcji6ru26i83kc7t7c1 2402775 2402774 2026-04-16T07:39:50Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1009.eqiad.wmnet 2402775 wikitext text/x-wiki srf6xnkrjme5z7qp5pvf7sfbkj8mafu 2402776 2402775 2026-04-16T07:40:27Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2169.codfw.wmnet with reason: Reimage to Trixie 2402776 wikitext text/x-wiki awk0eomjyh7edam1f0cgx8z2j32qikq 2402777 2402776 2026-04-16T07:40:33Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db2169: Reimage to Trixie 2402777 wikitext text/x-wiki f5invil2nq4s997o755qmj5lvjb201n 2402778 2402777 2026-04-16T07:40:52Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: Reimage to Trixie 2402778 wikitext text/x-wiki 6mtbjbc0hevc0dageoql9wgfj2baxss 2402779 2402778 2026-04-16T07:41:35Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db2169.codfw.wmnet with OS trixie 2402779 wikitext text/x-wiki i5jprny4zwvvhhzj1x2qx3cqttta8c7 2402780 2402779 2026-04-16T07:44:04Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P90884 and previous config saved to /var/cache/conftool/dbconfig/20260416-074402-fceratto.json 2402780 wikitext text/x-wiki 1yku7h0r2yw5hmkfopkai9dw4fv6kle 2402781 2402780 2026-04-16T07:44:50Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P90885 and previous config saved to /var/cache/conftool/dbconfig/20260416-074448-fceratto.json 2402781 wikitext text/x-wiki j4rtwv1n3fem5uyef3em8y9nw8v1dwb 2402782 2402781 2026-04-16T07:45:16Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1009.eqiad.wmnet 2402782 wikitext text/x-wiki 75civk1ht6zurhtnd69o6fl10imoiue 2402783 2402782 2026-04-16T07:45:19Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1009.eqiad.wmnet 2402783 wikitext text/x-wiki 8yzglx76kqp90rmw3f0inmwqviufd73 2402784 2402783 2026-04-16T07:45:22Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad 2402784 wikitext text/x-wiki 1dh5lfhvmjv1bddmul5j1y9ri8qc20j 2402785 2402784 2026-04-16T07:54:12Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P90886 and previous config saved to /var/cache/conftool/dbconfig/20260416-075410-fceratto.json 2402785 wikitext text/x-wiki fpxaxr6awddi8mc2dhbfd8f191qc9qq 2402786 2402785 2026-04-16T07:55:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T419635)', diff saved to https://phabricator.wikimedia.org/P90887 and previous config saved to /var/cache/conftool/dbconfig/20260416-075457-fceratto.json 2402786 wikitext text/x-wiki 1cl28gee2m4obec151yg9rx7mpyasli 2402787 2402786 2026-04-16T07:55:15Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance 2402787 wikitext text/x-wiki fcpthewh1le4nuhelzej1irx50wdysi 2402788 2402787 2026-04-16T07:55:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1244 (T419635)', diff saved to https://phabricator.wikimedia.org/P90888 and previous config saved to /var/cache/conftool/dbconfig/20260416-075522-fceratto.json 2402788 wikitext text/x-wiki 5g2tok0phse965iy56zuiqq5uon6cs1 2402790 2402788 2026-04-16T08:00:49Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2169.codfw.wmnet with reason: host reimage 2402790 wikitext text/x-wiki ebhxt6qv1yoljxyldgwx8hylesphmw3 2402791 2402790 2026-04-16T08:04:22Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T419961)', diff saved to https://phabricator.wikimedia.org/P90889 and previous config saved to /var/cache/conftool/dbconfig/20260416-080420-fceratto.json 2402791 wikitext text/x-wiki 0h1c2ry61dzxsqgne7o8xroafbgt5h2 2402792 2402791 2026-04-16T08:04:37Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1201.eqiad.wmnet with reason: Maintenance 2402792 wikitext text/x-wiki 8na5dw4uvueqrc3zaufzeabg057b6wm 2402793 2402792 2026-04-16T08:04:47Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1201 (T419961)', diff saved to https://phabricator.wikimedia.org/P90890 and previous config saved to /var/cache/conftool/dbconfig/20260416-080445-fceratto.json 2402793 wikitext text/x-wiki ez5ugob3p59la0xjnoeb602zt61bsf0 2402794 2402793 2026-04-16T08:05:05Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2169.codfw.wmnet with reason: host reimage 2402794 wikitext text/x-wiki otx0150nltu91d3nhcl058r6jwjcokf 2402795 2402794 2026-04-16T08:13:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T419961)', diff saved to https://phabricator.wikimedia.org/P90891 and previous config saved to /var/cache/conftool/dbconfig/20260416-081305-fceratto.json 2402795 wikitext text/x-wiki 7ik1tc9x2ztlu7om7be3trd36iu6kly 2402796 2402795 2026-04-16T08:23:16Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P90892 and previous config saved to /var/cache/conftool/dbconfig/20260416-082314-fceratto.json 2402796 wikitext text/x-wiki azzeom9fand178spyuqyswc8e8fpyoq 2402799 2402796 2026-04-16T08:27:40Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2169.codfw.wmnet with OS trixie 2402799 wikitext text/x-wiki 9pzaxo4dutqg4a3l1jzxd4967u0vjbi 2402801 2402799 2026-04-16T08:31:13Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2169: repool after maintenance 2402801 wikitext text/x-wiki eys7pyvn70oxn8tnwpeo7v2w9763vdi 2402802 2402801 2026-04-16T08:33:25Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P90894 and previous config saved to /var/cache/conftool/dbconfig/20260416-083323-fceratto.json 2402802 wikitext text/x-wiki ozo76am7z6t6pj4iyiylhnsit7r7hcg 2402803 2402802 2026-04-16T08:36:49Z Stashbot 7414 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup[1007,1014].eqiad.wmnet with reason: maintenance 2402803 wikitext text/x-wiki craioytsimld6ugyanh19tdlynn5k88 2402804 2402803 2026-04-16T08:43:33Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T419961)', diff saved to https://phabricator.wikimedia.org/P90895 and previous config saved to /var/cache/conftool/dbconfig/20260416-084331-fceratto.json 2402804 wikitext text/x-wiki 3lui95ks0516smmargp6msji7r8325c 2402805 2402804 2026-04-16T08:43:53Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance 2402805 wikitext text/x-wiki ljcf7z6z9k4lefbye43vgoqul59zy2g 2402807 2402805 2026-04-16T08:50:13Z Stashbot 7414 jmm@dns1004: START - running authdns-update 2402807 wikitext text/x-wiki bp5dj9pkne38t7wfy033j1k6pwj8xai 2402808 2402807 2026-04-16T08:51:40Z Stashbot 7414 jmm@dns1004: END - running authdns-update 2402808 wikitext text/x-wiki gae28vfejdi0o8p3hb3p5vcyymx1dch 2402809 2402808 2026-04-16T08:55:23Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply 2402809 wikitext text/x-wiki dhz97wfo42l8x8minfzjvhhlpjd9yxr 2402810 2402809 2026-04-16T08:55:46Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply 2402810 wikitext text/x-wiki 7kz4ptu0uufgs7rxt3xhos87epksnkg 2402811 2402810 2026-04-16T08:56:01Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply 2402811 wikitext text/x-wiki aql7d6g4znxmglrs6923q5fgw9cxwiv 2402812 2402811 2026-04-16T08:56:54Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply 2402812 wikitext text/x-wiki bugfc6shlx2hx19pcb1x5cq7bkuwf3e 2402814 2402812 2026-04-16T09:03:32Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.move-vlan for host backup1007 2402814 wikitext text/x-wiki mdjzb8dmom54bbbes82qmw110u0hu4x 2402815 2402814 2026-04-16T09:03:47Z Stashbot 7414 ayounsi@cumin1003: END (FAIL) - Cookbook sre.hosts.move-vlan (exit_code=99) for host backup1007 2402815 wikitext text/x-wiki 73aozm5cbkyil0bp47bxsvhmia4hvjf 2402817 2402815 2026-04-16T09:10:51Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.move-vlan for host backup1007 2402817 wikitext text/x-wiki tdy6kt9nu7ur89h5kkw3d1dh4w9g0ll 2402818 2402817 2026-04-16T09:11:06Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.netbox 2402818 wikitext text/x-wiki npdp81u9mi07e8pd6n73n0s8sx313ce 2402819 2402818 2026-04-16T09:11:07Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance 2402819 wikitext text/x-wiki ahf585oxv3dc2ovdm4j9ps2cbq917eo 2402820 2402819 2026-04-16T09:11:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1159 (T419961)', diff saved to https://phabricator.wikimedia.org/P90898 and previous config saved to /var/cache/conftool/dbconfig/20260416-091115-fceratto.json 2402820 wikitext text/x-wiki ptk6aiqsiieu2nw42yii1pe1mmyv7ut 2402822 2402820 2026-04-16T09:13:11Z Stashbot 7414 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply 2402822 wikitext text/x-wiki k1cx2adiytaouae1y3otkebcriywp9j 2402823 2402822 2026-04-16T09:13:18Z Stashbot 7414 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply 2402823 wikitext text/x-wiki qiwc76hkvyljxm2fyyuo9a6f6pd24ty 2402824 2402823 2026-04-16T09:14:53Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host backup1007 - ayounsi@cumin1003" 2402824 wikitext text/x-wiki avzjn5l7av69n2hf11w6vksvn7ge56f 2402825 2402824 2026-04-16T09:14:58Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host backup1007 - ayounsi@cumin1003" 2402825 wikitext text/x-wiki 6i28matqqbucqpp9uyq1zln7hjmew4h 2402826 2402825 2026-04-16T09:14:59Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2402826 wikitext text/x-wiki 2wit485ht5tazlo60s74z9a6dpbi5ow 2402827 2402826 2026-04-16T09:15:01Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.dns.wipe-cache backup1007.eqiad.wmnet 88.48.64.10.in-addr.arpa 8.8.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2402827 wikitext text/x-wiki sb45ub0s6mvngkqeov48zsp2id792cq 2402828 2402827 2026-04-16T09:15:03Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) backup1007.eqiad.wmnet 88.48.64.10.in-addr.arpa 8.8.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2402828 wikitext text/x-wiki rs50bisajvwzxiqk8n0qvjpb4d5gx0q 2402829 2402828 2026-04-16T09:15:05Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host backup1007 2402829 wikitext text/x-wiki 7mfzeknx17an0rmfshaqzf6045zz5l7 2402830 2402829 2026-04-16T09:15:17Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host backup1007 2402830 wikitext text/x-wiki tf9f7z0uizbicn3itqa8fx54moqp6mt 2402831 2402830 2026-04-16T09:15:19Z Stashbot 7414 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host backup1007 2402831 wikitext text/x-wiki htqjfgn9dxnmmofhc1a37ua9incwam0 2402832 2402831 2026-04-16T09:16:41Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: repool after maintenance 2402832 wikitext text/x-wiki 86b714g4d0tbkl65igmpm6q5iqcjnw1 2402833 2402832 2026-04-16T09:18:20Z Stashbot 7414 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply 2402833 wikitext text/x-wiki sqf1jw4tfq5iu5h8strq45km7sbj1lr 2402834 2402833 2026-04-16T09:20:27Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . 2402834 wikitext text/x-wiki nsvt3i4ttsr1g6ribexd1ae47ysimlq 2402835 2402834 2026-04-16T09:20:35Z Stashbot 7414 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply 2402835 wikitext text/x-wiki 48rs3w6f9wk838im23itgxyw9fwqlgm 2402836 2402835 2026-04-16T09:24:34Z Stashbot 7414 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply 2402836 wikitext text/x-wiki 3xyn1bsx772znyyast5zykvt7pbiacz 2402837 2402836 2026-04-16T09:24:50Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . 2402837 wikitext text/x-wiki s92pic1j8upn1nedccf0mhmfltevl20 2402838 2402837 2026-04-16T09:29:24Z Stashbot 7414 jynus: setting backup1014 in maintenance, no backup or recovery will run while it T421719 2402838 wikitext text/x-wiki dv6cvaxhgp7huoc67rx4zrjplk1in3g 2402839 2402838 2026-04-16T09:31:03Z Stashbot 7414 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply 2402839 wikitext text/x-wiki h88hzqg32uj0mmodt89z0g73v7pnm86 2402841 2402839 2026-04-16T09:37:42Z Stashbot 7414 moritzm: installing imagemagick security updates 2402841 wikitext text/x-wiki dv9cqy8xdpxqzt7bwvgdl4cmgjbxkts 2402843 2402841 2026-04-16T09:40:36Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.hosts.move-vlan for host backup1014 2402843 wikitext text/x-wiki 1zcyecw6gcq6hd7938xj38cp3ka645m 2402844 2402843 2026-04-16T09:41:17Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.dns.netbox 2402844 wikitext text/x-wiki l21ms9wr0r4on82iga1hksbknmsgfhr 2402845 2402844 2026-04-16T09:42:15Z Stashbot 7414 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: cp2041.codfw.wmnet 2402845 wikitext text/x-wiki so70cefwtw0589r142zitawrwkhbetd 2402846 2402845 2026-04-16T09:42:26Z Stashbot 7414 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: cp2042.codfw.wmnet 2402846 wikitext text/x-wiki 1axi7qavrqibwb888v6b2shp6t6vwe7 2402847 2402846 2026-04-16T09:44:21Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad 2402847 wikitext text/x-wiki titavk4hh1cdxrouvx4nd9ylfmlp176 2402848 2402847 2026-04-16T09:44:41Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T419635)', diff saved to https://phabricator.wikimedia.org/P90900 and previous config saved to /var/cache/conftool/dbconfig/20260416-094436-fceratto.json 2402848 wikitext text/x-wiki gja0c5pdymd4025rdv0d7egu6a2fpwo 2402849 2402848 2026-04-16T09:45:44Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host backup1014 - jynus@cumin1003" 2402849 wikitext text/x-wiki cvm21akv942bmwsqv6too04zebzcsh0 2402850 2402849 2026-04-16T09:45:50Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host backup1014 - jynus@cumin1003" 2402850 wikitext text/x-wiki i3ji9zs9wxpgy6fz20odav5id6x8g39 2402851 2402850 2026-04-16T09:45:52Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2402851 wikitext text/x-wiki ky7u686tb22mrbyjg8l06oi3ng4sx6a 2402852 2402851 2026-04-16T09:45:53Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.dns.wipe-cache backup1014.eqiad.wmnet 20.48.64.10.in-addr.arpa 0.2.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2402852 wikitext text/x-wiki 48bgz7n8j7ilns0rtkqvvcammcdxku5 2402853 2402852 2026-04-16T09:45:55Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) backup1014.eqiad.wmnet 20.48.64.10.in-addr.arpa 0.2.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2402853 wikitext text/x-wiki 0r68f11c6qcpuifktkd53h0wktaaroa 2402854 2402853 2026-04-16T09:45:57Z Stashbot 7414 jynus@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host backup1014 2402854 wikitext text/x-wiki 87ngrygz1liv3wmts9v7zua881dvy33 2402855 2402854 2026-04-16T09:47:38Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host backup1014 2402855 wikitext text/x-wiki ligkd5teot190ltqttcuko46n05cyjs 2402856 2402855 2026-04-16T09:47:43Z Stashbot 7414 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host backup1014 2402856 wikitext text/x-wiki 6h03kggv3auj4r18kghsdv9wuk9p8vi 2402857 2402856 2026-04-16T09:52:08Z Stashbot 7414 moritzm: installing qemu security updates 2402857 wikitext text/x-wiki h30tn7it7nycu91uoqzui6y8g9rvdfx 2402858 2402857 2026-04-16T09:54:24Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad 2402858 wikitext text/x-wiki 7kcowa58fb5g7enpfoa3q17kgqy8iks 2402859 2402858 2026-04-16T09:55:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P90901 and previous config saved to /var/cache/conftool/dbconfig/20260416-095455-fceratto.json 2402859 wikitext text/x-wiki kd26gd9ul2z8nem5ohfkgwnb23wo6pc 2402861 2402859 2026-04-16T10:05:07Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P90902 and previous config saved to /var/cache/conftool/dbconfig/20260416-100505-fceratto.json 2402861 wikitext text/x-wiki 2eojhypmnf44lyzz836glrpxwsmeerx 2402862 2402861 2026-04-16T10:09:22Z Stashbot 7414 jynus: backup1014 returns from maintenance, backups and recovery can flow as usual T421719 2402862 wikitext text/x-wiki 2frpj1c6ggmjxi4e4tjkmoxv0bfh17x 2402863 2402862 2026-04-16T10:11:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T419961)', diff saved to https://phabricator.wikimedia.org/P90903 and previous config saved to /var/cache/conftool/dbconfig/20260416-101135-fceratto.json 2402863 wikitext text/x-wiki 5moepma2p3kqr0f5o23fspcbzagzzat 2402864 2402863 2026-04-16T10:13:36Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2402864 wikitext text/x-wiki lkm714aau8s4y3pgzd4auetj61a9k6w 2402865 2402864 2026-04-16T10:14:05Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2402865 wikitext text/x-wiki djc23bwpn3qhnvqxfi3ywyyodob796j 2402866 2402865 2026-04-16T10:15:02Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2402866 wikitext text/x-wiki fwbpc30dk0s726mdsu12cddzcxlidte 2402867 2402866 2026-04-16T10:15:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T419635)', diff saved to https://phabricator.wikimedia.org/P90904 and previous config saved to /var/cache/conftool/dbconfig/20260416-101514-fceratto.json 2402867 wikitext text/x-wiki mt55jhd0n5i4dslc6444jq468jhnwy8 2402868 2402867 2026-04-16T10:15:33Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance 2402868 wikitext text/x-wiki 2hnl6my21mtlvn09api6rmdd6b70siq 2402869 2402868 2026-04-16T10:15:52Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2402869 wikitext text/x-wiki s3znwbom4gl7e191n9skbvavkzmw82q 2402871 2402869 2026-04-16T10:21:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P90905 and previous config saved to /var/cache/conftool/dbconfig/20260416-102143-fceratto.json 2402871 wikitext text/x-wiki 5wsy8tt7ucmnoxr62ubzpouawzpeolp 2402872 2402871 2026-04-16T10:31:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P90906 and previous config saved to /var/cache/conftool/dbconfig/20260416-103152-fceratto.json 2402872 wikitext text/x-wiki stfd0jp39wj1jvme1z3u0cqci8ktzay 2402874 2402872 2026-04-16T10:42:03Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T419961)', diff saved to https://phabricator.wikimedia.org/P90907 and previous config saved to /var/cache/conftool/dbconfig/20260416-104201-fceratto.json 2402874 wikitext text/x-wiki 53r2se39jfjvcr5ng0i0wq257azm64o 2402875 2402874 2026-04-16T10:42:24Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance 2402875 wikitext text/x-wiki i8jwwmevwv5m4hun4mhdo5ygorzn422 2402876 2402875 2026-04-16T10:42:33Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance 2402876 wikitext text/x-wiki 45soqbx7j41m01dre0f5v3rcxm8jveb 2402877 2402876 2026-04-16T10:42:42Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1161 (T419961)', diff saved to https://phabricator.wikimedia.org/P90908 and previous config saved to /var/cache/conftool/dbconfig/20260416-104240-fceratto.json 2402877 wikitext text/x-wiki 5utmosln0shjjvs8122g7pyk6jy4s66 2402878 2402877 2026-04-16T10:47:32Z Stashbot 7414 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: wikikube-ctrl2004.codfw.wmnet 2402878 wikitext text/x-wiki szruphm0wowmdx77b0m2gv4pa6fs0q6 2402879 2402878 2026-04-16T10:47:40Z Stashbot 7414 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: wikikube-ctrl2005.codfw.wmnet 2402879 wikitext text/x-wiki ibp2osq8klaj328z6ue2e6vz7ikjyp5 2402880 2402879 2026-04-16T10:50:13Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2402880 wikitext text/x-wiki m8wkexttq97ta69g6g6kiku9ztg5065 2402881 2402880 2026-04-16T10:50:43Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T419961)', diff saved to https://phabricator.wikimedia.org/P90909 and previous config saved to /var/cache/conftool/dbconfig/20260416-105040-fceratto.json 2402881 wikitext text/x-wiki tgu7mqekbej5yqltscwknxjqvdhrl1j 2402882 2402881 2026-04-16T10:50:56Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2402882 wikitext text/x-wiki slcy6rb2qpnnaoxjkusqxv0tkldtatj 2402883 2402882 2026-04-16T10:52:27Z Stashbot 7414 jmm@cumin2002: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: wikikube-ctrl2006.codfw.wmnet 2402883 wikitext text/x-wiki n3ihut4z23qqfe9tpeo7pwckcsxhsob 2402886 2402883 2026-04-16T10:55:53Z Stashbot 7414 moritzm: imported debdeploy 0.0.99.15 for bookworm-wikimedia (compat release for Cumin 6) 2402886 wikitext text/x-wiki 2n8xtkkncga4ohmo5qlap7ee85ike0p 2402887 2402886 2026-04-16T10:56:15Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2402887 wikitext text/x-wiki d09pr9vw4k8j9oq9mulqi2vjj2bbjxa 2402888 2402887 2026-04-16T10:56:46Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2402888 wikitext text/x-wiki abfioi8jnmoe6t0sb0px0hrnhnmwmic 2402889 2402888 2026-04-16T10:57:15Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply 2402889 wikitext text/x-wiki 3ggudrrpj9pw4ci03ini19uotvq74cx 2402890 2402889 2026-04-16T10:57:47Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply 2402890 wikitext text/x-wiki mw5hn7coj3cxzdhast8ftvl1iqzehp3 2402891 2402890 2026-04-16T11:00:51Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P90910 and previous config saved to /var/cache/conftool/dbconfig/20260416-110049-fceratto.json 2402891 wikitext text/x-wiki 0x4t1npjhshiptq254lw5scb4ddvcw2 2402893 2402891 2026-04-16T11:07:37Z Stashbot 7414 moritzm: updating debdeploy on bookworm to 0.0.99.15 2402893 wikitext text/x-wiki t8fohan7y9y241qc0cdfpjivx3otmeu 2402896 2402893 2026-04-16T11:10:20Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . 2402896 wikitext text/x-wiki 0z5t7vm5wlex1r2e2fb262uw2nk4qaa 2402897 2402896 2026-04-16T11:11:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P90911 and previous config saved to /var/cache/conftool/dbconfig/20260416-111058-fceratto.json 2402897 wikitext text/x-wiki kbh1pxpvrcrkh17a6ty5367zesn5zid 2402898 2402897 2026-04-16T11:16:20Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2402898 wikitext text/x-wiki 6bg7unkg1rrgl1y8m92wgt241t3ozyd 2402899 2402898 2026-04-16T11:16:50Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2402899 wikitext text/x-wiki 2akzygfa9ke43aw0yyrrgpktlleyf2e 2402900 2402899 2026-04-16T11:19:17Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2402900 wikitext text/x-wiki h9dqmlg1baj9swlqrqdhjqfssrcupqa 2402901 2402900 2026-04-16T11:19:53Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply 2402901 wikitext text/x-wiki andtgozdjmpc9frppehrsdcbib50vxj 2402902 2402901 2026-04-16T11:21:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T419961)', diff saved to https://phabricator.wikimedia.org/P90913 and previous config saved to /var/cache/conftool/dbconfig/20260416-112105-fceratto.json 2402902 wikitext text/x-wiki hoc8kdee4ezlnl9x93i65qug9kpk4hp 2402903 2402902 2026-04-16T11:21:29Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance 2402903 wikitext text/x-wiki rlyreoqjl7ym61ed04h4wwqu1njwr97 2402904 2402903 2026-04-16T11:21:38Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1185 (T419961)', diff saved to https://phabricator.wikimedia.org/P90914 and previous config saved to /var/cache/conftool/dbconfig/20260416-112136-fceratto.json 2402904 wikitext text/x-wiki ia8jzt7miupswx9wuies7d0mwz1agcx 2402905 2402904 2026-04-16T11:23:01Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{ml-serve1012.eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) 2402905 wikitext text/x-wiki b9373cxewif6ib7243wfxq84qrod1nl 2402906 2402905 2026-04-16T11:23:04Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1012.eqiad.wmnet 2402906 wikitext text/x-wiki tuku41kwvrn5x960rztr012ph42srhi 2402907 2402906 2026-04-16T11:28:03Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply 2402907 wikitext text/x-wiki 390bejs6x4dkdx6edzwuu2occ11ma8a 2402908 2402907 2026-04-16T11:28:36Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply 2402908 wikitext text/x-wiki n9y2zrwc9n8xssdr05ec2o2v3q0z4ej 2402909 2402908 2026-04-16T11:28:41Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply 2402909 wikitext text/x-wiki 8mgvzym7cp3qyvrz9j4mz7qybjp5i0u 2402910 2402909 2026-04-16T11:29:12Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply 2402910 wikitext text/x-wiki 6iekecivbf73pgpcb2jdtsxxug8q4lr 2402911 2402910 2026-04-16T11:29:18Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply 2402911 wikitext text/x-wiki 6ya5rlb0pzm1h06ihynd012r81sxdk7 2402912 2402911 2026-04-16T11:30:00Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply 2402912 wikitext text/x-wiki a4bzemborrz00453ngj8sw19enszpil 2402913 2402912 2026-04-16T11:30:06Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply 2402913 wikitext text/x-wiki pveggg6fu0fnxo7qn53l70grp9ss62f 2402914 2402913 2026-04-16T11:30:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T419961)', diff saved to https://phabricator.wikimedia.org/P90915 and previous config saved to /var/cache/conftool/dbconfig/20260416-113005-fceratto.json 2402914 wikitext text/x-wiki j1ndz0o6y17pb7l0w2aub89gr6luym6 2402915 2402914 2026-04-16T11:30:57Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply 2402915 wikitext text/x-wiki spj8batmde48sora9bgujwn9yzue1gh 2402916 2402915 2026-04-16T11:31:03Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply 2402916 wikitext text/x-wiki bb0pp1z1ixe9t7gndp11w1sx2ywqzbc 2402917 2402916 2026-04-16T11:31:32Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply 2402917 wikitext text/x-wiki eayszwwm5yy0yl9f1nbdjjk4spwypxr 2402918 2402917 2026-04-16T11:31:55Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply 2402918 wikitext text/x-wiki g6yhav5o0pg8nzmutjvyptx33li6829 2402919 2402918 2026-04-16T11:32:29Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply 2402919 wikitext text/x-wiki nlpa3n8f0fes3ra3f4g1ezuz116167s 2402920 2402919 2026-04-16T11:32:39Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply 2402920 wikitext text/x-wiki 7dn6q256bi6ehzrcmbwr8q78e3exug5 2402921 2402920 2026-04-16T11:33:14Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1012.eqiad.wmnet 2402921 wikitext text/x-wiki dr1e4wydoemxd0bot5bqa27m7iq6kyz 2402922 2402921 2026-04-16T11:33:29Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply 2402922 wikitext text/x-wiki lgw1bcmlyr35x6pd6bpm9u8xgatnye5 2402923 2402922 2026-04-16T11:38:44Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1012.eqiad.wmnet 2402923 wikitext text/x-wiki kkf5who23tjbo6a2qxwwk82kvea56jb 2402924 2402923 2026-04-16T11:38:46Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1012.eqiad.wmnet 2402924 wikitext text/x-wiki c5h6155ep4nydc3yix0trfelxhs9t8o 2402925 2402924 2026-04-16T11:38:48Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{ml-serve1012.eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) 2402925 wikitext text/x-wiki 72d1wjoqdgqxnvaoddu57sjrwxg1r0k 2402926 2402925 2026-04-16T11:40:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P90916 and previous config saved to /var/cache/conftool/dbconfig/20260416-114014-fceratto.json 2402926 wikitext text/x-wiki jwzbswty3vswr1pyj6abvijcy4899e2 2402927 2402926 2026-04-16T11:42:55Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply 2402927 wikitext text/x-wiki jet4m4ex9qfidmt7ldphtu1zzmmwt3s 2402928 2402927 2026-04-16T11:43:21Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply 2402928 wikitext text/x-wiki n4963x7ak01kmo513tg98niq8wtptax 2402929 2402928 2026-04-16T11:50:26Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P90918 and previous config saved to /var/cache/conftool/dbconfig/20260416-115024-fceratto.json 2402929 wikitext text/x-wiki 90pq9a1x8zvkpcoduu03l9eada5whcl 2402930 2402929 2026-04-16T11:50:44Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1247.eqiad.wmnet with reason: Maintenance 2402930 wikitext text/x-wiki nb97ro36w6w00tmd2pfdqul3v2dsn88 2402931 2402930 2026-04-16T11:50:59Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db1247 (T419635)', diff saved to https://phabricator.wikimedia.org/P90919 and previous config saved to /var/cache/conftool/dbconfig/20260416-115055-fceratto.json 2402931 wikitext text/x-wiki qwcyfq9l9o238d6szihjlktu9zt78v5 2402932 2402931 2026-04-16T11:53:04Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. 2402932 wikitext text/x-wiki g23v9987zgt4jtzanqdsvct8lltesjq 2402933 2402932 2026-04-16T11:53:09Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{ml-serve1013.eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) 2402933 wikitext text/x-wiki m4ux0nqktjimr547c6lecrkrw9eswi2 2402934 2402933 2026-04-16T11:53:13Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1013.eqiad.wmnet 2402934 wikitext text/x-wiki tqutepuqy9mjkkk4zx6drgcpde9nskp 2402936 2402934 2026-04-16T11:56:05Z Stashbot 7414 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. 2402936 wikitext text/x-wiki 21x8bflfurzwpbvplqnym9xvfakdlow 498 15050 2402595 2402007 2026-04-15T19:57:38Z Stashbot 7414 wmbot~lokal-profil@tools-bastion-15: Deploy 3df98c8 2402595 wikitext text/x-wiki 5qarjcydhveoinxns48kq1mffvkvoxg 2402620 2402595 2026-04-15T20:36:43Z Stashbot 7414 wmbot~lokal-profil@tools-bastion-15: Deploy 244e67c 2402620 wikitext text/x-wiki 1u47zz4xvroigedgjb91pbygeq1efnn 0 17290 2402466 2402102 2026-04-15T15:33:26Z Stashbot 7414 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/100 2402466 wikitext text/x-wiki 5o1h3ltinx95ztx7ns2sarq4u1iirwx 2402485 2402466 2026-04-15T16:39:29Z Stashbot 7414 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/commit/127d783b2176ac60b646a5fa4f1b1a872ca66340 2402485 wikitext text/x-wiki lo7nrkp0zm1y7ysd9efq1nhv22wwce0 2402507 2402485 2026-04-15T17:32:32Z Stashbot 7414 bd808: Unblock 176.86.0.0/16 2402507 wikitext text/x-wiki spvu3p6p8qn2mih7yu1tylkot40ec78 2402520 2402507 2026-04-15T17:39:04Z Stashbot 7414 bd808: Unblock 46.128.0.0/16 2402520 wikitext text/x-wiki avkfznch49fp29dxznjlyr5qfpjcs1y 2402537 2402520 2026-04-15T17:44:20Z Stashbot 7414 bd808: Unblock 176.0.0.0/13 2402537 wikitext text/x-wiki gcp89x0ilsez8rcick15iflgs9cmb8h 2402600 2402537 2026-04-15T20:05:53Z Stashbot 7414 James_F: Zuul: Configure REL1_46 CI, for T423257 2402600 wikitext text/x-wiki 5wyzlfii792f84db62l2cym9y9qs2a5 2402894 2402600 2026-04-16T11:07:50Z Stashbot 7414 Silvan_WMDE: sudo -u jenkins-deploy rm -fR /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node24/ # run on integration-castor06.integration.eqiad1.wikimedia.cloud 2402894 wikitext text/x-wiki nbzdzr7mpdtigwzl2evcfktka2hf8de 498 30942 2402435 2402080 2026-04-15T15:02:53Z Stashbot 7414 godog: deploy per-service oslo.messaging shared memory file name - T423378 2402435 wikitext text/x-wiki myvjhdrw9s0z4zu3r1vdrz8dbczc1fe 2402750 2402435 2026-04-16T06:43:20Z Stashbot 7414 godog: roll-restart nova-api to pick up changes - T423378 2402750 wikitext text/x-wiki hwishniry6bvryj0qv6gwhitu905rfi 3 442042 2402666 2091042 2026-04-15T22:09:25Z SSapaty (WMF) 28955 2402666 wikitext text/x-wiki ene2r715wrhwyopbaw2o5c7redngjbp 2402673 2402666 2026-04-15T22:36:25Z Mahir256 13837 /* Phab Ticket: T318624 */ 2402673 wikitext text/x-wiki hbu33681j0becx7vxk47bnmofc0bpuw 2402688 2402673 2026-04-15T23:23:54Z JJMC89 7474 h2 2402688 wikitext text/x-wiki tl6bvcax6ybumu846hd7yx49ihjh5wx 3 442061 2402663 2144552 2026-04-15T22:06:50Z SSapaty (WMF) 28955 2402663 wikitext text/x-wiki 69fc7gtlrdtjdolavgm9s2jv3ifhclr 2402687 2402663 2026-04-15T23:23:39Z JJMC89 7474 h2 2402687 wikitext text/x-wiki 36f34xbdc1z0586ymktowohi2uxsnnv 12 442277 2402797 2396061 2026-04-16T08:23:48Z Yug 1705 2402797 wikitext text/x-wiki tnezpfxcjf2e3kuxy8trpyuw9j77wbm 2402798 2402797 2026-04-16T08:26:02Z Yug 1705 Add key tools to know + general descriptions 2402798 wikitext text/x-wiki sennwu23mrm28j3m2fd493awyhrzrxb 2402800 2402798 2026-04-16T08:28:01Z Yug 1705 Maybe we need a table ? 2402800 wikitext text/x-wiki l1i2rp9d1o7if938tvv9ke1s3s6e5lt 0 443467 2402390 2178734 2026-04-15T14:16:00Z MLechvien-WMF 49197 Deprecating the page 2402390 wikitext text/x-wiki gjrlpa5m1okcath3l8ntqmnfqe1poed 0 443852 2402816 2371280 2026-04-16T09:05:16Z BJensen-WMF 48942 /* Known problems */ Indicate that docker-registry services need to be restarted when APUS changes pooled status. 2402816 wikitext text/x-wiki ooxnj7v2nim9cb4kol9b0ubnp524lrh 0 447311 2402411 2393729 2026-04-15T14:46:01Z KSoučková-WMF 35993 /* Puppet/hiera */ mention adding to deployment_server profile 2402411 wikitext text/x-wiki emicrlq74kobkty21j8u29p1cpbymok 0 449062 2402392 2178738 2026-04-15T14:21:47Z MLechvien-WMF 49197 Deprecating the page 2402392 wikitext text/x-wiki kxtzwvevvzm8zkg3drf5hlh2gq50iyh 0 449063 2402806 2210372 2026-04-16T08:49:54Z MLechvien-WMF 49197 Updating to remove usage of the scorecard 2402806 wikitext text/x-wiki 3vqezbwedllvecxvk44hpqbmxr1gpcy 0 449160 2402369 2402214 2026-04-15T13:47:45Z Dexbot 30554 Bot: Updating the report 2402369 wikitext text/x-wiki 64attre6s08waqi2dm8owlgt3dmxpwd 2402430 2402369 2026-04-15T15:00:00Z Dexbot 30554 Bot: Updating the report 2402430 wikitext text/x-wiki 6ero2br8xo7vmplmto7r883ubca1rz7 2402695 2402430 2026-04-16T00:01:23Z Dexbot 30554 Bot: Updating the report 2402695 wikitext text/x-wiki jp35fr9ku20o4asw8z7iga470iasm64 2402697 2402695 2026-04-16T00:34:28Z Dexbot 30554 Bot: Updating the report 2402697 wikitext text/x-wiki cex351909ucmy6a15f090ug7k37aaes 2402701 2402697 2026-04-16T00:58:31Z Dexbot 30554 Bot: Updating the report 2402701 wikitext text/x-wiki aqdf2808q7dfnptiea0ekah37ebzip7 2402713 2402701 2026-04-16T02:22:48Z Dexbot 30554 Bot: Updating the report 2402713 wikitext text/x-wiki jnp4a6drm7woscritk9izmr772hs0ou 2402765 2402713 2026-04-16T07:29:27Z Dexbot 30554 Bot: Updating the report 2402765 wikitext text/x-wiki cimhxsexk742mx8vkyriqfph2x1xp3y 2402821 2402765 2026-04-16T09:11:42Z Dexbot 30554 Bot: Updating the report 2402821 wikitext text/x-wiki 4pxdn4609b5j4dpknhdd7meld0oyazd 0 449219 2402579 2397071 2026-04-15T18:47:28Z LSandergreen-WMF 46345 remove drush 2402579 wikitext text/x-wiki mpbrewalyn2g3hoy4k78fwr55wfghlp 0 449315 2402679 2344498 2026-04-15T22:53:03Z LSandergreen-WMF 46345 /* Testing */ 2402679 wikitext text/x-wiki 7zqomxscrvm2jxmldu4063mf9db9wrp 2402691 2402679 2026-04-15T23:32:06Z LSandergreen-WMF 46345 /* Testing */ 2402691 wikitext text/x-wiki pzzqxg4u1a4w5kdph8pwnkv712xk8k8 0 451448 2402583 2401056 2026-04-15T19:39:10Z Jdforrester (WMF) 468 /* How to debug an issue */ Add example direct evaluator calls. 2402583 wikitext text/x-wiki sk9zurq7ge2mhn9eb288ndmft29j0qy 2402645 2402583 2026-04-15T21:05:54Z Jdforrester (WMF) 468 Slim down the helmfile context, we don't need ±5 lines. 2402645 wikitext text/x-wiki elbdmp4tz9rpn5ibspbhkn9ehn3bmkt 116 453906 2402493 2401859 2026-04-15T17:03:13Z Gitlabaccountapprovalbot 37332 lahari was rejected. 2402493 wikitext text/x-wiki eeql8dz36jk3hn9o0skjgtay92naw0x 0 454523 2402459 2398202 2026-04-15T15:27:08Z KHerron (WMF) 10463 /* Architecture */ 2402459 wikitext text/x-wiki 1k907a7j3iflyrdh130zmi2t6b6ps14 0 455715 2402345 2384703 2026-04-15T13:13:57Z BJensen-WMF 48942 Add instructions to restart docker-registry services when the pooled state of apus changes. 2402345 wikitext text/x-wiki 2umz3pu0ejjxffk933jn9d9pkpyqjx8 2402346 2402345 2026-04-15T13:14:59Z BJensen-WMF 48942 /* Pooling / Depooling */ Add a link for context. 2402346 wikitext text/x-wiki mstuvf11m8fo36fi6i4h8v71qyox0ir 2402813 2402346 2026-04-16T09:01:17Z BJensen-WMF 48942 /* Pooling / Depooling */ Correct cumin command, add coordination instructions. 2402813 wikitext text/x-wiki 28hm8lfqpskx8pk18glf47wx96osbpd 0 455948 2402935 2378765 2026-04-16T11:53:52Z AYounsi (WMF) 10040 add inplace instructions 2402935 wikitext text/x-wiki 4qr5j9t7xjt1ronhppzv674il4itwug 0 459226 2402884 2389868 2026-04-16T10:53:24Z BTullis (WMF) 25295 /* Redeploying all instances */ Added a diff version 2402884 wikitext text/x-wiki gfv6oz5lazut8gts3t8ooae30sbmdup 498 459243 2402842 2402081 2026-04-16T09:38:34Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502989226 (https://github.com/cluebotng/component-configs/commits/d1caa7e619b2f6df59cbdb32d5bdf150ceacfa19) 2402842 wikitext text/x-wiki ixk571wr51jcj1zsfgu23z0z4gadnbk 2402895 2402842 2026-04-16T11:09:26Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24506838480 (https://github.com/cluebotng/component-configs/commits/db30680b466b66aed23b2f87989cbe2e9fd90511) 2402895 wikitext text/x-wiki 2qxkdczm337u20ssajmg602m23o2l3o 498 459244 2402840 2401416 2026-04-16T09:36:42Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502903296 (https://github.com/cluebotng/component-configs/commits/09046ab7e1f4831d3acaf275abbb61a756291a24) 2402840 wikitext text/x-wiki e4yhh810f53s1ti03fkt1cpp3iclvc0 0 459265 2402453 2397628 2026-04-15T15:23:33Z KHerron (WMF) 10463 /* What use cases lead to a negative error budget? */ 2402453 wikitext text/x-wiki 23d58p4m0khj7hjxrcdq133eqigmmki 0 459283 2402873 2349325 2026-04-16T10:34:26Z FCeratto-WMF 39104 Update conf and suggestions 2402873 wikitext text/x-wiki gk52irt115al92bkp61fj2wtv2ivqu0 0 459699 2402661 2402115 2026-04-15T21:50:30Z Stashbot 7414 eileen: civicrm upgraded from 6f33b6d0 to 801847a7 2402661 wikitext text/x-wiki takksaanirnjxbiq3d7hsc8xxxzgcuj 2402715 2402661 2026-04-16T02:39:25Z Stashbot 7414 ejegg: standalone (IPN listener) SmashPig upgraded from 61fee241 to f1b3f3d9 2402715 wikitext text/x-wiki 98sltdi12a2r18nuq5ssmvp5d8ky0iv 0 459948 2402375 2395676 2026-04-15T13:52:09Z BTullis (WMF) 25295 /* Kubernetes API (control plane) */ typo 2402375 wikitext text/x-wiki n38khow3mijwqs83vmjuei7vtcfkdfv 2402388 2402375 2026-04-15T14:15:29Z BTullis (WMF) 25295 /* Kubernetes API (control plane) */ Add some detail based on experience with dse-k8s 2402388 wikitext text/x-wiki duplga67guder4pvvlodngs81514kw5 2402391 2402388 2026-04-15T14:19:36Z BTullis (WMF) 25295 /* Preparations */ typo 2402391 wikitext text/x-wiki egd6wu7swxop0ckjkvlxwbj86f49d12 2 460033 2402586 2402045 2026-04-15T19:52:11Z EEvans (WMF) 5035 EEvans (WMF) moved page [[User:EEvans (WMF)/Scratch/Adding Cassandra Hosts]] to [[User:EEvans (WMF)/Scratch/Provisioning Cassandra Hosts]]: More apropos name 2402045 wikitext text/x-wiki 3thoumdl8ohl8ukgwiu1b09nj0hmsvo 2402690 2402586 2026-04-15T23:29:54Z EEvans (WMF) 5035 /* Cleanups */ stubbed 2402690 wikitext text/x-wiki qlg1otgxty9t3vf4fx45jw9ghtqeyk9 2402693 2402690 2026-04-15T23:51:41Z EEvans (WMF) 5035 /* Configure for reuse */ 2402693 wikitext text/x-wiki 524yxuuj9fuiokv6byskayc2r9th24u 2402694 2402693 2026-04-15T23:52:44Z EEvans (WMF) 5035 /* Configure for reuse */ nit 2402694 wikitext text/x-wiki l8mf7ox942l8y3il426i9zdf3y0zr4d 0 460059 2402396 2026-04-15T14:31:57Z JMonton-WMF 47887 First version of the dbt documentation 2402396 wikitext text/x-wiki 19o72lmujcljaxncy6nw16fwquiz54e 2402409 2402396 2026-04-15T14:44:03Z JMonton-WMF 47887 title change 2402409 wikitext text/x-wiki cdi26yc61hi71vxjgaw5l818si6vlpp 2402422 2402409 2026-04-15T14:56:18Z GGoncalves-WMF 42848 /* Install and run on stat boxes */ Clarify how to run the setup commands. 2402422 wikitext text/x-wiki hr3lel6dcxrmn4apxig4c21ov2zy7eh 2 460060 2402585 2026-04-15T19:50:56Z EEvans (WMF) 5035 index scratch docs 2402585 wikitext text/x-wiki 9ksa6cy3iolsogtpra6i22v4x7jxbm9 2402588 2402585 2026-04-15T19:52:52Z EEvans (WMF) 5035 EEvans (WMF) moved page [[User:EEvans (WMF)/Scratch/]] to [[User:EEvans (WMF)/Scratch]]: Misspelled title 2402585 wikitext text/x-wiki 9ksa6cy3iolsogtpra6i22v4x7jxbm9 2402590 2402588 2026-04-15T19:53:22Z EEvans (WMF) 5035 renamed page 2402590 wikitext text/x-wiki 0ai8nc6zwkgw8a43vv9se9xeebv3s07 2 460061 2402587 2026-04-15T19:52:11Z EEvans (WMF) 5035 EEvans (WMF) moved page [[User:EEvans (WMF)/Scratch/Adding Cassandra Hosts]] to [[User:EEvans (WMF)/Scratch/Provisioning Cassandra Hosts]]: More apropos name 2402587 wikitext text/x-wiki 3t2b6180gimpjtqktai38lsz68ujh6b 2 460062 2402589 2026-04-15T19:52:52Z EEvans (WMF) 5035 EEvans (WMF) moved page [[User:EEvans (WMF)/Scratch/]] to [[User:EEvans (WMF)/Scratch]]: Misspelled title 2402589 wikitext text/x-wiki t7qe3sj2ioz50swoqbkduq0ep94tb3v 1 460063 2402662 2026-04-15T22:05:17Z Peter F. Patel-Schneider 42591 /* fundamental error in document */ new section 2402662 wikitext text/x-wiki o3lnbdouttpbesjgk81i54jv1awuwxz 2402860 2402662 2026-04-16T10:00:26Z So9q 22032 /* fundamental error in document */ Reply 2402860 wikitext text/x-wiki dykvqh78kvqiqgyj5djlt43cpcspie4 2402870 2402860 2026-04-16T10:16:58Z Peter F. Patel-Schneider 42591 /* fundamental error in document */ Reply 2402870 wikitext text/x-wiki 3qp64mz4fcq5tezghkmpdjogciopm2z 3 460064 2402669 2026-04-15T22:14:47Z SSapaty (WMF) 28955 Created page with "=== Phab Ticket: T318624 === Hello Jaya, we've had this security related [[:phab:T318624 | ticket]] that's been opened for a while for the bodh/bodh-backend tools. You are listed as one of the maintainers. Kindly take a look and address the issues. They tool may be disabled after 1-2 weeks. ~~~~" 2402669 wikitext text/x-wiki 9cfw4appoogu6rxu6qpc6jnbv4hoa68 2402689 2402669 2026-04-15T23:24:04Z JJMC89 7474 h2 2402689 wikitext text/x-wiki 68lykz2rbw510v1vn8gqfjggde50qj9