Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.2 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2413339 2413218 2026-05-13T12:07:06Z Stashbot 7414 godog: resume restarting webservices using default memory requests - T420565 2413339 wikitext text/x-wiki qbcae05ksmoh8ju8w92t8ptbncoreyj 0 4108 2413345 2413272 2026-05-13T12:37:47Z ScheduleDeploymentBot 37566 Add [[gerrit:1286890]] to Wednesday, May 13 UTC afternoon backport window 2413345 wikitext text/x-wiki roam2p1iipl6w8am0xs8q7fujyzutiv 2413353 2413345 2026-05-13T12:44:01Z ScheduleDeploymentBot 37566 Add [[gerrit:1286897]] to Wednesday, May 13 UTC afternoon backport window 2413353 wikitext text/x-wiki 1cwip84en5gmkvsprx5vs41s23dny7v 2413354 2413353 2026-05-13T12:45:25Z ScheduleDeploymentBot 37566 Add [[gerrit:1286891]] to Wednesday, May 13 UTC afternoon backport window 2413354 wikitext text/x-wiki 1gxbifaz492pozl0m1o17vmky7kqb3n 2413355 2413354 2026-05-13T12:45:31Z ScheduleDeploymentBot 37566 Add [[gerrit:1286892]] to Wednesday, May 13 UTC afternoon backport window 2413355 wikitext text/x-wiki 9u9xq2tv888g0wpuo0f10p8nnunjbue 2413362 2413355 2026-05-13T13:06:10Z ScheduleDeploymentBot 37566 Add [[gerrit:1286917]] to Wednesday, May 13 UTC afternoon backport window 2413362 wikitext text/x-wiki ax5ry20a2tzr9ywxa62sfjy87h9im06 2413469 2413362 2026-05-13T15:38:55Z ScheduleDeploymentBot 37566 Add [[gerrit:1269442]] to Thursday, May 14 UTC afternoon backport window 2413469 wikitext text/x-wiki pjqg9bxny3fz2ts4y5gdlb0ui74utxx 2413479 2413469 2026-05-13T15:50:47Z Lucas Werkmeister (WMDE) 10037 /* {{Deployment_day|date=2026-05-14}} */ remove myself from backport+config window, public holiday in Germany 2413479 wikitext text/x-wiki 6ssk8nl13yywtwwp01mso6ko21g0461 2413482 2413479 2026-05-13T16:04:37Z ScheduleDeploymentBot 37566 Add [[gerrit:1192921]] to Thursday, May 14 UTC late backport window 2413482 wikitext text/x-wiki 6dld0b2ikuertt9wegmiu4akfiqczig 2413483 2413482 2026-05-13T16:05:04Z ScheduleDeploymentBot 37566 Add [[gerrit:1286974]] to Thursday, May 14 UTC late backport window 2413483 wikitext text/x-wiki r4fm7m308g20sz9lhzvhgy69nvj4pnp 2413484 2413483 2026-05-13T16:05:22Z ScheduleDeploymentBot 37566 Add [[gerrit:1286975]] to Thursday, May 14 UTC late backport window 2413484 wikitext text/x-wiki j1gij3yjewnzg390kbtyg4fzi7nt9ea 2413498 2413484 2026-05-13T16:44:20Z ScheduleDeploymentBot 37566 Add [[gerrit:1286981]] to Wednesday, May 13 UTC late backport window 2413498 wikitext text/x-wiki cb2z5ok52u7ffr7edp04qeo282mikf1 2413538 2413498 2026-05-13T18:25:23Z ScheduleDeploymentBot 37566 Add [[gerrit:1286997]] to Wednesday, May 13 UTC late backport window 2413538 wikitext text/x-wiki e1j2prhwggb18n45v3t9daihz7pjqpo 2413545 2413538 2026-05-13T19:02:38Z WBrown (WMF) 37696 /* {{Deployment_day|date=2026-05-14}} */ + 2413545 wikitext text/x-wiki 5rzaxm9crebc3sq04bwb2hv2ln20vf0 2413550 2413545 2026-05-13T19:55:49Z ScheduleDeploymentBot 37566 Add [[gerrit:1287006]] to Wednesday, May 13 UTC late backport window 2413550 wikitext text/x-wiki 2x33ek1aduu9s87s8gvgp2sv3sv4pnq 2413597 2413550 2026-05-14T00:17:11Z ScheduleDeploymentBot 37566 Add [[gerrit:1287043]] to Thursday, May 14 UTC afternoon backport window 2413597 wikitext text/x-wiki 4387t9itncnbj2ngr35mggo4kno04g0 2413638 2413597 2026-05-14T09:17:01Z ScheduleDeploymentBot 37566 Add [[gerrit:1287363]] to Thursday, May 14 UTC afternoon backport window 2413638 wikitext text/x-wiki 1lhaiwzgg5jcziv3ii14awad05nzks2 2413650 2413638 2026-05-14T09:59:49Z ScheduleDeploymentBot 37566 Add [[gerrit:1287368]] to Thursday, May 14 UTC afternoon backport window 2413650 wikitext text/x-wiki nb7z78ja9kjin4tzyi963lv7hizai0l 2413651 2413650 2026-05-14T10:00:27Z ScheduleDeploymentBot 37566 Add [[gerrit:1287367]] to Thursday, May 14 UTC afternoon backport window 2413651 wikitext text/x-wiki gttovicvasnpk2ttl5wtytcqjty0bt5 0 4120 2413488 2404336 2026-05-13T16:14:21Z BCornwall-WMF 31521 /* Setting up your SSH config */ mention replacing prod.key too 2413488 wikitext text/x-wiki j58p3qgvszfy6g1w9x0t58u5znq50kb 498 6726 2413490 2413199 2026-05-13T16:16:24Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway 2413490 wikitext text/x-wiki bi0b7k9g30usgbh3k3lvrwutdyxmseh 2413491 2413490 2026-05-13T16:16:26Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component istio-gateway 2413491 wikitext text/x-wiki lhdlz0tq9dio8ftiiqtest3llwmor4u 2413492 2413491 2026-05-13T16:16:42Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway 2413492 wikitext text/x-wiki ejod6ufvoj33t0i8ev1jp6keyesaepz 2413493 2413492 2026-05-13T16:16:50Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway 2413493 wikitext text/x-wiki lb8kjlspvbugbmo7wqaxrenwjh2zxl0 2413495 2413493 2026-05-13T16:27:34Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway 2413495 wikitext text/x-wiki 06xgq8fq948b4m7h6xmo44o5rw8a64z 2413499 2413495 2026-05-13T16:44:23Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway 2413499 wikitext text/x-wiki 8u4021chcmp1dhybtp6ma343jmfiklv 2413500 2413499 2026-05-13T16:47:51Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway 2413500 wikitext text/x-wiki pi639rm0fe7kkf3z3tnv9wnawn9knjm 2413504 2413500 2026-05-13T17:04:00Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway 2413504 wikitext text/x-wiki 2jmhy4w8ysxgx3cmsne8s2el4evauw5 2413674 2413504 2026-05-14T10:49:41Z Stashbot 7414 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway (T425479) 2413674 wikitext text/x-wiki 8yjcgv179dl6mn7cqbdm703658zql4j 2413684 2413674 2026-05-14T11:04:11Z Stashbot 7414 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway (T425479) 2413684 wikitext text/x-wiki khn2ydgg83xtqycwp7yeceksr2rqj7j 0 7919 2413338 2413337 2026-05-13T12:02:38Z Stashbot 7414 topranks: add ibgp peering between cr1-esams and cr2-esams over loopback IPs T424611 2413338 wikitext text/x-wiki koxk9yb0xkl97wom6lhvm7tr8zwjs29 2413340 2413338 2026-05-13T12:09:51Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetboard2003.codfw.wmnet 2413340 wikitext text/x-wiki 14wcf6eydhpzjn1e0dbzc9dqriv0iyb 2413341 2413340 2026-05-13T12:13:42Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard2003.codfw.wmnet 2413341 wikitext text/x-wiki 0o7fq2o8boun6cvjj8628nvkt8b70so 2413342 2413341 2026-05-13T12:37:22Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1236: Migration of db1236.eqiad.wmnet completed 2413342 wikitext text/x-wiki azbtwjhudmx4vttk1kfp65wuub5m46g 2413343 2413342 2026-05-13T12:37:24Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) 2413343 wikitext text/x-wiki s53nl7yfkil7915xhqit0m48gspu9ni 2413346 2413343 2026-05-13T12:38:42Z Stashbot 7414 topranks: add ibgp peering between cr1-magru and cr2-magru over loopback IPs T424611 2413346 wikitext text/x-wiki nbhup4txyhmn1v2vuvr0i37hevoihqg 2413347 2413346 2026-05-13T12:40:54Z Stashbot 7414 fabfur: depool cp7001 to test haproxy-awslc (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1286526) (T419825) 2413347 wikitext text/x-wiki 06vcahz0u3kt28rcex9m15crau8j4s4 2413348 2413347 2026-05-13T12:41:14Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp7001.* 2413348 wikitext text/x-wiki joy5qnuh7asp389x4s3jkanafyzm5ky 2413352 2413348 2026-05-13T12:43:56Z Stashbot 7414 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1286884|Fix TypeError on saving userrights interwiki (T426185)]] 2413352 wikitext text/x-wiki 754ehnalhg0p9iwaeh2rsytas91ztgb 2413356 2413352 2026-05-13T12:45:52Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Backport for [[gerrit:1286884|Fix TypeError on saving userrights interwiki (T426185)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2413356 wikitext text/x-wiki qnbipfml4nrox6mgpjiqup24g5vzv3c 2413357 2413356 2026-05-13T12:46:25Z Stashbot 7414 mszwarc@deploy1003: mszwarc: Continuing with deployment 2413357 wikitext text/x-wiki 6u5z4srfq2vh6v5po727568yl09g0p8 2413358 2413357 2026-05-13T12:50:38Z Stashbot 7414 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1286884|Fix TypeError on saving userrights interwiki (T426185)]] (duration: 06m 42s) 2413358 wikitext text/x-wiki 9q863yxhcgtoiwkn3pw3w5p799njt4c 2413359 2413358 2026-05-13T13:03:55Z Stashbot 7414 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1286359|Add configurable user-agent and sparql endpoint url (T425389)]] 2413359 wikitext text/x-wiki 1uic0cpm3zem1sitcmo4314beygyqoh 2413360 2413359 2026-05-13T13:04:32Z Stashbot 7414 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=codfw 2413360 wikitext text/x-wiki 0kmnbra9uc77a0i36xhc72moc866nzu 2413361 2413360 2026-05-13T13:05:55Z Stashbot 7414 sbisson@deploy1003: sbisson: Backport for [[gerrit:1286359|Add configurable user-agent and sparql endpoint url (T425389)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2413361 wikitext text/x-wiki 1m3l9rmesaob3djf5xdyd3ebcs8rc34 2413363 2413361 2026-05-13T13:08:00Z Stashbot 7414 sbisson@deploy1003: sbisson: Continuing with deployment 2413363 wikitext text/x-wiki g606g3ipao4s7l602kn5g6w1xmt21z7 2413364 2413363 2026-05-13T13:12:14Z Stashbot 7414 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1286359|Add configurable user-agent and sparql endpoint url (T425389)]] (duration: 08m 18s) 2413364 wikitext text/x-wiki 763023mx4eawzg6478jgn1ru9pjus06 2413366 2413364 2026-05-13T13:18:21Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki-root1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2413366 wikitext text/x-wiki agp4a4gs1nw8oxrvo7cbvrtbd4u6tir 2413370 2413366 2026-05-13T13:25:01Z Stashbot 7414 moritzm: installing openjdk-11 security updates 2413370 wikitext text/x-wiki rk4unfua3msief9zbxa5l8qruja7oqz 2413371 2413370 2026-05-13T13:27:18Z Stashbot 7414 mfossati@deploy1003: Started scap sync-world: Backport for [[gerrit:1286518|[Share Highlight] Exclude section edit links, footnotes from selection (T423658)]], [[gerrit:1286838|Add robust color fallbacks for QuoteCard average-color styling (T425358)]], [[gerrit:1286839|Fixed card width (T425710)]], [[gerrit:1286844|Adjust image size to match fixed width (T425710)]], [[gerrit:1286846|ShareHighlight: exclude browsers th 2413371 wikitext text/x-wiki aickmerqzmzaqiwnyqbau2mt2v66l2u 2413372 2413371 2026-05-13T13:27:22Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki-root1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2413372 wikitext text/x-wiki im026slegwplz21v6wlmdzrunkagwru 2413373 2413372 2026-05-13T13:28:57Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.cassandra.roll-restart for nodes matching A:cassandra-dev: Java security update - jmm@cumin2002 2413373 wikitext text/x-wiki 8gmdpqxikn0ip9kva4u3xxwg96fffoo 2413374 2413373 2026-05-13T13:29:14Z Stashbot 7414 mfossati@deploy1003: jdlrobson, mfossati: Backport for [[gerrit:1286518|[Share Highlight] Exclude section edit links, footnotes from selection (T423658)]], [[gerrit:1286838|Add robust color fallbacks for QuoteCard average-color styling (T425358)]], [[gerrit:1286839|Fixed card width (T425710)]], [[gerrit:1286844|Adjust image size to match fixed width (T425710)]], [[gerrit:1286846|ShareHighlight: exclude browsers that d 2413374 wikitext text/x-wiki olz5defhmmcq1as8k2n263pxxzrr2tf 2413375 2413374 2026-05-13T13:36:18Z Stashbot 7414 mfossati@deploy1003: jdlrobson, mfossati: Continuing with deployment 2413375 wikitext text/x-wiki lrr1tcsye50axbd5x4ks2k71uqsl55i 2413376 2413375 2026-05-13T13:40:39Z Stashbot 7414 mfossati@deploy1003: Finished scap sync-world: Backport for [[gerrit:1286518|[Share Highlight] Exclude section edit links, footnotes from selection (T423658)]], [[gerrit:1286838|Add robust color fallbacks for QuoteCard average-color styling (T425358)]], [[gerrit:1286839|Fixed card width (T425710)]], [[gerrit:1286844|Adjust image size to match fixed width (T425710)]], [[gerrit:1286846|ShareHighlight: exclude browsers t 2413376 wikitext text/x-wiki 1nymtwhfakdnkq4s9lz5y4iq041yb5h 2413377 2413376 2026-05-13T13:42:34Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1284900|Completely disable MediaWiki page patrolling functions on German Wikipedia (T316393)]] 2413377 wikitext text/x-wiki g99e878rszfueyat3ckigl3y41bupnm 2413378 2413377 2026-05-13T13:44:30Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, codenamenoreste: Backport for [[gerrit:1284900|Completely disable MediaWiki page patrolling functions on German Wikipedia (T316393)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2413378 wikitext text/x-wiki 26w1ij495sk23oksx4tl9jadyjjt5wq 2413379 2413378 2026-05-13T13:45:57Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, codenamenoreste: Continuing with deployment 2413379 wikitext text/x-wiki ix41p57tq89hg3xiyjp05k73apgl1oz 2413380 2413379 2026-05-13T13:49:06Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:cassandra-dev: Java security update - jmm@cumin2002 2413380 wikitext text/x-wiki hi4l1pla2mjlk76dgh3t9rv6tih2an4 2413381 2413380 2026-05-13T13:50:10Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1284900|Completely disable MediaWiki page patrolling functions on German Wikipedia (T316393)]] (duration: 07m 36s) 2413381 wikitext text/x-wiki 0c3dxy4nnta73nuzn5f1u82w4xwesiv 2413382 2413381 2026-05-13T13:56:59Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host install3004.wikimedia.org 2413382 wikitext text/x-wiki 5sx443f2ofu1bny0qefi86gud9ya8b0 2413383 2413382 2026-05-13T13:58:00Z Stashbot 7414 fabfur: repooling cp7001 to test haproxy-awslc behavior (T419825) 2413383 wikitext text/x-wiki 3o7vztiaeem523yvwb8v56qdmt98r3u 2413384 2413383 2026-05-13T13:59:11Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1286890|ApiQueryGlobalUsers: Fix parsing logic for legacy log_params entries (T426033)]], [[gerrit:1286897|ApiQueryGlobalUsers: Fix parsing logic for legacy log_params entries (T426033)]], [[gerrit:1286891|Add 'Promise-Non-Write-API-Action' to $wgAllowedCorsHeaders (T425972)]], [[gerrit:1286892|Add 'Promise-Non-Write-API-Action' to $wgAll 2413384 wikitext text/x-wiki 8himti6kr7o8atlkz9nokla2deejs8x 2413385 2413384 2026-05-13T13:59:49Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-canary: Restart for upgrade to JVM 11.0.31 - eevans@cumin1003 2413385 wikitext text/x-wiki r0umrsgf9bh88qo0ahv0976ay66f1gy 2413386 2413385 2026-05-13T13:59:57Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/mathoid: apply 2413386 wikitext text/x-wiki g6kxzvhqm3mi8yf9z7t2tmwyekdjdow 2413387 2413386 2026-05-13T14:00:12Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host pki-root1002.eqiad.wmnet with OS trixie 2413387 wikitext text/x-wiki dw4za8vqctmivz303yvtgaurwcv0u8p 2413388 2413387 2026-05-13T14:00:17Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/mathoid: apply 2413388 wikitext text/x-wiki dgzn73fg39ymyaa3p61xi99b1emchqv 2413389 2413388 2026-05-13T14:01:05Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/mathoid: apply 2413389 wikitext text/x-wiki 732aoycnadgmjo6jd2dyms92jbftpuj 2413390 2413389 2026-05-13T14:01:13Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: dragoniez, matmarex, lucaswerkmeister-wmde: Backport for [[gerrit:1286890|ApiQueryGlobalUsers: Fix parsing logic for legacy log_params entries (T426033)]], [[gerrit:1286897|ApiQueryGlobalUsers: Fix parsing logic for legacy log_params entries (T426033)]], [[gerrit:1286891|Add 'Promise-Non-Write-API-Action' to $wgAllowedCorsHeaders (T425972)]], [[gerrit:1286892|Add 'Promise-Non-Write-AP 2413390 wikitext text/x-wiki d09wwfvnc9alyl4p1h3vsfapw46gudr 2413391 2413390 2026-05-13T14:01:38Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/mathoid: apply 2413391 wikitext text/x-wiki ace7mmfxm5p0br1njq8vl65c2yossrp 2413392 2413391 2026-05-13T14:01:43Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/mathoid: apply 2413392 wikitext text/x-wiki tfxit5qa0am445tm3yf2zu9xk63t32m 2413393 2413392 2026-05-13T14:02:18Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mathoid: apply 2413393 wikitext text/x-wiki s0muv624bjoqbxcjax461j4sl2kyp3w 2413394 2413393 2026-05-13T14:03:40Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install3004.wikimedia.org 2413394 wikitext text/x-wiki av8uau2087t7l9m5365lfqa09dcbyui 2413395 2413394 2026-05-13T14:03:41Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp7001.* 2413395 wikitext text/x-wiki l056d3adpu0zauahnmzljqhrb3xm3zm 2413396 2413395 2026-05-13T14:03:46Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: dragoniez, matmarex, lucaswerkmeister-wmde: Continuing with deployment 2413396 wikitext text/x-wiki j1tflkr2qoqf43c7wivzssm4pgbszad 2413397 2413396 2026-05-13T14:06:14Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply 2413397 wikitext text/x-wiki h1fbnih3u81aay6ocq0qtnt4f4bhz3u 2413398 2413397 2026-05-13T14:06:49Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply 2413398 wikitext text/x-wiki 792w7vkr4ntak8bpbk1184wpjgm03f9 2413399 2413398 2026-05-13T14:07:56Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1286890|ApiQueryGlobalUsers: Fix parsing logic for legacy log_params entries (T426033)]], [[gerrit:1286897|ApiQueryGlobalUsers: Fix parsing logic for legacy log_params entries (T426033)]], [[gerrit:1286891|Add 'Promise-Non-Write-API-Action' to $wgAllowedCorsHeaders (T425972)]], [[gerrit:1286892|Add 'Promise-Non-Write-API-Action' to $wgAl 2413399 wikitext text/x-wiki r9aqdlo8y9lienlm2zjdtublidw32nt 2413400 2413399 2026-05-13T14:08:29Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2413400 wikitext text/x-wiki 9twdagf5vhti91iox62dyjt0dsnqr8k 2413401 2413400 2026-05-13T14:08:31Z Stashbot 7414 Lucas_WMDE: UTC afternoon backport+config window done 2413401 wikitext text/x-wiki ti91xvhw38djjcu9xl0qr7tuqkm2zn3 2413402 2413401 2026-05-13T14:08:57Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-canary: Restart for upgrade to JVM 11.0.31 - eevans@cumin1003 2413402 wikitext text/x-wiki ebz4econm2wp5ttcbyzkusok4vobwc9 2413403 2413402 2026-05-13T14:09:23Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2413403 wikitext text/x-wiki fvdzvzqhoyviwu25rmfnii7a9py3soq 2413404 2413403 2026-05-13T14:10:00Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2413404 wikitext text/x-wiki 3ryu8d8o8i5v7gj7k5b9tr0qenz3ieg 2413405 2413404 2026-05-13T14:10:45Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2413405 wikitext text/x-wiki tuf21yjuyp1zemke45ph1bu2130fybm 2413406 2413405 2026-05-13T14:10:51Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2413406 wikitext text/x-wiki hvocvqrbpe39qhdf266r5r776mriubf 2413407 2413406 2026-05-13T14:11:33Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2413407 wikitext text/x-wiki sbg95ifa8lg59xx6e3djukzfsd4v4ld 2413408 2413407 2026-05-13T14:12:54Z Stashbot 7414 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1286924|Disable wgWikiLambdaEnableAbstractClientMode everywhere (T422647)]] 2413408 wikitext text/x-wiki on0t3vd1je770fqfqo1ezh6xj9pc1m4 2413409 2413408 2026-05-13T14:14:33Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2413409 wikitext text/x-wiki dgwhjixh7rhfm2qml63hwgrvuiqnrhb 2413410 2413409 2026-05-13T14:14:55Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2413410 wikitext text/x-wiki h8085rgek0vcbhzi8y36tsrg0wlzxdo 2413411 2413410 2026-05-13T14:14:57Z Stashbot 7414 jforrester@deploy1003: jforrester: Backport for [[gerrit:1286924|Disable wgWikiLambdaEnableAbstractClientMode everywhere (T422647)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2413411 wikitext text/x-wiki kswrfj0sz7lwnmqbso1w0zzicld1yim 2413412 2413411 2026-05-13T14:15:03Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2413412 wikitext text/x-wiki 3d8mroq98i6kzgeloessvucyv7g83o4 2413413 2413412 2026-05-13T14:15:19Z Stashbot 7414 jforrester@deploy1003: jforrester: Continuing with deployment 2413413 wikitext text/x-wiki eseggffkq3gs82i2y23xz5lu50sgpkt 2413414 2413413 2026-05-13T14:15:55Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2413414 wikitext text/x-wiki 7mb879zftiidfahm0b9ixgz6qcuyei9 2413415 2413414 2026-05-13T14:15:58Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-codfw: Enable Java security updates - klausman@cumin1003 2413415 wikitext text/x-wiki p1kcfbde2q7csniuqg91tkvbxbqjbr8 2413416 2413415 2026-05-13T14:16:55Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2413416 wikitext text/x-wiki ducxc4t2mwov8tivgszppj12jdhgigc 2413417 2413416 2026-05-13T14:17:33Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on pki-root1002.eqiad.wmnet with reason: host reimage 2413417 wikitext text/x-wiki hmji8zlen4k2fms92x8wyqumf0rov9q 2413418 2413417 2026-05-13T14:17:36Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2413418 wikitext text/x-wiki tv6vr5rpzh3dnwxflj2tsxfchdonmuf 2413419 2413418 2026-05-13T14:19:29Z Stashbot 7414 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1286924|Disable wgWikiLambdaEnableAbstractClientMode everywhere (T422647)]] (duration: 06m 35s) 2413419 wikitext text/x-wiki s7ixqj3hcu88o19fbm3fe3fl56iown6 2413420 2413419 2026-05-13T14:25:06Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pki-root1002.eqiad.wmnet with reason: host reimage 2413420 wikitext text/x-wiki s1k13x4zfb9shh4f82exgovvv4vx9d9 2413421 2413420 2026-05-13T14:28:02Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply 2413421 wikitext text/x-wiki nnyjhknv60gsqrupanru9j13pp51tq4 2413422 2413421 2026-05-13T14:28:17Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply 2413422 wikitext text/x-wiki i9facuqdlxan3k68wbhkxc5ggn6jctc 2413423 2413422 2026-05-13T14:28:44Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.dns.netbox 2413423 wikitext text/x-wiki q90b7t11xg58mobhdkanw7eb0lir7gp 2413424 2413423 2026-05-13T14:33:41Z Stashbot 7414 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add missing DNS name for uslfo network new swtiches - pt1979@cumin2002" 2413424 wikitext text/x-wiki 894l6gd5q47f33ifxv0m6e4721u29a2 2413425 2413424 2026-05-13T14:33:44Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-codfw: Enable Java security updates - klausman@cumin1003 2413425 wikitext text/x-wiki mjyfv2vut9rh4jxjs5hdew4epcagtit 2413426 2413425 2026-05-13T14:33:46Z Stashbot 7414 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add missing DNS name for uslfo network new swtiches - pt1979@cumin2002" 2413426 wikitext text/x-wiki b2xjb6nr50un99tvotk005xb4nds8fw 2413427 2413426 2026-05-13T14:33:47Z Stashbot 7414 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2413427 wikitext text/x-wiki gpoe65817both1hwrb603vaoe9n8zyu 2413428 2413427 2026-05-13T14:33:59Z Stashbot 7414 klausman@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-eqiad: Enable Java security updates - klausman@cumin1003 2413428 wikitext text/x-wiki 6yrynss2wk9r6532oy9pkhjsusv8ljx 2413429 2413428 2026-05-13T14:34:47Z Stashbot 7414 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1286917|WikiEditor: Populate user_groups in EditAttemptStep events (T424010)]] 2413429 wikitext text/x-wiki jnyq8c21kgclds4ckhjwf2d4o6c7emu 2413430 2413429 2026-05-13T14:36:46Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply 2413430 wikitext text/x-wiki bu5aloowwdqjqxwmwa56f60imtz8kuh 2413431 2413430 2026-05-13T14:36:49Z Stashbot 7414 kharlan@deploy1003: kharlan: Backport for [[gerrit:1286917|WikiEditor: Populate user_groups in EditAttemptStep events (T424010)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2413431 wikitext text/x-wiki e5i8bnwa0p7m9eh1sr24xwc6g7n70f5 2413432 2413431 2026-05-13T14:37:51Z Stashbot 7414 kharlan@deploy1003: kharlan: Continuing with deployment 2413432 wikitext text/x-wiki fo16cb4pkwqwkn23cu43msfmwnki9mg 2413433 2413432 2026-05-13T14:42:04Z Stashbot 7414 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1286917|WikiEditor: Populate user_groups in EditAttemptStep events (T424010)]] (duration: 07m 17s) 2413433 wikitext text/x-wiki 865jjwe5jceqwveel5o1762vu8qhmf8 2413434 2413433 2026-05-13T14:43:35Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pki-root1002.eqiad.wmnet with OS trixie 2413434 wikitext text/x-wiki i18beodpjdp6gx0d2c2n4md5wqke9fk 2413435 2413434 2026-05-13T14:47:04Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply 2413435 wikitext text/x-wiki mfkyfzzwc68ubfprr3bql5ggbckn7v8 2413436 2413435 2026-05-13T14:49:44Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply 2413436 wikitext text/x-wiki pvuri56ywp2l8ktiyq2t9qzilephc5i 2413437 2413436 2026-05-13T14:49:46Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply 2413437 wikitext text/x-wiki id2vig2malgpgogfoz86tjcxivt9khu 2413438 2413437 2026-05-13T14:49:50Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply 2413438 wikitext text/x-wiki b21hd9oxw9x4uuctxh1138rji3gxd5n 2413439 2413438 2026-05-13T14:50:13Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2413439 wikitext text/x-wiki 5ti4snztbupz94r666y8v6tscolq9ie 2413440 2413439 2026-05-13T14:51:43Z Stashbot 7414 klausman@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-eqiad: Enable Java security updates - klausman@cumin1003 2413440 wikitext text/x-wiki jv3rdkovr9mrn7l4ce43d6s26gt82vb 2413441 2413440 2026-05-13T14:53:28Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update dns records for missing ulsfo subnets - cmooney@cumin1003" 2413441 wikitext text/x-wiki fjranccf615uvrvd0tobop4j5hxjpqt 2413442 2413441 2026-05-13T14:53:59Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update dns records for missing ulsfo subnets - cmooney@cumin1003" 2413442 wikitext text/x-wiki rk7af4nxi03vrj8ctzm65mzm4aoc7w0 2413443 2413442 2026-05-13T14:54:00Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2413443 wikitext text/x-wiki o2l8864qwu0aigtrsiurjb8snbx6dyc 2413444 2413443 2026-05-13T14:57:20Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2413444 wikitext text/x-wiki 3vcmmtn3qjg7zihwldpa2u8rcjfhdjr 2413445 2413444 2026-05-13T15:00:04Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply 2413445 wikitext text/x-wiki tfx9c0uk6o60srnkel7r0vg7c1qanil 2413446 2413445 2026-05-13T15:00:44Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply 2413446 wikitext text/x-wiki 45tmorj2oxdcs248djvj6fcwidmhkeo 2413447 2413446 2026-05-13T15:01:15Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply 2413447 wikitext text/x-wiki gpz9lxizymtmjtism3zx2m7u125f071 2413448 2413447 2026-05-13T15:04:06Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update dns records for missing ulsfo subnets - cmooney@cumin1003" 2413448 wikitext text/x-wiki 8s1k699p9zys7c6fc87qh4yj34ti53r 2413449 2413448 2026-05-13T15:04:12Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update dns records for missing ulsfo subnets - cmooney@cumin1003" 2413449 wikitext text/x-wiki 1v2h7cvdad1xedb23cqcr0icubutd1m 2413450 2413449 2026-05-13T15:04:13Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2413450 wikitext text/x-wiki shb338d7obnujqul92z7m5j8ddur2hu 2413451 2413450 2026-05-13T15:04:48Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply 2413451 wikitext text/x-wiki 22vx6dtzopovf6t15lwjhdys8x72ttw 2413452 2413451 2026-05-13T15:06:14Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply 2413452 wikitext text/x-wiki fmums5ht9qskvtnmbqiopo2ste5ak9m 2413453 2413452 2026-05-13T15:11:05Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-canary: Restart for upgrade to JVM 11.0.31 - eevans@cumin1003 2413453 wikitext text/x-wiki rqffazv3wu59iqgreqnb93neaasjrex 2413454 2413453 2026-05-13T15:15:40Z Stashbot 7414 cmooney@dns2005: START - running authdns-update 2413454 wikitext text/x-wiki 1hn6zg0cluczf7gm3m3dpzn9b5zhdpi 2413455 2413454 2026-05-13T15:16:57Z Stashbot 7414 cmooney@dns2005: END - running authdns-update 2413455 wikitext text/x-wiki c3dzse0degbb2hocr4nrowcxgjswm04 2413456 2413455 2026-05-13T15:18:50Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-canary: Restart for upgrade to JVM 11.0.31 - eevans@cumin1003 2413456 wikitext text/x-wiki q55oe6uws796wzu6ujyhg6q6pxdic2z 2413458 2413456 2026-05-13T15:27:45Z Stashbot 7414 fabfur: depooling cp7009 to install haproxy-awslc (T419825) 2413458 wikitext text/x-wiki 0h5grbt3ew7cbpsj938u0srzve5xbhj 2413459 2413458 2026-05-13T15:27:50Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=no; selector: name=cp7009.* 2413459 wikitext text/x-wiki a7312xc9xsin23e97ono7bce4mb3fbm 2413460 2413459 2026-05-13T15:31:01Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2413460 wikitext text/x-wiki 6gjmordw5wez0gnjp3mw9u8knctv7ll 2413461 2413460 2026-05-13T15:31:08Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs-test2001.codfw.wmnet with OS bookworm 2413461 wikitext text/x-wiki cgc0z7nmg0l23o8dkqcq3i9ixm38rqc 2413463 2413461 2026-05-13T15:32:39Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2413463 wikitext text/x-wiki rvr77afg3z99uunq0rnttom1s5tgtcw 2413464 2413463 2026-05-13T15:32:54Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs-test2001.codfw.wmnet with OS bookworm 2413464 wikitext text/x-wiki p1y0mrtiuqd0wra018zpzlsjno2p88k 2413465 2413464 2026-05-13T15:36:28Z Stashbot 7414 fabfur: repooling cp7009 to test haproxy-awslc behavior (T419825) 2413465 wikitext text/x-wiki q3f4uo79xr401t30dw3rp7da7zynjnk 2413466 2413465 2026-05-13T15:37:33Z Stashbot 7414 fabfur@cumin1003: conftool action : set/pooled=yes; selector: name=cp7009.* 2413466 wikitext text/x-wiki ojul953v1l58fnbmvsxpb63nx7eu7u9 2413467 2413466 2026-05-13T15:37:34Z Stashbot 7414 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2413467 wikitext text/x-wiki kv9ryqq4cnorwq7cz1nnifx46qbqkt5 2413468 2413467 2026-05-13T15:37:54Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2413468 wikitext text/x-wiki br7qezy95rtb9u4ezsunbp9lsmcje0n 2413470 2413468 2026-05-13T15:40:20Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply 2413470 wikitext text/x-wiki ffh6pg5fxp3agmafiplv0upkg6zkyjj 2413471 2413470 2026-05-13T15:40:55Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply 2413471 wikitext text/x-wiki a7s79t08zlxq4x42fmnezg597h9se8b 2413472 2413471 2026-05-13T15:42:08Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-codfw: Restart for upgrade to JVM 11.0.31 - eevans@cumin1003 2413472 wikitext text/x-wiki 0ywxi5dvf0g5latnkwxlyo2aro3djsv 2413473 2413472 2026-05-13T15:42:52Z Stashbot 7414 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2413473 wikitext text/x-wiki nzzy720x9wdjsogli6efcs1toh7n9wc 2413474 2413473 2026-05-13T15:44:01Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2413474 wikitext text/x-wiki 16ioc3dg8dlnxowka7ixvnss60n3xy1 2413475 2413474 2026-05-13T15:44:45Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply 2413475 wikitext text/x-wiki 2cc9eib5r942ctv872rp2vz7lfqu4h3 2413476 2413475 2026-05-13T15:44:58Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply 2413476 wikitext text/x-wiki 8mamihdzxnlfw6xee1v7w26kg6bgs2z 2413477 2413476 2026-05-13T15:45:05Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply 2413477 wikitext text/x-wiki guwc17x861t9wszcxm2g03xa3etwzcj 2413478 2413477 2026-05-13T15:45:28Z Stashbot 7414 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply 2413478 wikitext text/x-wiki qtcft39iyrz2qtw2z4ouok76xzt9i6u 2413480 2413478 2026-05-13T15:53:09Z Stashbot 7414 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-wdqs-test2001.codfw.wmnet with OS bookworm 2413480 wikitext text/x-wiki 2iboqne5sm2xvyoia65vmqpplekzzpt 2413481 2413480 2026-05-13T15:53:26Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs-test2001.codfw.wmnet with OS bookworm 2413481 wikitext text/x-wiki 5wr4fzggx810god27szbpq24wxqubpo 2413485 2413481 2026-05-13T16:10:24Z Stashbot 7414 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2413485 wikitext text/x-wiki q9y3xqd93lndg3ln4fog8rcntokkg5q 2413486 2413485 2026-05-13T16:10:48Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2413486 wikitext text/x-wiki rg9kb4czpomur578edr80o9c8l2vck2 2413487 2413486 2026-05-13T16:10:55Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2413487 wikitext text/x-wiki 2s2tsahaq53x2g9svnqhcso98as2l6k 2413494 2413487 2026-05-13T16:20:02Z Stashbot 7414 topranks: update OSPF config on eqsin core routers to shift traffic to switch links T424611 2413494 wikitext text/x-wiki mywxbkdh93v03liguwtojzd1tf9zmwy 2413496 2413494 2026-05-13T16:29:02Z Stashbot 7414 topranks: update OSPF config on drmrs core routers to shift traffic to switch links T424611 2413496 wikitext text/x-wiki imxnu857rvy2uz1zmn067skoiaouqlv 2413497 2413496 2026-05-13T16:43:48Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs-test2001.codfw.wmnet with OS bookworm 2413497 wikitext text/x-wiki bw0qzuyywtuyf4a5pyxmgrcp1908rur 2413501 2413497 2026-05-13T16:55:00Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reboot-single for host ncmonitor1001.eqiad.wmnet 2413501 wikitext text/x-wiki 0k0zf78eegs7gigcv7ugm13pl64k011 2413502 2413501 2026-05-13T16:58:56Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ncmonitor1001.eqiad.wmnet 2413502 wikitext text/x-wiki ndk4qmqinrfcgyoitobvjefx6ucpb1t 2413503 2413502 2026-05-13T17:03:43Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.vrts.upgrade on VRTS host vrts1003.eqiad.wmnet 2413503 wikitext text/x-wiki n7fopwdqmhbsb4sgtz84i1k0n1i9qxx 2413505 2413503 2026-05-13T17:05:39Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.vrts.upgrade (exit_code=0) on VRTS host vrts1003.eqiad.wmnet 2413505 wikitext text/x-wiki m841doefacwfbylvz2pz08mbjtevc2b 2413507 2413505 2026-05-13T17:19:54Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-logging-external: apply 2413507 wikitext text/x-wiki 06xitkzbdejmxwfaws226pqaf6u9nhr 2413508 2413507 2026-05-13T17:20:25Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-logging-external: apply 2413508 wikitext text/x-wiki 4ge2oy9c7n6tkrqdoj6oulve3bq08ku 2413509 2413508 2026-05-13T17:24:00Z Stashbot 7414 topranks: update OSPF config on esams core routers to shift traffic to switch links T424611 2413509 wikitext text/x-wiki ganplckhqoajo415q23jjcq2x2k54zh 2413510 2413509 2026-05-13T17:26:14Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-logging-external: apply 2413510 wikitext text/x-wiki 1yhwycsxis1wxsw7kaldmgwbdbnt8ya 2413511 2413510 2026-05-13T17:26:25Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-logging-external: apply 2413511 wikitext text/x-wiki l39nf1d8aiwuwvb6tz5jg2s0okyi1jt 2413512 2413511 2026-05-13T17:26:49Z Stashbot 7414 mutante: zuul1001 - stopping zuul-web; then manually running: /usr/sbin/usermod -u 923 zuul 2413512 wikitext text/x-wiki 42s0s2n66xpkwlzkw3kawki0j1cacye 2413513 2413512 2026-05-13T17:28:01Z Stashbot 7414 mutante: zuul1001 systemctl start zuul-scheduler ; /usr/bin/docker exec zuul-scheduler zuul-scheduler smart-reconfigure 2413513 wikitext text/x-wiki t7xoe2b429ix80k2mz5sdwluthjqjxq 2413514 2413513 2026-05-13T17:33:54Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-logging-external: apply 2413514 wikitext text/x-wiki lo4va2oyz50phjqjm6z3yelk4ktknxg 2413515 2413514 2026-05-13T17:34:12Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-logging-external: apply 2413515 wikitext text/x-wiki ey91vg113cdpktrj3nu4sxryy2dx9v6 2413516 2413515 2026-05-13T17:36:49Z Stashbot 7414 topranks: update OSPF config on magru core routers to shift traffic to switch links T424611 2413516 wikitext text/x-wiki 9zzoerkf5741g1pxvq0tqmtxcn289bu 2413517 2413516 2026-05-13T17:42:59Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-analytics-external: apply 2413517 wikitext text/x-wiki r97z4i1qvaubc5lj3vw6zqc7jkt48ex 2413518 2413517 2026-05-13T17:43:13Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics-external: apply 2413518 wikitext text/x-wiki gr90ayans8x4oamhwx1ow11a2rss300 2413519 2413518 2026-05-13T17:47:15Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-codfw: Restart for upgrade to JVM 11.0.31 - eevans@cumin1003 2413519 wikitext text/x-wiki j2cn9ijnwf9o9khl8xv901abn7b5l38 2413520 2413519 2026-05-13T17:47:30Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply 2413520 wikitext text/x-wiki b2872rt4w4nvmbkz50tckd5z057livq 2413521 2413520 2026-05-13T17:47:56Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply 2413521 wikitext text/x-wiki qlnlvejtg3jfre4lrun7fhjxtcn37j8 2413522 2413521 2026-05-13T17:50:19Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-main: apply 2413522 wikitext text/x-wiki gxrodz7k2a3v6ugoii8mip5nk5c1qt5 2413523 2413522 2026-05-13T17:50:30Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-main: apply 2413523 wikitext text/x-wiki mq1o4v02z0v02e2eakzl9a3jz673p1i 2413525 2413523 2026-05-13T18:00:33Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply 2413525 wikitext text/x-wiki lb5sbi86w6irxewr1hlc2uepp237fz3 2413526 2413525 2026-05-13T18:01:16Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply 2413526 wikitext text/x-wiki 9lioso0ly9bi4hchid9lym28y0oykbb 2413529 2413526 2026-05-13T18:05:46Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-eqiad: Restart for upgrade to JVM 11.0.31 - eevans@cumin1003 2413529 wikitext text/x-wiki lua7crg2sk1gdit77ywgfjqg2fw3n6w 2413530 2413529 2026-05-13T18:09:05Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2413530 wikitext text/x-wiki kbulofqziq0eooe17o40xs01jpyrpbz 2413531 2413530 2026-05-13T18:13:36Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new entries for ulsfo and eqsin IPs - cmooney@cumin1003" 2413531 wikitext text/x-wiki 8b6n7dd3dv98nit5x6f3nrkfv4j7ku7 2413532 2413531 2026-05-13T18:13:43Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new entries for ulsfo and eqsin IPs - cmooney@cumin1003" 2413532 wikitext text/x-wiki rsrl2dbbp6z237rr9wqqam3exi9utog 2413533 2413532 2026-05-13T18:13:44Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2413533 wikitext text/x-wiki gtikvdciti2mu4vl9e6w50dp2mdb2yl 2413534 2413533 2026-05-13T18:13:55Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply 2413534 wikitext text/x-wiki 79talktvy5yu157paw7vqvzho8aooyu 2413535 2413534 2026-05-13T18:14:34Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply 2413535 wikitext text/x-wiki qxiqdyzm72zsi2q8v5b1xwcb7bm0q74 2413536 2413535 2026-05-13T18:19:44Z Stashbot 7414 cmooney@dns2005: START - running authdns-update 2413536 wikitext text/x-wiki 5k7wo4r6j8xc8ym2mkx25hu27zbcbbq 2413537 2413536 2026-05-13T18:20:58Z Stashbot 7414 cmooney@dns2005: END - running authdns-update 2413537 wikitext text/x-wiki ick2r5vdicem0opkzechjvizooa98ey 2413539 2413537 2026-05-13T18:25:35Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply 2413539 wikitext text/x-wiki mofltqax2fpalhshsd2ns44a7ogsivj 2413540 2413539 2026-05-13T18:25:39Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply 2413540 wikitext text/x-wiki ncu5jt66nv4wauqw3fkkvm8109fh8mp 2413541 2413540 2026-05-13T18:26:05Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply 2413541 wikitext text/x-wiki 6ysiz4sabmpthvru4r700f0yf36fsyg 2413542 2413541 2026-05-13T18:27:16Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply 2413542 wikitext text/x-wiki 9cb8i5w62qbclhhr22ti9l0nsrav24v 2413543 2413542 2026-05-13T18:37:43Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: apply 2413543 wikitext text/x-wiki rwxvwm8dti1jh1gwfnujwavi4vvt6ho 2413544 2413543 2026-05-13T18:38:33Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: apply 2413544 wikitext text/x-wiki pwbwd5fpobkyig1y5nufc66rroakguj 2413546 2413544 2026-05-13T19:09:13Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: apply 2413546 wikitext text/x-wiki ndrlmfskw5fh6d13gvzamjv0mm316dy 2413547 2413546 2026-05-13T19:09:30Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: apply 2413547 wikitext text/x-wiki 8nukxjxl7wqxfn119u0quevx05495a7 2413548 2413547 2026-05-13T19:23:02Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: apply 2413548 wikitext text/x-wiki 2u6hpi837zj14m4gdoebedyf0rrdb5u 2413549 2413548 2026-05-13T19:23:33Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: apply 2413549 wikitext text/x-wiki b4vgxdh02pp8ejhmohmfm7pxq3vkxma 2413551 2413549 2026-05-13T20:07:01Z Stashbot 7414 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1286981|Revert "Add wikibase.v1 module to the sandbox were it is present" (T422403)]] 2413551 wikitext text/x-wiki swoipoawr5vkquuywy62dweek47ng2m 2413552 2413551 2026-05-13T20:09:04Z Stashbot 7414 cjming@deploy1003: bpirkle, cjming: Backport for [[gerrit:1286981|Revert "Add wikibase.v1 module to the sandbox were it is present" (T422403)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2413552 wikitext text/x-wiki e01dc7vdsg3bvrokp7tfxgf03iotlz0 2413553 2413552 2026-05-13T20:09:38Z Stashbot 7414 cjming@deploy1003: bpirkle, cjming: Continuing with deployment 2413553 wikitext text/x-wiki 3hwqj2j1hc2sv255ap8pq1h6739pb2p 2413554 2413553 2026-05-13T20:13:03Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-eqiad: Restart for upgrade to JVM 11.0.31 - eevans@cumin1003 2413554 wikitext text/x-wiki p91mxr8713i66ctbxhgva4mn2owlfkg 2413555 2413554 2026-05-13T20:13:48Z Stashbot 7414 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1286981|Revert "Add wikibase.v1 module to the sandbox were it is present" (T422403)]] (duration: 06m 47s) 2413555 wikitext text/x-wiki 3cdgoi2md5wucakrgw1y0mdcz5zw2gz 2413556 2413555 2026-05-13T20:16:31Z Stashbot 7414 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1286997|Revert "cirrus: AB test query suggester variants" (T407432)]] 2413556 wikitext text/x-wiki hkqukj6vdfnk55msj053rmo9ubspsw1 2413557 2413556 2026-05-13T20:17:20Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2413557 wikitext text/x-wiki e9bevn91ncu3k5pxqdzijewxz99iufh 2413558 2413557 2026-05-13T20:17:44Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2413558 wikitext text/x-wiki p8u4en9kdtgl0hy3e9ctc5oxnlc96q2 2413559 2413558 2026-05-13T20:18:29Z Stashbot 7414 ebernhardson@deploy1003: ebernhardson: Backport for [[gerrit:1286997|Revert "cirrus: AB test query suggester variants" (T407432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2413559 wikitext text/x-wiki 5uh4ci4zwaa353rvq0elaeob49ecuc1 2413560 2413559 2026-05-13T20:19:31Z Stashbot 7414 ebernhardson@deploy1003: ebernhardson: Continuing with deployment 2413560 wikitext text/x-wiki 43erwj3c3cismz1w5j7o3v000a50x2k 2413561 2413560 2026-05-13T20:21:36Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2413561 wikitext text/x-wiki cn6y753crnx66sqvxi5x7firlqab9nj 2413562 2413561 2026-05-13T20:21:42Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2413562 wikitext text/x-wiki 1otbj4aqw5ua81llkut07pjkcm9pi7b 2413563 2413562 2026-05-13T20:23:37Z Stashbot 7414 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1286997|Revert "cirrus: AB test query suggester variants" (T407432)]] (duration: 07m 06s) 2413563 wikitext text/x-wiki iikntv4d5b4o9dr1p7gixorh72zlin6 2413564 2413563 2026-05-13T20:25:00Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2413564 wikitext text/x-wiki pwn1izopljg7bjori5rctqxlny6ghnd 2413565 2413564 2026-05-13T20:25:05Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2413565 wikitext text/x-wiki ddfv086s8pfs4zbmjcav84bbbvjevjl 2413566 2413565 2026-05-13T20:25:40Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1287006|Update small size for Swedish Wikipedia (T424910)]] 2413566 wikitext text/x-wiki kb9f4qwnk3hb4l1dtqghfjdgr3as0sa 2413567 2413566 2026-05-13T20:27:41Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1287006|Update small size for Swedish Wikipedia (T424910)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2413567 wikitext text/x-wiki 3kxmfvocvr7pvvhe9hxbs6kiqbinpxa 2413568 2413567 2026-05-13T20:29:00Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Continuing with deployment 2413568 wikitext text/x-wiki lcgozbb7ygdaf30oclqrk4i0zge6i8w 2413569 2413568 2026-05-13T20:33:06Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1287006|Update small size for Swedish Wikipedia (T424910)]] (duration: 07m 26s) 2413569 wikitext text/x-wiki 26dtfz0ofgngq6c02ramnmxjx5faxg4 2413570 2413569 2026-05-13T20:35:28Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1287000|Handle share-highlight images w/o resizeUrl (T426215)]] 2413570 wikitext text/x-wiki boq6949nl8yoid15ycvx1ard7a5qkor 2413571 2413570 2026-05-13T20:37:25Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1287000|Handle share-highlight images w/o resizeUrl (T426215)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2413571 wikitext text/x-wiki 4qe2mnra91jy7jih950oh6ra5arfcc2 2413572 2413571 2026-05-13T20:38:53Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Continuing with deployment 2413572 wikitext text/x-wiki mr5pwil4i2xjf1avuniywto5n5mv8h6 2413573 2413572 2026-05-13T20:41:06Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2413573 wikitext text/x-wiki 8e6u7gc1b03ckskw95e6h0ujfmry3hd 2413574 2413573 2026-05-13T20:41:37Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2413574 wikitext text/x-wiki j3n9riwbtrajdw6ea9qwl3g3trxy0jy 2413575 2413574 2026-05-13T20:42:08Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2413575 wikitext text/x-wiki az3so1uylybsrl0c5xojfihtnve11yc 2413576 2413575 2026-05-13T20:43:01Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1287000|Handle share-highlight images w/o resizeUrl (T426215)]] (duration: 07m 32s) 2413576 wikitext text/x-wiki i61kdv1nov6gmhron01530rc7mlby2l 2413577 2413576 2026-05-13T20:43:15Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2413577 wikitext text/x-wiki ki7r3qgygzj3yetkn1xv87ateoeqe9z 2413578 2413577 2026-05-13T20:43:31Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2413578 wikitext text/x-wiki s33dw9w6x6p1l9nwa6bzyxmgjyasj2s 2413579 2413578 2026-05-13T20:43:34Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2413579 wikitext text/x-wiki 4desy3ae54rhhubnyspp4j7x0fykpkc 2413580 2413579 2026-05-13T20:43:40Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2413580 wikitext text/x-wiki hpzu1bqwbhzhin9qw1rosp9tvspumjj 2413581 2413580 2026-05-13T20:43:48Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply 2413581 wikitext text/x-wiki 3bpzm4gevdxua73ty6pg8ovhlcboxls 2413583 2413581 2026-05-13T20:47:56Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1287022|wgThumbLimits: Remove the exception for itwikiquote (T376152)]] 2413583 wikitext text/x-wiki ola19udhcsrtwvetgdvb75xrd4umnck 2413585 2413583 2026-05-13T20:49:59Z Stashbot 7414 jdlrobson@deploy1003: ladsgroup, jdlrobson: Backport for [[gerrit:1287022|wgThumbLimits: Remove the exception for itwikiquote (T376152)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2413585 wikitext text/x-wiki klw8aiww6pjjcqd4jl1itkaazvl6hu0 2413587 2413585 2026-05-13T20:51:34Z Stashbot 7414 jdlrobson@deploy1003: ladsgroup, jdlrobson: Continuing with deployment 2413587 wikitext text/x-wiki qs87m9z64a14pq1glo4uxbxl0svw85p 2413588 2413587 2026-05-13T20:55:45Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1287022|wgThumbLimits: Remove the exception for itwikiquote (T376152)]] (duration: 07m 48s) 2413588 wikitext text/x-wiki abqw5sqajcz4gx1uetwiyd9z9oje26x 2413589 2413588 2026-05-13T21:06:53Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs: Restart for upgrade to JVM 11.0.31 - eevans@cumin1003 2413589 wikitext text/x-wiki 1s3rusbuoqq9v18es1q1tjf8i3038p3 2413590 2413589 2026-05-13T21:12:07Z Stashbot 7414 Amir1: remapping thumbsize of 0 to 2 in all group0 wikis (T376152) 2413590 wikitext text/x-wiki nvg0eb2qk3ggny6j6e3kgyvzpbizjgl 2413596 2413590 2026-05-14T00:07:25Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs: Restart for upgrade to JVM 11.0.31 - eevans@cumin1003 2413596 wikitext text/x-wiki fwdf4ailqgetbjew19h39u719yuveof 2413598 2413596 2026-05-14T02:00:38Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2413598 wikitext text/x-wiki e9nyruo8dim63vgyybk31fp0wih1lnf 2413599 2413598 2026-05-14T02:07:27Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 49s) 2413599 wikitext text/x-wiki 0abwi670xf2f53pgbajmwsawdm5qwzr 2413600 2413599 2026-05-14T05:04:29Z Stashbot 7414 marostegui@cumin1003: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 5:00:00 on 13 hosts with reason: Sanitarium s2 master: reimage to Debian Trixie 2413600 wikitext text/x-wiki k8ny5cwxgf0l0a3peblmle6l6x6lzf0 2413601 2413600 2026-05-14T05:04:55Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 13 hosts with reason: Sanitarium s7 master: reimage to Debian Trixie 2413601 wikitext text/x-wiki 4qf07gl0b2u4u7s6imq2g3mv4qk5n95 2413602 2413601 2026-05-14T05:05:46Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1158.eqiad.wmnet with reason: Reimage to Trixie 2413602 wikitext text/x-wiki lnmwgcnsmuhcw8191cp4e6a1v9a0q3z 2413603 2413602 2026-05-14T05:05:51Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1158: Reimage to Trixie 2413603 wikitext text/x-wiki 2ea5qmz9koy3pxaf4lmj52vyy4g8gqm 2413604 2413603 2026-05-14T05:06:19Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1158: Reimage to Trixie 2413604 wikitext text/x-wiki 1ao5mulfq40nbo89k36tyxnrrika3o2 2413605 2413604 2026-05-14T05:12:27Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1158.eqiad.wmnet with OS trixie 2413605 wikitext text/x-wiki mida3xq4vfu1caksekp7i6sh41n0uov 2413606 2413605 2026-05-14T05:25:43Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1158.eqiad.wmnet with reason: host reimage 2413606 wikitext text/x-wiki d349lft1izpu3kr547ipk35gqc1mql6 2413607 2413606 2026-05-14T05:29:30Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1158.eqiad.wmnet with reason: host reimage 2413607 wikitext text/x-wiki d04f5uwf7njnwwnbqo38c1x5lcbg72d 2413608 2413607 2026-05-14T05:51:21Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1158.eqiad.wmnet with OS trixie 2413608 wikitext text/x-wiki 6j0l85suq18bkfxaqae0vnpjfjjz0wt 2413609 2413608 2026-05-14T05:54:01Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1158: after reimage to trixie 2413609 wikitext text/x-wiki gdkl5knb7u1n6wg3g88gn3kgn552q2q 2413610 2413609 2026-05-14T06:39:26Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1158: after reimage to trixie 2413610 wikitext text/x-wiki 5cdjl6d6rh3jjr69n2xno0h3uthqbuf 2413611 2413610 2026-05-14T06:39:35Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc2013: Replacing HW T418973 2413611 wikitext text/x-wiki 7kvpt4r5hgdg32a0c6q5lraduzc84rl 2413612 2413611 2026-05-14T06:39:37Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2413612 wikitext text/x-wiki mvdznq9bsqhze3eyszj07rzl0h7zj5u 2413613 2413612 2026-05-14T06:39:42Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2413613 wikitext text/x-wiki h857qzvtfh4pedoz4ua03yad8ypo2u3 2413614 2413613 2026-05-14T06:39:44Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc2013: Replacing HW T418973 2413614 wikitext text/x-wiki kq0bp1g29iiniojdz0wj6v7wbd7x0pv 2413615 2413614 2026-05-14T06:40:00Z Stashbot 7414 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply 2413615 wikitext text/x-wiki di21qvde9vjtzyxgs01shi5phqhr00l 2413616 2413615 2026-05-14T06:40:36Z Stashbot 7414 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply 2413616 wikitext text/x-wiki jx6vh1rfva7czgphhggqgfcni6dxjag 2413617 2413616 2026-05-14T06:41:03Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc[2013,2023].codfw.wmnet,pc1013.eqiad.wmnet with reason: Maintenance on pc3 2413617 wikitext text/x-wiki 84goitt9knp4gf7ephe5e859ps3smxe 2413620 2413617 2026-05-14T07:00:05Z Stashbot 7414 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply 2413620 wikitext text/x-wiki 0zh52bjfyseww7o2nwsnv7bp9xz3u81 2413621 2413620 2026-05-14T07:00:40Z Stashbot 7414 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply 2413621 wikitext text/x-wiki hu18g0q2s8fzg0brr044io6zm5hz89u 2413622 2413621 2026-05-14T07:01:36Z Stashbot 7414 kart_: Update cxserver to 2026-04-23-114216-production (T423002) 2413622 wikitext text/x-wiki 1ke5lixcswa714fp8gb756rhw2dj3gc 2413623 2413622 2026-05-14T08:08:52Z Stashbot 7414 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.2 refs T423911 2413623 wikitext text/x-wiki 8ue1byucvn2pvbqew28fin0ca451z7e 2413624 2413623 2026-05-14T08:39:19Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Remove db2149 T424341', diff saved to https://phabricator.wikimedia.org/P92520 and previous config saved to /var/cache/conftool/dbconfig/20260514-083916-marostegui.json 2413624 wikitext text/x-wiki q2gnsr753fdaba4818j886gtbmo1tmj 2413625 2413624 2026-05-14T08:54:17Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hosts.reimage for host mc1065.eqiad.wmnet with OS bullseye 2413625 wikitext text/x-wiki 46roux1rffx9oxuklsuck968dumwlr8 2413626 2413625 2026-05-14T08:54:19Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host mc1066.eqiad.wmnet with OS bullseye 2413626 wikitext text/x-wiki smumk6nk8krdbnu7elp207uk49kvsis 2413627 2413626 2026-05-14T08:54:21Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host mc1067.eqiad.wmnet with OS bullseye 2413627 wikitext text/x-wiki gwulzzp0kqx37c5qmprmc4n5qninzs3 2413628 2413627 2026-05-14T08:54:28Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host mc1068.eqiad.wmnet with OS bullseye 2413628 wikitext text/x-wiki ckmwayps0ig49t9war7z6u1okpcz4q5 2413629 2413628 2026-05-14T08:55:14Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/ratelimit: apply 2413629 wikitext text/x-wiki rvpwtw0zdwt9ic8j45yg1swm628qnc9 2413630 2413629 2026-05-14T08:55:39Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply 2413630 wikitext text/x-wiki obgs0vgy9rlsiupie900h34quz2fwt8 2413632 2413630 2026-05-14T09:06:33Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage 2413632 wikitext text/x-wiki bj7cviw6htd7nvz7ivhus5wywyurc3x 2413633 2413632 2026-05-14T09:06:38Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage 2413633 wikitext text/x-wiki 5ocz51c42qs63z7ltkre0fek49t9zmz 2413634 2413633 2026-05-14T09:06:50Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage 2413634 wikitext text/x-wiki cdb90nqilzh2ambraxvann6ln9jqbae 2413635 2413634 2026-05-14T09:06:52Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage 2413635 wikitext text/x-wiki q8d08b54c5pimn51j7whi8vwoys8f2w 2413636 2413635 2026-05-14T09:10:38Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage 2413636 wikitext text/x-wiki m2y4fs98202vyv2oh4z35xulj9a0aoa 2413637 2413636 2026-05-14T09:14:29Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage 2413637 wikitext text/x-wiki 1nbyjdo9yhzfypg874p3bvvvne8mu7b 2413639 2413637 2026-05-14T09:18:40Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage 2413639 wikitext text/x-wiki k005hxibbedj6vszkmy5mm1bj4nze44 2413640 2413639 2026-05-14T09:20:57Z Stashbot 7414 Emperor: rebalance codfw swift rings T354872 2413640 wikitext text/x-wiki fxcjbd9xdks7jnkqf3of5v1t7wum880 2413641 2413640 2026-05-14T09:23:21Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage 2413641 wikitext text/x-wiki mzrag858j2cxsg47k6ntpuo6if3a85p 2413642 2413641 2026-05-14T09:26:00Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1066.eqiad.wmnet with OS bullseye 2413642 wikitext text/x-wiki an9duyf0kasap25p5xcmbxqighua2b2 2413643 2413642 2026-05-14T09:30:28Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1068.eqiad.wmnet with OS bullseye 2413643 wikitext text/x-wiki dvoq7wu3jjjbkki1y4xwgnug6ruhrm6 2413644 2413643 2026-05-14T09:33:55Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1065.eqiad.wmnet with OS bullseye 2413644 wikitext text/x-wiki tfr9k357tofpewkshgmkul97vmezjxf 2413645 2413644 2026-05-14T09:39:23Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1067.eqiad.wmnet with OS bullseye 2413645 wikitext text/x-wiki 1ouzme08pvzweacxbxay5rrz0o9n5al 2413646 2413645 2026-05-14T09:49:51Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2152.codfw.wmnet: Host will be decommissioned 2413646 wikitext text/x-wiki 3jzrf9zut06n6ygixyt2rupwccj55mt 2413647 2413646 2026-05-14T09:51:37Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: apply 2413647 wikitext text/x-wiki rmb02rsx7kenlyf8y1g7mz7f9dk5uzd 2413648 2413647 2026-05-14T09:52:00Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply 2413648 wikitext text/x-wiki payn65a5coj0lfe8l9fbujii79q1y8h 2413649 2413648 2026-05-14T09:54:53Z Stashbot 7414 cwilliams@cumin1003: END (ERROR) - Cookbook sre.mysql.depool (exit_code=97) depool db2152.codfw.wmnet: Host will be decommissioned 2413649 wikitext text/x-wiki paf9y5iqata8cgb5284x7b3vu57z4v4 2413652 2413649 2026-05-14T10:02:08Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2152: Host will be decommissioned 2413652 wikitext text/x-wiki 9iw0pi6dmxj3xbpouzdevtk9k5prln5 2413653 2413652 2026-05-14T10:02:28Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2152: Host will be decommissioned 2413653 wikitext text/x-wiki 4nnj9z1btwkp6fyo8tt1dixy8ylf040 2413655 2413653 2026-05-14T10:14:03Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2413655 wikitext text/x-wiki f87boajumugjen8sfgh9bxbzgijs1qp 2413656 2413655 2026-05-14T10:14:10Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2413656 wikitext text/x-wiki jucbdyl1pmci0xax28lsuhy4kasigrl 2413657 2413656 2026-05-14T10:14:36Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host mc1069.eqiad.wmnet with OS bullseye 2413657 wikitext text/x-wiki d37g4dl86b8ndrzlnvf57t17m8tdqog 2413658 2413657 2026-05-14T10:15:22Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.reimage for host mc1063.eqiad.wmnet with OS bullseye 2413658 wikitext text/x-wiki gksu21zkm83duf71q8d6qcdj204b46f 2413659 2413658 2026-05-14T10:17:59Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2413659 wikitext text/x-wiki 73jph2ebjxr0amlf2poqkn6q3kktpyt 2413660 2413659 2026-05-14T10:19:19Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply 2413660 wikitext text/x-wiki 61t0bmlass450yh4kw7ye1sf294d0dl 2413661 2413660 2026-05-14T10:19:24Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply 2413661 wikitext text/x-wiki qa7y0jeyt3wq0gc8bfz9lmfrpzlj1l6 2413662 2413661 2026-05-14T10:25:27Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply 2413662 wikitext text/x-wiki p4cas1n33qt3qr9qh59t97m6zswym4s 2413663 2413662 2026-05-14T10:25:33Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply 2413663 wikitext text/x-wiki 8v5fftqitadluyl356jx6d2e92lawgt 2413664 2413663 2026-05-14T10:27:09Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1069.eqiad.wmnet with reason: host reimage 2413664 wikitext text/x-wiki qcr5a09umayh0hatw8y19e2spcw4w9r 2413665 2413664 2026-05-14T10:27:27Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1063.eqiad.wmnet with reason: host reimage 2413665 wikitext text/x-wiki 8st754yce0rz13s5tc73gg3upnpo59m 2413666 2413665 2026-05-14T10:34:14Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1069.eqiad.wmnet with reason: host reimage 2413666 wikitext text/x-wiki qacfk5yyef7glfwvybd9ts46uixzmrl 2413667 2413666 2026-05-14T10:34:18Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply 2413667 wikitext text/x-wiki knylb66mnxd7lowg9f28ifhpuqk9lal 2413668 2413667 2026-05-14T10:34:37Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply 2413668 wikitext text/x-wiki asio03hlnkzk28o44aooaioliocp1xd 2413669 2413668 2026-05-14T10:38:14Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1063.eqiad.wmnet with reason: host reimage 2413669 wikitext text/x-wiki 89nb0zahv1hiix73lk7wfbpckjqn3dv 2413670 2413669 2026-05-14T10:40:54Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. 2413670 wikitext text/x-wiki buj2j78pw6680vm24x7qyyobun17gw3 2413671 2413670 2026-05-14T10:41:34Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. 2413671 wikitext text/x-wiki 23ea7efy6shhtywgn77s9ruswm9zpha 2413672 2413671 2026-05-14T10:45:24Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Remove db2152 from dbctl T424344', diff saved to https://phabricator.wikimedia.org/P92523 and previous config saved to /var/cache/conftool/dbconfig/20260514-104521-marostegui.json 2413672 wikitext text/x-wiki tj96mqp92dhb3avyimv2jxcm8s8k6re 2413673 2413672 2026-05-14T10:49:30Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1069.eqiad.wmnet with OS bullseye 2413673 wikitext text/x-wiki 7xxsdi5orfcw2ofovy236mfpa3n0mzp 2413675 2413673 2026-05-14T10:53:26Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1063.eqiad.wmnet with OS bullseye 2413675 wikitext text/x-wiki f0jv6a0hqh5q67o0qeuuj2ulyxctqrm 2413676 2413675 2026-05-14T10:53:56Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply 2413676 wikitext text/x-wiki kbz2i16lf0kc7hvnrxchucvu2sfti21 2413677 2413676 2026-05-14T10:53:59Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply 2413677 wikitext text/x-wiki opo8231vfkdsco3q2e63lwplvufvzdf 2413679 2413677 2026-05-14T11:00:12Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: apply 2413679 wikitext text/x-wiki 247wt0mnv9vkv6pbjktv7vnb3j32jge 2413680 2413679 2026-05-14T11:00:27Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply 2413680 wikitext text/x-wiki t4reoyiq4jrv2ianl968p1yuxlj8kpn 2413681 2413680 2026-05-14T11:00:53Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: sync 2413681 wikitext text/x-wiki 21lzbvrnklqt2eccywgfzkt6flbvo1z 2413682 2413681 2026-05-14T11:01:05Z Stashbot 7414 jiji@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: sync 2413682 wikitext text/x-wiki 6qagap8rl83439ej1lph3scel7qqplz 2413683 2413682 2026-05-14T11:02:04Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs-test1001.eqiad.wmnet with OS bookworm 2413683 wikitext text/x-wiki h97ra4b8lcbt9oxa93szr4nkzupf9r9 2413685 2413683 2026-05-14T11:08:57Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply 2413685 wikitext text/x-wiki jrk1fwwphpa0nrbin4nlw9z16cp98l3 2413686 2413685 2026-05-14T11:09:00Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply 2413686 wikitext text/x-wiki im9zx64orhqb020adxj8kp468zo3cz6 2413687 2413686 2026-05-14T11:31:13Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply 2413687 wikitext text/x-wiki 9cpx563r24iz25qq68jqs2yfgvhgyb9 2413688 2413687 2026-05-14T11:31:23Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply 2413688 wikitext text/x-wiki kjvjl9ba1s9hlksbl1y754f86c5ugxz 498 8712 2413618 2199200 2026-05-14T06:50:43Z Stashbot 7414 komla@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase by 176 gigabytes (T425905) 2413618 wikitext text/x-wiki cij8ndxnkpblzym0zmmn3g9k5pi7ks1 2413619 2413618 2026-05-14T06:50:49Z Stashbot 7414 komla@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 176 gigabytes (T425905) 2413619 wikitext text/x-wiki ivmoe2ztos66gmvbx16sou4r8n3n79x 0 17290 2413344 2412987 2026-05-13T12:37:38Z Stashbot 7414 James_F: Zuul: [mediawiki/extensions/WikiLambda] Drop AF and SB deps down to phan-only, for T423180 2413344 wikitext text/x-wiki 6ab9727wk154sh6z2z7pt0e3e322nbh 2413349 2413344 2026-05-13T12:42:15Z Stashbot 7414 James_F: Zuul: [mediawiki/extensions/ChatIntegration] Add dependency on VisualEditor 2413349 wikitext text/x-wiki bd4ibx1gq9gs3nrtu0wq7lmue936jy8 2413350 2413349 2026-05-13T12:42:30Z Stashbot 7414 James_F: Zuul: [mediawiki/extensions/ChatBot] Add dependencies on VisualEditor and BlueSpiceFoundation 2413350 wikitext text/x-wiki pjknr7x8cqr1ihhgenz2ap3sc56zprd 2413351 2413350 2026-05-13T12:42:44Z Stashbot 7414 James_F: Zuul: [mediawiki/extensions/Springboard] Add AdminLinks Phan dependency 2413351 wikitext text/x-wiki 4dkg9ei2h4v09u0uh10cz1j69awmvx5 1 26157 2413489 2404121 2026-05-13T16:14:25Z BCornwall-WMF 31521 /* Question about the note beneath 'Setting up your SSH config' code */ Reply 2413489 wikitext text/x-wiki aotaz6e1sl0njzwvoellw9o5m7xophu 0 33280 2413528 2413316 2026-05-13T18:05:20Z Novem Linguae 23682 add requestctl to ops and wmf. ce. 2413528 wikitext text/x-wiki gra9jc0tvly0vqvx7cvmsi67i6bz7no 0 122303 2413457 2405227 2026-05-13T15:26:53Z FNegri-WMF 32595 /* Regenerate .kube/config */ fix the configmap name 2413457 wikitext text/x-wiki tbnwxdzdhgs0jskrml89tffyffre7l2 2413462 2413457 2026-05-13T15:31:21Z FNegri-WMF 32595 /* Regenerate .kube/config */ clarify what project means in this context 2413462 wikitext text/x-wiki 3fal2r7oeb25it0nsw47db6ew4c3xxo 498 443949 2413367 2412203 2026-05-13T13:19:17Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 0b33d6e (Refresh landing page) 2413367 wikitext text/x-wiki cwg10hva0jho2uftfqgjvxn37hwxc3t 2413368 2413367 2026-05-13T13:19:18Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy e2636db (Update documentation link in README following reorganization) 2413368 wikitext text/x-wiki aqmg9s40bmvvocy48o06igyjy64r4s9 2413369 2413368 2026-05-13T13:19:19Z Stashbot 7414 wmbot~jeanfred@tools-bastion-15: Deploy 5e938ef (Update link to Toolforge in base HTML template) 2413369 wikitext text/x-wiki lub3l9gt9oxhaagotkidbutm1x8cdac 12 444256 2413594 2380460 2026-05-13T22:06:52Z HYang-WMF 53712 Clarify <shell-username> 2413594 wikitext text/x-wiki ki3fbthe4hxm9cvhosuxrth7ujgybna 0 446795 2413654 2131576 2026-05-14T10:07:08Z CWilliams-WMF 55316 Updating command to depool the host 2413654 wikitext text/x-wiki iv7q6lmxltao1vye8cqpx5vofqowyui 2413678 2413654 2026-05-14T10:58:34Z CWilliams-WMF 55316 Fixed Puppet FQDN 2413678 wikitext text/x-wiki d8aymsiiays8uh2cg4af3ga60nehfr6 0 449160 2413595 2413256 2026-05-14T00:00:44Z Dexbot 30554 Bot: Updating the report 2413595 wikitext text/x-wiki smbnkjecg99kg7bwppopcct5p46hlqu 0 453101 2413592 2399840 2026-05-13T21:57:13Z DWisehaupt (WMF) 14911 add stub for analytics data center swap 2413592 wikitext text/x-wiki t6jjdxeovbbt8vvlpe49b4krlkys4ac 0 453382 2413365 2351883 2026-05-13T13:16:41Z CMooney (WMF) 24288 /* Processors */ 2413365 wikitext text/x-wiki 1sa54rfb2zuhi20byzmfbb0z0yx971l 0 457961 2413524 2400649 2026-05-13T17:59:36Z Novem Linguae 23682 /* Access control */ I think these docs can be viewed a bit wider than SRE 2413524 wikitext text/x-wiki hqpc3nqral1920tbbwks4bpmbtyouhn 0 459290 2413506 2413020 2026-05-13T17:09:08Z XCollazo-WMF 31846 /* NEXT TRAIN */ 2413506 wikitext text/x-wiki 0wxd87v5sofc9yfii2tzgku6y26sk7s 2413527 2413506 2026-05-13T18:02:46Z XCollazo-WMF 31846 /* NEXT TRAIN */ 2413527 wikitext text/x-wiki 2gupooakdk1qytra9laxipg1pylw2r8 2 460039 2413631 2412419 2026-05-14T08:57:09Z Atsuko-WMF 54082 2413631 wikitext text/x-wiki k88p32jf0ci57lwfbq11zidsdq5g72d 4 460155 2413591 2412104 2026-05-13T21:55:13Z BLiviero-WMF 53603 2413591 wikitext text/x-wiki td5bumbc58eukqh4ab34vh6942dyjoj 0 460169 2413582 2026-05-13T20:47:08Z GModena (WMF) 20194 Created page with "{{irdoc|status=draft}} == Summary == {{Incident scorecard | task = T425758 | paged-num = 0 | responders-num = Brian King, Ryan Kemper, Guillaume Lederrey, Gabriele Modena, Ben Tullis | coordinators = Gabriele Modena | start = 2026-05-07 15:10:00 | end = 2026-05-11 13:50:00 | metrics = Both Uptime (availability) percentage as well as Excessive lag percentage | impact = We serve stale data for >20 hours from 6 nodes, and at peak 50% of WDQS external endpoint requests we..." 2413582 wikitext text/x-wiki s856oyqfn8gmjun80r91xxosk4z9uz4 2413584 2413582 2026-05-13T20:48:18Z GModena (WMF) 20194 2413584 wikitext text/x-wiki dflowvekmpwv18byr3bgi2tyt470jkx 2413586 2413584 2026-05-13T20:50:10Z GModena (WMF) 20194 2413586 wikitext text/x-wiki c3v6d8j3qh9o4cdyn30ijrox067flbd 0 460170 2413593 2026-05-13T21:58:34Z DWisehaupt (WMF) 14911 shell of data center swap from working etherpad doc. 2413593 wikitext text/x-wiki jiic37ugnbndwwfc7r7ij5ga2ukgvqq