Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.46.0-wmf.23 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2399797 2399606 2026-04-08T15:45:38Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder 2399797 wikitext text/x-wiki 99dzudufmfxywcoiw6g72abw5mp4ix7 2399799 2399797 2026-04-08T15:50:55Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder 2399799 wikitext text/x-wiki 03bk3onayesba28y04g7uwvtqivomrs 2399839 2399799 2026-04-08T17:31:45Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2399839 wikitext text/x-wiki nhisrawtckavqvwb5mh9eu87u9tyk5q 2399847 2399839 2026-04-08T17:41:19Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api 2399847 wikitext text/x-wiki 8i1eufhxfq67knucw3i0gsmc1y9h11b 2399859 2399847 2026-04-08T17:50:28Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2399859 wikitext text/x-wiki qlynib9rqwvxzgq39lcii0klmiy87oq 2399861 2399859 2026-04-08T17:55:21Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2399861 wikitext text/x-wiki dy8ismgzl5wbsi2aavxulklscgclayn 2399951 2399861 2026-04-09T06:06:53Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx 2399951 wikitext text/x-wiki qyaoqvq0xdj7wj1g48b45u985c3dplf 2399952 2399951 2026-04-09T06:12:08Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx 2399952 wikitext text/x-wiki lnr84tme7appkpgkgircxyrarskld8q 2399955 2399952 2026-04-09T06:19:00Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx 2399955 wikitext text/x-wiki 706jyhh6659a6r3egmgqjbpbrns0pyp 2399956 2399955 2026-04-09T06:24:15Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx 2399956 wikitext text/x-wiki iz4u3vu918b6ewypw2mv9hs9jt05h1c 2399957 2399956 2026-04-09T06:28:19Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx 2399957 wikitext text/x-wiki olz55jte6xu3erk3iu4fyut8mlvy8qb 2399958 2399957 2026-04-09T06:29:57Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx 2399958 wikitext text/x-wiki lp0lxp98d0pgj1kod5p46843mqhgjcq 0 3205 2399835 2396984 2026-04-08T17:25:23Z BCornwall-WMF 31521 /* Planned reboot of LVS servers */ Note inevitable alerts 2399835 wikitext text/x-wiki 6c553yfl7fgpluvutpaap7xyswftmwe 0 3219 2399838 2382884 2026-04-08T17:30:13Z BCornwall-WMF 31521 Remove outdated mention of icinga1001 being the installation 2399838 wikitext text/x-wiki tqqr7v9qfr8pq56mfymv1rd9nzcuq58 0 4108 2399701 2399651 2026-04-08T13:25:53Z ScheduleDeploymentBot 37566 Add [[gerrit:1268948]] to Wednesday, April 08 UTC afternoon backport window 2399701 wikitext text/x-wiki m3056f2p2wrppg8sb4i6xvlbhzas54x 2399702 2399701 2026-04-08T13:26:00Z ScheduleDeploymentBot 37566 Add [[gerrit:1268947]] to Wednesday, April 08 UTC afternoon backport window 2399702 wikitext text/x-wiki nxzvohizzb4hz7w8j39udtrp1unszvd 2399915 2399702 2026-04-08T21:59:03Z ScheduleDeploymentBot 37566 Add [[gerrit:1269063]] to Thursday, April 09 UTC afternoon backport window 2399915 wikitext text/x-wiki jlrmac6gamv9jpx38tmrgrllyy8gj48 2400056 2399915 2026-04-09T11:58:29Z ScheduleDeploymentBot 37566 Add [[gerrit:1269351]] to Thursday, April 09 UTC afternoon backport window 2400056 wikitext text/x-wiki 46wl25y8tufju1nyfuykup7905m96fw 2400057 2400056 2026-04-09T11:58:52Z ScheduleDeploymentBot 37566 Add [[gerrit:1269339]] to Thursday, April 09 UTC afternoon backport window 2400057 wikitext text/x-wiki lzkncikuprg990u2teemosyygn1av1y 0 4627 2399803 2395467 2026-04-08T15:53:07Z Taavi 13997 expand hatnote 2399803 wikitext text/x-wiki 4ra6scw0s0sycsi7cudv23gonkf5pih 498 6726 2399774 2399596 2026-04-08T15:20:00Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder 2399774 wikitext text/x-wiki 4mcl160x2pz60t8c112ry1a3s19p9r8 2399777 2399774 2026-04-08T15:24:38Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder 2399777 wikitext text/x-wiki k16xh3tx4hx144duhpqabt3k7zjs2od 2399833 2399777 2026-04-08T17:20:29Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2399833 wikitext text/x-wiki csq9gktom5ils5wyzzvjo5dm8dpg7kp 2399836 2399833 2026-04-08T17:29:13Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api 2399836 wikitext text/x-wiki a6qhivhk43omu73fdkr4c0f3pejufr8 2399846 2399836 2026-04-08T17:41:09Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2399846 wikitext text/x-wiki 9o40udytzmwo4uomn5z6hnln2wdongq 2399857 2399846 2026-04-08T17:45:47Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2399857 wikitext text/x-wiki tw51c59ec6vr7a1c2qu6mwre8g1gi1j 2399947 2399857 2026-04-09T06:03:32Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx 2399947 wikitext text/x-wiki kasbhs0c2qg4ce13wmuh0mru5pyw0kt 2399948 2399947 2026-04-09T06:03:35Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx 2399948 wikitext text/x-wiki o9433yvz5ivb1g7pmu1cide23hyrwuk 2399949 2399948 2026-04-09T06:04:29Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx 2399949 wikitext text/x-wiki 8hpiz32y1tsz9m211hzu7jbast96ff7 2399950 2399949 2026-04-09T06:06:45Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx 2399950 wikitext text/x-wiki 72vu0zn1k2xhstbir96ga5jo0oexemp 2399953 2399950 2026-04-09T06:16:29Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx 2399953 wikitext text/x-wiki ciddaskzlispo99ccrjbwio0q9egkha 2399954 2399953 2026-04-09T06:18:56Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx 2399954 wikitext text/x-wiki bn770ael6x213e3vybx9jjbbtxgl138 0 7195 2399815 2393378 2026-04-08T16:29:21Z Elisapoly 54397 Add missing info 2399815 wikitext text/x-wiki 71117fu6dt4lzsns1pk2g690jp5aphr 0 7478 2399811 2372151 2026-04-08T16:13:10Z AndrewBogott (WMF) 51047 2399811 wikitext text/x-wiki 94u4kndoegk8nem9cdk2nk0j5s3cb82 0 7919 2399674 2399669 2026-04-08T12:07:29Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM wikikube-worker-exp1001.eqiad.wmnet 2399674 wikitext text/x-wiki 3wx1a8dp19xuex6svajgoacm7pk93xo 2399675 2399674 2026-04-08T12:13:34Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM wikikube-worker-exp1001.eqiad.wmnet 2399675 wikitext text/x-wiki 4t8srkttyulxm29fyh2n229oj0lphji 2399676 2399675 2026-04-08T12:15:18Z Stashbot 7414 mszwarc@deploy1003: mwscript-k8s job started: foreachwikiindblist all backfillInterwikiRightsLog.php --remote-wiki metawiki 20260311190000 # T6055 2399676 wikitext text/x-wiki nb534ceumhm6at44leq4swjdts5spxx 2399677 2399676 2026-04-08T12:27:00Z Stashbot 7414 jiji@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM wikikube-worker-exp2001.codfw.wmnet 2399677 wikitext text/x-wiki qthhe2sf3z4iu3uu37tv03sfikdctls 2399678 2399677 2026-04-08T12:27:33Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2399678 wikitext text/x-wiki qyqvf7jyd8eei97vpfqm9lybfxixnaw 2399679 2399678 2026-04-08T12:27:52Z Stashbot 7414 jiji@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM wikikube-worker-exp2001.codfw.wmnet 2399679 wikitext text/x-wiki kwnivbgio8rpslhrf50ijh6iprzbgjv 2399680 2399679 2026-04-08T12:28:05Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2399680 wikitext text/x-wiki 5n4n6tgid1wtrhqoor0id5g25edg29a 2399681 2399680 2026-04-08T12:28:37Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2399681 wikitext text/x-wiki 0frloexl28oqgcusmyk18qf7b5ds8lc 2399682 2399681 2026-04-08T12:29:18Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2399682 wikitext text/x-wiki e0u2haeyvlamtavtce610s8hiftjqyd 2399683 2399682 2026-04-08T12:31:59Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2399683 wikitext text/x-wiki l6iqv9ures3geyyociq48j4rq8w1chu 2399684 2399683 2026-04-08T12:32:06Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2399684 wikitext text/x-wiki kulscimp0n9gyh7xhxzvxt0yw6w3ceb 2399685 2399684 2026-04-08T12:32:12Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2399685 wikitext text/x-wiki kt75ybobuzhra4ddvzkjkb4r2g9ugsu 2399686 2399685 2026-04-08T12:32:16Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2399686 wikitext text/x-wiki 7391jh24r7mj0wv3t4oy0dxngu17337 2399687 2399686 2026-04-08T12:32:40Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2399687 wikitext text/x-wiki t1yqx8dt235xbdwpqznlvsgyhyh4eec 2399688 2399687 2026-04-08T12:40:18Z Stashbot 7414 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host testvm2006.codfw.wmnet with OS trixie 2399688 wikitext text/x-wiki pyvdz6jofnrp7cr2zaqjhy5gzadsmzl 2399689 2399688 2026-04-08T12:43:12Z Stashbot 7414 taavi: restarting pybal on lvs1020 2399689 wikitext text/x-wiki n1qz7qsskcqohvjyld4497f7cxhuwhy 2399690 2399689 2026-04-08T12:49:20Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2399690 wikitext text/x-wiki qaf2fcmae98j7q9kuawsq5ozijfi3uw 2399693 2399690 2026-04-08T13:04:52Z Stashbot 7414 taavi: restarting pybal on lvs1018 2399693 wikitext text/x-wiki 1wn5vrasfqiv7nqrn6k4f4chhdat1z6 2399694 2399693 2026-04-08T13:08:38Z Stashbot 7414 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1268679|Turn on Parsoid Read Views for eswiki (T422524)]] 2399694 wikitext text/x-wiki nk8ykxiokvp7r1nsjir3e9ybfgmvezu 2399695 2399694 2026-04-08T13:10:35Z Stashbot 7414 cscott@deploy1003: cscott: Backport for [[gerrit:1268679|Turn on Parsoid Read Views for eswiki (T422524)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2399695 wikitext text/x-wiki snsss8w8g62e6mson9s6te3luqsowgp 2399696 2399695 2026-04-08T13:11:48Z Stashbot 7414 cscott@deploy1003: cscott: Continuing with sync 2399696 wikitext text/x-wiki knv7pn3ya9o7l3t5d5otyz8ysbs11fv 2399697 2399696 2026-04-08T13:15:44Z Stashbot 7414 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1268679|Turn on Parsoid Read Views for eswiki (T422524)]] (duration: 07m 06s) 2399697 wikitext text/x-wiki mdvues6b2jd54e8u0bocqgcnzj2tpsq 2399698 2399697 2026-04-08T13:22:26Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1268820|cswiki: lift IP cap for workshop (T422520)]] 2399698 wikitext text/x-wiki 21j4ykg028m0o8stjmsmojy9xosrtq5 2399699 2399698 2026-04-08T13:24:18Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: anzx, lucaswerkmeister-wmde: Backport for [[gerrit:1268820|cswiki: lift IP cap for workshop (T422520)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2399699 wikitext text/x-wiki k5j3alxzoi3uivvt3nrizps064v97y4 2399700 2399699 2026-04-08T13:25:02Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: anzx, lucaswerkmeister-wmde: Continuing with sync 2399700 wikitext text/x-wiki a3se1jr0l5tkd9j3ki9m35sa71v4tqq 2399703 2399700 2026-04-08T13:26:27Z Stashbot 7414 moritzm: upgrade debdeploy-server on cumin2002 to 0.0.99.14-1+deb12u1+exp1 (temporary build with Cumin 6 compat before we have Cumin 6 universally) 2399703 wikitext text/x-wiki 3p67slmrau963tfw7s5eh4bc33rpbgr 2399704 2399703 2026-04-08T13:28:49Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1268820|cswiki: lift IP cap for workshop (T422520)]] (duration: 06m 22s) 2399704 wikitext text/x-wiki j4l3k2fpd4ttfsl88obb0w4iwg76yjr 2399705 2399704 2026-04-08T13:31:47Z Stashbot 7414 ayounsi@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host testvm2006.codfw.wmnet with OS trixie 2399705 wikitext text/x-wiki 93sh7i2ncgsicxyfzfa4qvu7jtbzzhh 2399706 2399705 2026-04-08T13:33:24Z Stashbot 7414 phuedx@deploy1003: Started scap sync-world: Backport for [[gerrit:1268948|PHP SDK: Measure known experiments correctly (T422112)]], [[gerrit:1268947|PHP SDK: Measure known experiments correctly (T422112)]] 2399706 wikitext text/x-wiki q20fb9z0wq4khxepjvcto6pgtsl6wu2 2399707 2399706 2026-04-08T13:33:43Z Stashbot 7414 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=wdqs-internal-scholarly,name=eqiad 2399707 wikitext text/x-wiki tnejoc4otx9deagwjx6s80dx6nzp9il 2399709 2399707 2026-04-08T13:35:18Z Stashbot 7414 phuedx@deploy1003: phuedx: Backport for [[gerrit:1268948|PHP SDK: Measure known experiments correctly (T422112)]], [[gerrit:1268947|PHP SDK: Measure known experiments correctly (T422112)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2399709 wikitext text/x-wiki bg4dvbjfz2dgg4r4icrm8wxcxhrah3u 2399710 2399709 2026-04-08T13:37:36Z Stashbot 7414 phuedx@deploy1003: phuedx: Continuing with sync 2399710 wikitext text/x-wiki 4sgz18hc8chz14slqkn34hkk2ss8xbd 2399711 2399710 2026-04-08T13:41:23Z Stashbot 7414 phuedx@deploy1003: Finished scap sync-world: Backport for [[gerrit:1268948|PHP SDK: Measure known experiments correctly (T422112)]], [[gerrit:1268947|PHP SDK: Measure known experiments correctly (T422112)]] (duration: 07m 58s) 2399711 wikitext text/x-wiki byh6x6oa4k6l93rg34f21hw8jfyh4zw 2399712 2399711 2026-04-08T13:42:56Z Stashbot 7414 Lucas_WMDE: UTC afternoon backport+config window done 2399712 wikitext text/x-wiki r1neflf9xay3rmvegj9jzbba0i2nrfj 2399713 2399712 2026-04-08T13:45:39Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2399713 wikitext text/x-wiki fv0v0z0eri7o0wsbk77k7k9n5szc8mo 2399716 2399713 2026-04-08T14:03:21Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2399716 wikitext text/x-wiki 3fo1pi6bhp9ndefpvth616nbav70s31 2399717 2399716 2026-04-08T14:04:10Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2399717 wikitext text/x-wiki n0t9hxzt1wdl4zqi176fkr3flpmutdq 2399718 2399717 2026-04-08T14:07:25Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2399718 wikitext text/x-wiki qk20zoclws5k6skfi27wgp0y29yoqav 2399719 2399718 2026-04-08T14:07:38Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2399719 wikitext text/x-wiki hl88ca2jj9ivqorqr0g45v7geekwuvs 2399720 2399719 2026-04-08T14:08:30Z Stashbot 7414 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2399720 wikitext text/x-wiki 0hs2h5oq5q645sstwupfs6i5hopyegk 2399721 2399720 2026-04-08T14:09:05Z Stashbot 7414 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2399721 wikitext text/x-wiki qvkgt7oh3i3kgroqogeni17qfslhwgo 2399722 2399721 2026-04-08T14:10:19Z Stashbot 7414 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2399722 wikitext text/x-wiki l4ujfmgzdqi9ta1ev2iiqs92tgi6zwu 2399723 2399722 2026-04-08T14:11:04Z Stashbot 7414 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2399723 wikitext text/x-wiki r1omam9j12avdmzup2cyzvwbb3un4yt 2399724 2399723 2026-04-08T14:11:14Z Stashbot 7414 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2399724 wikitext text/x-wiki saxpyb28aims4uw2luq8lf5xlttnpfo 2399725 2399724 2026-04-08T14:11:54Z Stashbot 7414 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2399725 wikitext text/x-wiki te7c06vlziudmcmp1opusfzbey1ax0m 2399726 2399725 2026-04-08T14:16:33Z Stashbot 7414 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2399726 wikitext text/x-wiki o8ogp2utwcwb6igzyz0j37hspxrc54t 2399727 2399726 2026-04-08T14:17:14Z Stashbot 7414 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2399727 wikitext text/x-wiki 9leibitb4432mngjep2lvhrygxy1el5 2399728 2399727 2026-04-08T14:17:58Z Stashbot 7414 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2399728 wikitext text/x-wiki orv1wowfog5duooywixn0x8aozkut6c 2399729 2399728 2026-04-08T14:18:34Z Stashbot 7414 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2399729 wikitext text/x-wiki r23x6iee63fs81eeohpqpz0fswm11s1 2399730 2399729 2026-04-08T14:18:46Z Stashbot 7414 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2399730 wikitext text/x-wiki 0ad04dkdzdot0o0l1ccqtbm6iq0w4u4 2399731 2399730 2026-04-08T14:19:17Z Stashbot 7414 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2399731 wikitext text/x-wiki fzlc1ksdbfc53zmfk6k8sydhl3a0cim 2399732 2399731 2026-04-08T14:32:10Z Stashbot 7414 fabfur: upgrading eqsin to haproxy 3.2 (T421402) 2399732 wikitext text/x-wiki g1sw7gfz5fh5rr79quxy5eca4djyobt 2399733 2399732 2026-04-08T14:36:12Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host seaborgium.wikimedia.org 2399733 wikitext text/x-wiki m8f1dlxjjavz2fjnxoazwloykikqk0e 2399734 2399733 2026-04-08T14:37:12Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqsin - 3.2 upgrade (T421402) 2399734 wikitext text/x-wiki i1x6mxiwdlr85dnrclsh1jkvzvroffa 2399735 2399734 2026-04-08T14:37:27Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqsin - 3.2 upgrade (T421402) 2399735 wikitext text/x-wiki 5yjeis1xqfgl59ijt9drq581m2uo6gc 2399736 2399735 2026-04-08T14:39:57Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host seaborgium.wikimedia.org 2399736 wikitext text/x-wiki 8s52bisdgrhootzdhd7xiq8smo838wi 2399737 2399736 2026-04-08T14:41:08Z Stashbot 7414 taavi@dns1004: START - running authdns-update 2399737 wikitext text/x-wiki pfuxp6jsfufdl6nz64wvhf9abvzgwp1 2399738 2399737 2026-04-08T14:42:27Z Stashbot 7414 taavi@dns1004: END - running authdns-update 2399738 wikitext text/x-wiki meshcpicdg6vduur8qbblc3jacqnux1 2399739 2399738 2026-04-08T14:48:00Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2399739 wikitext text/x-wiki mat86z32ww1rfgu2ubeqkae6xjmch83 2399740 2399739 2026-04-08T14:48:57Z Stashbot 7414 taavi: serve dumps rsync traffic via new LVS service T422040 2399740 wikitext text/x-wiki pmuejbxs6yq4fq2s7ka2uaxrq625a5s 2399741 2399740 2026-04-08T14:49:43Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2399741 wikitext text/x-wiki gd3f6wmx2rhrf8bregr57wblnj5phsc 2399742 2399741 2026-04-08T14:53:30Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2399742 wikitext text/x-wiki 7xgk589fk2mm76p4gjwv5y3jfz5fj0n 2399743 2399742 2026-04-08T14:57:25Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2399743 wikitext text/x-wiki od7nekmznynkap4q0863bp6osyg9x7t 2399744 2399743 2026-04-08T14:58:42Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2399744 wikitext text/x-wiki 0wp19z8188xgnk17xa7igd3yf5mbgvn 2399745 2399744 2026-04-08T14:58:44Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2399745 wikitext text/x-wiki chqzxz9rhh46z0nj5lmmr3u1e3kwtmm 2399746 2399745 2026-04-08T15:00:49Z Stashbot 7414 derick@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=zhwiki --logwiki=metawiki 'Mr Kazi Tuhin' KaziHasanTuhin # T422677 2399746 wikitext text/x-wiki o09e3d2nkc47nt3mp47863xb00easck 2399747 2399746 2026-04-08T15:05:23Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch1081.eqiad.wmnet with OS bullseye 2399747 wikitext text/x-wiki bkvhp2p515nw3o0rxkueqky8jxcz2pr 2399748 2399747 2026-04-08T15:05:53Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host cirrussearch1081 2399748 wikitext text/x-wiki au06pz6wkf0a4rt1uin96qs7kpro4yh 2399749 2399748 2026-04-08T15:06:12Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.dns.netbox 2399749 wikitext text/x-wiki j515zth8jzg58wjsi6tyear0q22uzqe 2399750 2399749 2026-04-08T15:10:07Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1081 - bking@cumin2002" 2399750 wikitext text/x-wiki 2xitj9g6xk63bz86yt05c5p7d0otccy 2399751 2399750 2026-04-08T15:10:12Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1081 - bking@cumin2002" 2399751 wikitext text/x-wiki hx87e6d3na5dwzzud4nj2azzvd31cbf 2399752 2399751 2026-04-08T15:10:14Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2399752 wikitext text/x-wiki as6sn7yihkuxwyihyivzhnc1u3zimgb 2399753 2399752 2026-04-08T15:10:14Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.dns.wipe-cache cirrussearch1081.eqiad.wmnet 166.32.64.10.in-addr.arpa 6.6.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2399753 wikitext text/x-wiki 7siuwqfjvl94fy20jglanl3irzrhvgl 2399754 2399753 2026-04-08T15:10:17Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cirrussearch1081.eqiad.wmnet 166.32.64.10.in-addr.arpa 6.6.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2399754 wikitext text/x-wiki bt5dgd6wvfm45ibpofbkbjiarlpjvwx 2399755 2399754 2026-04-08T15:10:18Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1081 2399755 wikitext text/x-wiki js2ixn3wocljc5gwn1ig5pzsj4w1omv 2399756 2399755 2026-04-08T15:11:30Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1081 2399756 wikitext text/x-wiki f8rfa56uqhdopudtxdlqpbnwniqkkbh 2399757 2399756 2026-04-08T15:11:31Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1081 2399757 wikitext text/x-wiki d4nurmrncibyrsmj18brnvd9ods25kg 2399759 2399757 2026-04-08T15:16:24Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2399759 wikitext text/x-wiki te9pyfjyxqp2l725itbzn2micp3wqdu 2399760 2399759 2026-04-08T15:16:30Z Stashbot 7414 andrew@cumin2002: START - Cookbook sre.hosts.decommission for hosts cloudcephmon2004-dev.codfw.wmnet 2399760 wikitext text/x-wiki nichym2nxu276po9vzgkqfvtdme2i1f 2399761 2399760 2026-04-08T15:16:33Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2399761 wikitext text/x-wiki kmffl6bute1gix2ejm480wpfemjxyio 2399762 2399761 2026-04-08T15:16:58Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2399762 wikitext text/x-wiki 26i6xfnh1rzrg4ux91bzz7jlez8h0wc 2399763 2399762 2026-04-08T15:17:04Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2399763 wikitext text/x-wiki 04wsmelswzujffjs80jefmepbxkkvud 2399764 2399763 2026-04-08T15:17:10Z Stashbot 7414 sukhe: sukhe@lvs1020:~$ sudo systemctl restart pybal.service 2399764 wikitext text/x-wiki 9jvwgez7lkdmqi93uz8rql3q4kj1b7d 2399765 2399764 2026-04-08T15:17:23Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2399765 wikitext text/x-wiki fgcrsfopd1f6nsamh0wjcnqy6if9hvx 2399766 2399765 2026-04-08T15:17:29Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2399766 wikitext text/x-wiki 7k5pufhq8cgq2lsri9xuoh0f64un5zg 2399767 2399766 2026-04-08T15:17:59Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2399767 wikitext text/x-wiki 33917m1pwl7th65swjlv6ssvribyfih 2399768 2399767 2026-04-08T15:18:04Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2399768 wikitext text/x-wiki 5wigtzqluora6q2jyikd41cbst73k8n 2399769 2399768 2026-04-08T15:18:41Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2399769 wikitext text/x-wiki bctrp2c8wjh6e5er3ryuxiqnmh972u9 2399770 2399769 2026-04-08T15:18:49Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2399770 wikitext text/x-wiki 24iwh64hmmuda9ilofwn12t6fzgniqm 2399771 2399770 2026-04-08T15:19:03Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2399771 wikitext text/x-wiki sdxm45l23j7n2x3fpwwjs875wnmivbc 2399772 2399771 2026-04-08T15:19:10Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2399772 wikitext text/x-wiki 3ambfj480jor3nlvzfzhvnhlb4073ek 2399773 2399772 2026-04-08T15:19:55Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch1087.eqiad.wmnet with OS bullseye 2399773 wikitext text/x-wiki 9ccno6kv9tgqalq19yyydz0dufox900 2399775 2399773 2026-04-08T15:20:27Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host cirrussearch1087 2399775 wikitext text/x-wiki 004uypxei3iecw3q21a0n1aly9uhb8u 2399776 2399775 2026-04-08T15:20:49Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.dns.netbox 2399776 wikitext text/x-wiki j69dd8uc3jrcc3n880rii5vs69pekz7 2399778 2399776 2026-04-08T15:26:04Z Stashbot 7414 andrew@cumin2002: START - Cookbook sre.dns.netbox 2399778 wikitext text/x-wiki 7b4i8606tnn8hx5yulzr8m949k8350c 2399779 2399778 2026-04-08T15:26:17Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1087 - bking@cumin2002" 2399779 wikitext text/x-wiki l14wq3fuyqswukklmz5u0ni3ms0gkt6 2399780 2399779 2026-04-08T15:27:04Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1081.eqiad.wmnet with reason: host reimage 2399780 wikitext text/x-wiki qdnd3fnvregmcd4rrzb77uy4wtd7v6y 2399781 2399780 2026-04-08T15:27:26Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1087 - bking@cumin2002" 2399781 wikitext text/x-wiki cw4uyedf18pf6mtr7hal5el2ahm6m7b 2399782 2399781 2026-04-08T15:27:27Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2399782 wikitext text/x-wiki 299o58srzejaemne6f06xfeqw9jmuqo 2399783 2399782 2026-04-08T15:27:28Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.dns.wipe-cache cirrussearch1087.eqiad.wmnet 174.32.64.10.in-addr.arpa 4.7.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2399783 wikitext text/x-wiki ksz45pktuhe0l0fofclagd5hy547rtu 2399784 2399783 2026-04-08T15:27:31Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cirrussearch1087.eqiad.wmnet 174.32.64.10.in-addr.arpa 4.7.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2399784 wikitext text/x-wiki fqlqgs4076plu494pd8se39z3ja0pcn 2399785 2399784 2026-04-08T15:27:32Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1087 2399785 wikitext text/x-wiki n0xr3jma72u7gpcdg42vg4ggzq4gfis 2399786 2399785 2026-04-08T15:28:09Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1087 2399786 wikitext text/x-wiki bnx94mys5hw9pf13lf88ltr1npxdjhk 2399787 2399786 2026-04-08T15:28:10Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1087 2399787 wikitext text/x-wiki kh1gpjyofnm7csmd0g2pw1j63bof8uo 2399788 2399787 2026-04-08T15:28:47Z Stashbot 7414 andrew@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2399788 wikitext text/x-wiki bgvybtppdrblbqmhf6lw44yid9ne3if 2399789 2399788 2026-04-08T15:28:49Z Stashbot 7414 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephmon2004-dev.codfw.wmnet 2399789 wikitext text/x-wiki t060s279jclu4pnqfbpup5j30jk19fi 2399790 2399789 2026-04-08T15:29:20Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_eqsin - 3.2 upgrade (T421402) 2399790 wikitext text/x-wiki libxhmgc7ui4qwr4k9kqv7h744hbc8r 2399791 2399790 2026-04-08T15:35:00Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1081.eqiad.wmnet with reason: host reimage 2399791 wikitext text/x-wiki qygxofgd9yccdqcjarndtnjyrafqfp0 2399792 2399791 2026-04-08T15:36:47Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_eqsin - 3.2 upgrade (T421402) 2399792 wikitext text/x-wiki 7ns7m7ck6z1twe35gflnz8anqy32egn 2399793 2399792 2026-04-08T15:41:57Z Stashbot 7414 fabfur: upgrading codfw to haproxy 3.2 (T421402) 2399793 wikitext text/x-wiki bp8efc0c6lbtuuafe4up3oke9a9fdkz 2399794 2399793 2026-04-08T15:42:08Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_codfw - 3.2 upgrade (T421402) 2399794 wikitext text/x-wiki 8o88z3qazu65j8i3qm1v4lww1cnx2qu 2399795 2399794 2026-04-08T15:42:18Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_codfw - 3.2 upgrade (T421402) 2399795 wikitext text/x-wiki 5vx8ocxejzbfek009yp8f5lgaj0b3u6 2399796 2399795 2026-04-08T15:43:54Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1087.eqiad.wmnet with reason: host reimage 2399796 wikitext text/x-wiki s39mfsvly5dc5kdcfwba4knjjh2zdi2 2399798 2399796 2026-04-08T15:49:10Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1087.eqiad.wmnet with reason: host reimage 2399798 wikitext text/x-wiki 2iqy42170kxulvt4dg7japu9zd3kb1g 2399800 2399798 2026-04-08T15:52:14Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host testvm2002.codfw.wmnet with OS trixie 2399800 wikitext text/x-wiki j8xxzvxtk60b3aa5it2buhagbk2ml29 2399801 2399800 2026-04-08T15:52:29Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.remove-downtime for aqs1023.eqiad.wmnet 2399801 wikitext text/x-wiki bwd8vvc53cpvdtmhyfbfr1mhqtj6d29 2399802 2399801 2026-04-08T15:52:30Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for aqs1023.eqiad.wmnet 2399802 wikitext text/x-wiki 5rxhwbag4j1pja83y6qtq7gfxh6fupc 2399805 2399802 2026-04-08T16:00:37Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1081.eqiad.wmnet with OS bullseye 2399805 wikitext text/x-wiki 2ts2due701am2hnvwpwmayha2itfegr 2399808 2399805 2026-04-08T16:07:36Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1087.eqiad.wmnet with OS bullseye 2399808 wikitext text/x-wiki oi5btwkrab1kpms1xiyh94ag0sfw07n 2399812 2399808 2026-04-08T16:19:16Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_codfw - 3.2 upgrade (T421402) 2399812 wikitext text/x-wiki p6ssqd28t69lldkcy1tbk7wsd7gk42l 2399813 2399812 2026-04-08T16:20:29Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_codfw - 3.2 upgrade (T421402) 2399813 wikitext text/x-wiki j1z5ipoeq02bnao33dbqrnl96ahybq9 2399820 2399813 2026-04-08T17:00:11Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch1088.eqiad.wmnet with OS bullseye 2399820 wikitext text/x-wiki kp4u9n97pv516glj5uvq6611vjs4a4p 2399821 2399820 2026-04-08T17:01:12Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host cirrussearch1088 2399821 wikitext text/x-wiki 0z680uxr28glc4bye9ff4wyojlekdr4 2399822 2399821 2026-04-08T17:02:53Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.dns.netbox 2399822 wikitext text/x-wiki rmpdoikbujn7rr3cq0ojap7ufoe89q9 2399824 2399822 2026-04-08T17:06:45Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1088 - bking@cumin2002" 2399824 wikitext text/x-wiki a90fm052h5nl7wua9qds5tatjvjn7j4 2399825 2399824 2026-04-08T17:06:51Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1088 - bking@cumin2002" 2399825 wikitext text/x-wiki jhxhznyc4jiv55t8np49n2kinyy9vwd 2399826 2399825 2026-04-08T17:06:52Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2399826 wikitext text/x-wiki lr11l9kgx4cufjcwzaqtbce0xjmqehb 2399827 2399826 2026-04-08T17:06:53Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.dns.wipe-cache cirrussearch1088.eqiad.wmnet 176.32.64.10.in-addr.arpa 6.7.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2399827 wikitext text/x-wiki easpz9bc75v7umjgpt5k1s83sxbwhwz 2399828 2399827 2026-04-08T17:06:55Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cirrussearch1088.eqiad.wmnet 176.32.64.10.in-addr.arpa 6.7.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2399828 wikitext text/x-wiki t3lpqdyf3ug2bebk3st464myail1nim 2399829 2399828 2026-04-08T17:06:56Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1088 2399829 wikitext text/x-wiki 1hmwe6o69iga3z7vau091flp49trpsv 2399830 2399829 2026-04-08T17:07:31Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1088 2399830 wikitext text/x-wiki bl3b763lv9vbi8wt9e01fg5mgnafght 2399831 2399830 2026-04-08T17:07:32Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1088 2399831 wikitext text/x-wiki mrzyx4jtwkmral2diq6fpxprwl6tf0a 2399832 2399831 2026-04-08T17:08:17Z Stashbot 7414 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host testvm2002.codfw.wmnet with OS trixie 2399832 wikitext text/x-wiki 27c0xz5dz0sro2emitvpr2spcfnhrl0 2399834 2399832 2026-04-08T17:23:39Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1088.eqiad.wmnet with reason: host reimage 2399834 wikitext text/x-wiki 5orf7rom3dcxkookfim3876649ezk8u 2399837 2399834 2026-04-08T17:29:50Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1088.eqiad.wmnet with reason: host reimage 2399837 wikitext text/x-wiki dcj0bw4zyp9xhbwdrr4afpsguj8310q 2399841 2399837 2026-04-08T17:35:57Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch1089.eqiad.wmnet with OS bullseye 2399841 wikitext text/x-wiki m070gkfexb0h703cptd0q89q9geppao 2399842 2399841 2026-04-08T17:36:09Z Stashbot 7414 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cirrussearch1089.eqiad.wmnet with OS bullseye 2399842 wikitext text/x-wiki ko563jq6t7hdxiwbpudu4o7nx1z5yrr 2399843 2399842 2026-04-08T17:36:45Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch1103.eqiad.wmnet with OS bullseye 2399843 wikitext text/x-wiki 2y3k6qxukp43boa0cc3ku860ibyv4ij 2399844 2399843 2026-04-08T17:37:16Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host cirrussearch1103 2399844 wikitext text/x-wiki ih8cs4de96tlh6k5gxftnhh18uz1gsp 2399845 2399844 2026-04-08T17:39:02Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.dns.netbox 2399845 wikitext text/x-wiki kij9bpt3udkscker1r6f6lxz3ytrp0h 2399848 2399845 2026-04-08T17:42:41Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1103 - bking@cumin2002" 2399848 wikitext text/x-wiki b0ehuaco7uncvzii4esxckfvhs5c5qw 2399849 2399848 2026-04-08T17:42:46Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch1103 - bking@cumin2002" 2399849 wikitext text/x-wiki fwmne4ovd6hqe2xczmoshs0vyi0anb3 2399850 2399849 2026-04-08T17:42:47Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2399850 wikitext text/x-wiki ft0nukmfld8m53or16s7t0botr8otwn 2399851 2399850 2026-04-08T17:42:48Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.dns.wipe-cache cirrussearch1103.eqiad.wmnet 43.48.64.10.in-addr.arpa 3.4.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2399851 wikitext text/x-wiki 3p4rqhs3whv46vtp5e8xqrhirgbyffx 2399852 2399851 2026-04-08T17:42:50Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cirrussearch1103.eqiad.wmnet 43.48.64.10.in-addr.arpa 3.4.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2399852 wikitext text/x-wiki 5n949hhe97ows1btcdbg0x8tfugn20c 2399853 2399852 2026-04-08T17:42:51Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1103 2399853 wikitext text/x-wiki sn4i7xxiv6qqzpr5j0iu1n13n0s77nr 2399854 2399853 2026-04-08T17:43:42Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1103 2399854 wikitext text/x-wiki t3noq480w8zb4id52kyg9w7fmv3py0w 2399855 2399854 2026-04-08T17:43:43Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1103 2399855 wikitext text/x-wiki 6x3ukgw70duwz61emkew87iq91p8uk6 2399858 2399855 2026-04-08T17:49:30Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1088.eqiad.wmnet with OS bullseye 2399858 wikitext text/x-wiki tf4hi9g10qgpivy9afpq5pxta32b050 2399862 2399858 2026-04-08T18:00:22Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1103.eqiad.wmnet with reason: host reimage 2399862 wikitext text/x-wiki e1b0hcmr8v2n18504r99w00ufztn15l 2399863 2399862 2026-04-08T18:04:23Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1103.eqiad.wmnet with reason: host reimage 2399863 wikitext text/x-wiki jlsy88kcs5krv7mry5xn2q81b4bkhta 2399864 2399863 2026-04-08T18:08:28Z Stashbot 7414 dancy@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.46.0-wmf.23 refs T420481 2399864 wikitext text/x-wiki noog279l9rhqzlrsa4o2dq9l2ycjotr 2399867 2399864 2026-04-08T18:25:40Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1103.eqiad.wmnet with OS bullseye 2399867 wikitext text/x-wiki n7piqbwp1235zgshq9jk1m8vkwipxug 2399872 2399867 2026-04-08T18:32:53Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch1103.eqiad.wmnet with OS bullseye 2399872 wikitext text/x-wiki m26atrglmhxue2lfs10bfjc08u5s244 2399873 2399872 2026-04-08T18:33:15Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.move-vlan for host cirrussearch1103 2399873 wikitext text/x-wiki n8j4917keoz7knyhl6dan0cp3p8drfy 2399874 2399873 2026-04-08T18:33:16Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1103 2399874 wikitext text/x-wiki sy48r4bcg9lrb7f57rdcknbxiipzeqm 2399875 2399874 2026-04-08T18:49:39Z Stashbot 7414 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1103.eqiad.wmnet with reason: host reimage 2399875 wikitext text/x-wiki bsu1gem1gfhfszqcou1rfi5eu57vndw 2399876 2399875 2026-04-08T18:54:36Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1103.eqiad.wmnet with reason: host reimage 2399876 wikitext text/x-wiki nekd1ph0xlxne6pdf5kkvanvpdevnip 2399877 2399876 2026-04-08T18:55:55Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/mw-mcrouter: apply 2399877 wikitext text/x-wiki khdmkwee81nxxeoh0emu2f0ne06w7ez 2399879 2399877 2026-04-08T18:56:50Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-mcrouter: apply 2399879 wikitext text/x-wiki no2ut9oi02knx7kntpxvz6ddtqv7y88 2399880 2399879 2026-04-08T18:57:26Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply 2399880 wikitext text/x-wiki 07lzd2676tobk3eernorq2oulft04jv 2399882 2399880 2026-04-08T19:01:34Z Stashbot 7414 eevans@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on aqs1024.eqiad.wmnet with reason: Bootstrapping — T412830 2399882 wikitext text/x-wiki pixyfl1xbi6e8mzqkf4xf3gv7ch0w1x 2399884 2399882 2026-04-08T19:10:00Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply 2399884 wikitext text/x-wiki 1vl4ozbsalskg1ryfdi8h403jio4k7q 2399886 2399884 2026-04-08T19:14:39Z Stashbot 7414 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1103.eqiad.wmnet with OS bullseye 2399886 wikitext text/x-wiki qhzsd7g2xhamun3fdy2ktsp84xjel54 2399887 2399886 2026-04-08T19:22:38Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply 2399887 wikitext text/x-wiki 05tmi6z660ejjzgkbftn4qjdhw78mqv 2399890 2399887 2026-04-08T19:35:14Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply 2399890 wikitext text/x-wiki msdcdaqwaf4qbmw36bnd4ykwtnm2whc 2399892 2399890 2026-04-08T20:07:34Z Stashbot 7414 toyofuku@deploy1003: Started scap sync-world: Backport for [[gerrit:1269036|Disable extension:WP25EasterEggs from Wikipedias. (T422548)]] 2399892 wikitext text/x-wiki sry12a994hwpni31x0046c6e9rt1zj9 2399893 2399892 2026-04-08T20:09:27Z Stashbot 7414 toyofuku@deploy1003: jdrewniak, toyofuku: Backport for [[gerrit:1269036|Disable extension:WP25EasterEggs from Wikipedias. (T422548)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2399893 wikitext text/x-wiki e54gasyy4grjuqsdb2tr7jnv20d702c 2399894 2399893 2026-04-08T20:13:09Z Stashbot 7414 toyofuku@deploy1003: jdrewniak, toyofuku: Continuing with sync 2399894 wikitext text/x-wiki bcibpnzcqysyiaao9da46a89mf8e9xt 2399895 2399894 2026-04-08T20:17:02Z Stashbot 7414 toyofuku@deploy1003: Finished scap sync-world: Backport for [[gerrit:1269036|Disable extension:WP25EasterEggs from Wikipedias. (T422548)]] (duration: 09m 27s) 2399895 wikitext text/x-wiki ggsr2vco090vbyup55w25zim0yor1jy 2399896 2399895 2026-04-08T20:21:13Z Stashbot 7414 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1269046|Bumping portals to master (T128546)]] 2399896 wikitext text/x-wiki djht8v2f1v4esbj1xi9ffxr0c19nhpv 2399897 2399896 2026-04-08T20:23:11Z Stashbot 7414 jdrewniak@deploy1003: jdrewniak: Backport for [[gerrit:1269046|Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2399897 wikitext text/x-wiki 0d390wwjl0aniehoeyjoaokmcn2n8qy 2399898 2399897 2026-04-08T20:24:23Z Stashbot 7414 jdrewniak@deploy1003: jdrewniak: Continuing with sync 2399898 wikitext text/x-wiki e9xaa27vbhyjv9o1n6uedf8u9m1wut5 2399900 2399898 2026-04-08T20:34:01Z Stashbot 7414 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1269046|Bumping portals to master (T128546)]] 2399900 wikitext text/x-wiki admdpe0iu8toldn02jsyp2y0m9dndfn 2399901 2399900 2026-04-08T20:35:50Z Stashbot 7414 jdrewniak@deploy1003: jdrewniak: Backport for [[gerrit:1269046|Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2399901 wikitext text/x-wiki l1fa1xz5joklpphrk0e5ic4f20e1iz4 2399902 2399901 2026-04-08T20:36:25Z Stashbot 7414 jdrewniak@deploy1003: jdrewniak: Continuing with sync 2399902 wikitext text/x-wiki qyt3rbdfwdrfaemzyds6cgferomev2x 2399903 2399902 2026-04-08T20:40:16Z Stashbot 7414 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1269046|Bumping portals to master (T128546)]] (duration: 06m 14s) 2399903 wikitext text/x-wiki fg4q8bfv5gccb4d5nv3x8selobwehjh 2399904 2399903 2026-04-08T20:58:14Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1269050|Use envoy for swift inside mediawiki (T328872)]] 2399904 wikitext text/x-wiki ns7a8wmg66lth6a9f72cgqgg2mntjyq 2399905 2399904 2026-04-08T21:00:06Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1269050|Use envoy for swift inside mediawiki (T328872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2399905 wikitext text/x-wiki 01yaoghrh9m0t3d5aokru8z9h1lss6p 2399906 2399905 2026-04-08T21:00:46Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with sync 2399906 wikitext text/x-wiki as1fel5bm8blx374m089ra7qi9dz4rk 2399907 2399906 2026-04-08T21:04:41Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1269050|Use envoy for swift inside mediawiki (T328872)]] (duration: 06m 27s) 2399907 wikitext text/x-wiki 27tvcwafomovwa0dmasx1cgwb8dzn3i 2399908 2399907 2026-04-08T21:17:26Z Stashbot 7414 rzl@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2399908 wikitext text/x-wiki 6w5c973kyyl3qyr84u226ew1fq5n3v4 2399909 2399908 2026-04-08T21:27:42Z Stashbot 7414 rzl@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2399909 wikitext text/x-wiki qfggvbywqqwnw95gx4q0tiq7wgt52hy 2399911 2399909 2026-04-08T21:45:28Z Stashbot 7414 rzl@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2399911 wikitext text/x-wiki 97tvzm4kiyh0ra923emoa3fuyu5tabw 2399912 2399911 2026-04-08T21:45:41Z Stashbot 7414 rzl@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2399912 wikitext text/x-wiki 2fo2u2gzq6oewber2ar36rtbhuasrg9 2399913 2399912 2026-04-08T21:46:00Z Stashbot 7414 rzl@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2399913 wikitext text/x-wiki dxy69vmmfp3in3llzq5ptuhcmjiq3wm 2399914 2399913 2026-04-08T21:57:35Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1269067|Revert "Use envoy for swift inside mediawiki"]] 2399914 wikitext text/x-wiki q5moxcb6ifbovt4wote4971yuvutwca 2399916 2399914 2026-04-08T21:59:30Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1269067|Revert "Use envoy for swift inside mediawiki"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2399916 wikitext text/x-wiki 20zxjtoyarv84z21e8b0vargxkw3m9r 2399917 2399916 2026-04-08T22:00:36Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with sync 2399917 wikitext text/x-wiki thq6lf4swo15bf9y3r8z6ahx75l5k24 2399919 2399917 2026-04-08T22:04:29Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1269067|Revert "Use envoy for swift inside mediawiki"]] (duration: 06m 54s) 2399919 wikitext text/x-wiki l8ohw18ah6hi5yyr81if2k9sp8b0jsb 2399925 2399919 2026-04-09T00:22:23Z Stashbot 7414 eevans@cumin1003: START - Cookbook sre.hosts.remove-downtime for aqs1024.eqiad.wmnet 2399925 wikitext text/x-wiki n1v9edyf2kbohj142qwmgr1p65o7req 2399926 2399925 2026-04-09T00:22:25Z Stashbot 7414 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for aqs1024.eqiad.wmnet 2399926 wikitext text/x-wiki dxl9dpcja86ivhqlcrpq5q8b6pm86rd 2399928 2399926 2026-04-09T00:49:47Z Stashbot 7414 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1268574|Start reading from new file tables everwhere except enwiki and commons (T416548)]] 2399928 wikitext text/x-wiki daxhilnbu2d8nipz4objd3gn58sge0m 2399929 2399928 2026-04-09T00:51:39Z Stashbot 7414 zabe@deploy1003: zabe: Backport for [[gerrit:1268574|Start reading from new file tables everwhere except enwiki and commons (T416548)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2399929 wikitext text/x-wiki 8kq5hnb2ydsvonixli6bggnyko859p3 2399930 2399929 2026-04-09T00:53:34Z Stashbot 7414 zabe@deploy1003: zabe: Continuing with sync 2399930 wikitext text/x-wiki k6v8xrf78jom8ahyo1nrrysx69x3pkd 2399931 2399930 2026-04-09T00:57:27Z Stashbot 7414 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1268574|Start reading from new file tables everwhere except enwiki and commons (T416548)]] (duration: 07m 40s) 2399931 wikitext text/x-wiki l9r1ln2u0vx18mhjr76bi21xexoqa8y 2399933 2399931 2026-04-09T02:00:55Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2399933 wikitext text/x-wiki d5513w1aihjy9xe2f4h7oek7nndimhf 2399934 2399933 2026-04-09T02:07:06Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 11s) 2399934 wikitext text/x-wiki 0stcqemhm7nt3mtxtribwgxjubtmis0 2399935 2399934 2026-04-09T02:09:23Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2399935 wikitext text/x-wiki 8s1b2s7bmo5ihlaaszwdui5lt5g86a9 2399936 2399935 2026-04-09T02:31:24Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2399936 wikitext text/x-wiki q5js93mc24o2dpmoasg1zg95mpwcrx4 2399937 2399936 2026-04-09T05:09:30Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool pc1016: Reimage 2399937 wikitext text/x-wiki lyawiusb9guuvjal6vrautklhwi6521 2399938 2399937 2026-04-09T05:09:32Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2399938 wikitext text/x-wiki lksllh2jtl41elts7dnd1x3fcc4q0qq 2399939 2399938 2026-04-09T05:09:38Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2399939 wikitext text/x-wiki c1tspy9zpusy6fdtn6oog7itvur1rki 2399940 2399939 2026-04-09T05:09:39Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1016: Reimage 2399940 wikitext text/x-wiki fxo56hsnkh189h1j1v1vt5sui4w8m3r 2399941 2399940 2026-04-09T05:10:54Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on pc2016.codfw.wmnet,pc1016.eqiad.wmnet with reason: Reimage to Debian Trixie 2399941 wikitext text/x-wiki gxbkahqfppkqyaadysnl35z9mczqudl 2399942 2399941 2026-04-09T05:13:45Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host pc2016.codfw.wmnet with OS trixie 2399942 wikitext text/x-wiki 6dmf9oioo6ib3gv4e6j2yur5cjumegj 2399943 2399942 2026-04-09T05:31:44Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on pc2016.codfw.wmnet with reason: host reimage 2399943 wikitext text/x-wiki bbpqunjkgk7mp64w9a44ngoxrs1cs93 2399944 2399943 2026-04-09T05:35:26Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc2016.codfw.wmnet with reason: host reimage 2399944 wikitext text/x-wiki ls0e1mdc8jc7gg2imgjfer5a8aaeq53 2399946 2399944 2026-04-09T05:59:18Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc2016.codfw.wmnet with OS trixie 2399946 wikitext text/x-wiki apoxm3at5vaxsxe6p1x2se2y0g82nu3 2399959 2399946 2026-04-09T06:38:45Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host pc1016.eqiad.wmnet with OS trixie 2399959 wikitext text/x-wiki hxwsedoxy73mo01mekc4hx460mrf1nk 2399960 2399959 2026-04-09T06:38:52Z Stashbot 7414 marostegui@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host pc1016.eqiad.wmnet with OS trixie 2399960 wikitext text/x-wiki nftuif38jr0eaxs4741bwox4ffdlsuo 2399961 2399960 2026-04-09T06:39:36Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host pc1016.eqiad.wmnet with OS trixie 2399961 wikitext text/x-wiki 63saac9x4j47prkiw93cteaaixlg02m 2399962 2399961 2026-04-09T06:55:09Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on pc1016.eqiad.wmnet with reason: host reimage 2399962 wikitext text/x-wiki f254xgj5jdsqrw3pbau0ivviyly76ep 2399963 2399962 2026-04-09T06:59:11Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1016.eqiad.wmnet with reason: host reimage 2399963 wikitext text/x-wiki g0754dz7fissgtuzz73mkvqgt0xvwbs 2399964 2399963 2026-04-09T07:03:59Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on pc2016.codfw.wmnet with reason: Maintenance 2399964 wikitext text/x-wiki kuk1sizy5aunzmiuvbivy4ox35h7twr 2399965 2399964 2026-04-09T07:16:56Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc1016.eqiad.wmnet with OS trixie 2399965 wikitext text/x-wiki sefdytcfm1l6w5kl9omib66lsjq3vj6 2399966 2399965 2026-04-09T07:17:15Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool pc1016: After reimage 2399966 wikitext text/x-wiki gqlv0236vc4t1sf7ncbmq247qrte0x0 2399967 2399966 2026-04-09T07:17:16Z Stashbot 7414 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache 2399967 wikitext text/x-wiki pjx631jm6cgk2ktmjg1wenvlihjchph 2399968 2399967 2026-04-09T07:17:29Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2399968 wikitext text/x-wiki d5x6dfwxexs9vkk1rqtnpdw8woxjex9 2399969 2399968 2026-04-09T07:17:30Z Stashbot 7414 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1016: After reimage 2399969 wikitext text/x-wiki cn9f8oph8wv4p8hmr8d54yeniamwgia 2399973 2399969 2026-04-09T08:11:53Z Stashbot 7414 fabfur: upgrading eqiad to haproxy 3.2 (T421402) 2399973 wikitext text/x-wiki dqx10yjb9eky8extkwdas4i60jx8fyg 2399974 2399973 2026-04-09T08:14:37Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqiad - 3.2 upgrade (T421402) 2399974 wikitext text/x-wiki 4pb0k81jkjdy7kaek65dc954i0qwmdz 2399975 2399974 2026-04-09T08:14:40Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqiad - 3.2 upgrade (T421402) 2399975 wikitext text/x-wiki 520usybkw9uyrdx5pi1xioi5iy0r8pt 2399976 2399975 2026-04-09T08:20:39Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-fe1013.eqiad.wmnet with OS bullseye 2399976 wikitext text/x-wiki geqanhuowdb3732rmmg3p0ajvhen9b1 2399977 2399976 2026-04-09T08:21:06Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-cluster pool all services in codfw/aux-codfw: maintenance 2399977 wikitext text/x-wiki cc09ezymmtt6dqrerseo4kojsev4puf 2399978 2399977 2026-04-09T08:21:10Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-fe1013 2399978 wikitext text/x-wiki retb1n5pehurutqjgd7m2iyar8szerg 2399979 2399978 2026-04-09T08:21:21Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.netbox 2399979 wikitext text/x-wiki c2cxm9nzbd6kuzsf8uqf5blrdd3ezlx 2399980 2399979 2026-04-09T08:23:26Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.k8s.pool-depool-cluster (exit_code=93) pool all services in codfw/aux-codfw: maintenance 2399980 wikitext text/x-wiki gpnfycz5lp3kflk8u7otzlaw77c3z8c 2399981 2399980 2026-04-09T08:26:27Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2157.codfw.wmnet with reason: Maintenance 2399981 wikitext text/x-wiki 3ve2675j3llmkjcc11uvzl7qgbceopv 2399982 2399981 2026-04-09T08:26:36Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2157 (T419635)', diff saved to https://phabricator.wikimedia.org/P90334 and previous config saved to /var/cache/conftool/dbconfig/20260409-082633-fceratto.json 2399982 wikitext text/x-wiki q7jer9aw41g4zzn56ywlb5remqoeuln 2399984 2399982 2026-04-09T08:41:08Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe1013 - mvernon@cumin2002" 2399984 wikitext text/x-wiki gee2nmbgtbil74fo6v579cr8v3q8uo4 2399985 2399984 2026-04-09T08:41:13Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe1013 - mvernon@cumin2002" 2399985 wikitext text/x-wiki ksx6241y8mk91948gobzfl8c1aelh54 2399986 2399985 2026-04-09T08:41:14Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2399986 wikitext text/x-wiki 2e31921fhvoz4b4obxca9db9uwip8tg 2399987 2399986 2026-04-09T08:41:15Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-fe1013.eqiad.wmnet 149.48.64.10.in-addr.arpa 9.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2399987 wikitext text/x-wiki 7zoitqnnubb7dlsuwqvvmszw08emeb1 2399988 2399987 2026-04-09T08:41:17Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-fe1013.eqiad.wmnet 149.48.64.10.in-addr.arpa 9.4.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2399988 wikitext text/x-wiki q216m2av6mwodk2xzivvriilz6xpuko 2399989 2399988 2026-04-09T08:41:18Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-fe1013 2399989 wikitext text/x-wiki c44hsngc38r0piroc9oc8woerwb6yfb 2399990 2399989 2026-04-09T08:41:39Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-fe1013 2399990 wikitext text/x-wiki fynlma6suzjwx5kdoqwre8mqo0hcdr7 2399991 2399990 2026-04-09T08:41:40Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-fe1013 2399991 wikitext text/x-wiki aj1jftvdlap0q9yp7tk4aqc4krh3b1v 2399992 2399991 2026-04-09T08:51:23Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_eqiad - 3.2 upgrade (T421402) 2399992 wikitext text/x-wiki c1lbwl09may8g0s415kt5cossbysk7y 2399993 2399992 2026-04-09T08:56:24Z Stashbot 7414 oblivian@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=k8s-ingress-aux-rw,name=codfw 2399993 wikitext text/x-wiki 5ufem3dnxnmzsqfct5gkt7pwfl6qda6 2399994 2399993 2026-04-09T08:57:43Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2157: Pooling in 2399994 wikitext text/x-wiki nq7mu4iwc7yysybmu5t0bihvbaq1l6k 2399995 2399994 2026-04-09T08:58:13Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1013.eqiad.wmnet with reason: host reimage 2399995 wikitext text/x-wiki h9dhllssaffadcca0vy40chx2885jpk 2399996 2399995 2026-04-09T08:59:09Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_eqiad - 3.2 upgrade (T421402) 2399996 wikitext text/x-wiki 14w79e4ymrr1qsda393qkudck4uf82m 2399997 2399996 2026-04-09T09:04:10Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1013.eqiad.wmnet with reason: host reimage 2399997 wikitext text/x-wiki p7c33bga5bfmflexm0m0e8vx5k15key 2399998 2399997 2026-04-09T09:06:12Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2399998 wikitext text/x-wiki 7fd681q2nbss0dwyvs63lx37ottm7tv 2399999 2399998 2026-04-09T09:07:17Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2399999 wikitext text/x-wiki 8zdnarfav704w6uarf3pkqceg7o9f4o 2400000 2399999 2026-04-09T09:10:38Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2400000 wikitext text/x-wiki b1g3ot3pg948d6nozkfda4qa2x86afe 2400001 2400000 2026-04-09T09:11:43Z Stashbot 7414 fabfur: upgrading esams to haproxy 3.2 (T421402) 2400001 wikitext text/x-wiki jbj88gy093ivobs3dyytilgo4bqce7k 2400002 2400001 2026-04-09T09:12:09Z Stashbot 7414 elukey: remove /var/run/confd-template/_var_lib_gdnsd_discovery-k8s-ingress-aux-rw.state.err on dns5004 and restart confd 2400002 wikitext text/x-wiki fg12oucgb12nwbygmrmuoi6bgsptc84 2400003 2400002 2026-04-09T09:15:51Z Stashbot 7414 elukey: remove /var/run/confd-template/_var_lib_gdnsd_discovery-k8s-ingress-aux-rw.state.err on affected dns servers and restart confd 2400003 wikitext text/x-wiki sig4zkq9383veu2rntpxw2cfp6kxm9c 2400004 2400003 2026-04-09T09:16:20Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_esams - 3.2 upgrade (T421402) 2400004 wikitext text/x-wiki hbuy7t09ziyekex1yaoasxzusby5qr3 2400005 2400004 2026-04-09T09:16:27Z Stashbot 7414 fabfur@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_esams - 3.2 upgrade (T421402) 2400005 wikitext text/x-wiki cxytwhk3qdqf2y3el3ibtoiw63uutik 2400006 2400005 2026-04-09T09:22:06Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1013.eqiad.wmnet with OS bullseye 2400006 wikitext text/x-wiki bs1lchomqj853ntc42h1mx34fjwm0p7 2400007 2400006 2026-04-09T09:25:03Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on P{ms-fe[1009-1012,1014-1024].eqiad.wmnet} and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) 2400007 wikitext text/x-wiki nb9mt5ytp40r864c3lhuq75kip6rg7t 2400008 2400007 2026-04-09T09:33:30Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on P{ms-fe[1009-1012,1014-1024].eqiad.wmnet} and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) 2400008 wikitext text/x-wiki l0eajs11xm47oa2b6rdi9c1wshidrml 2400009 2400008 2026-04-09T09:43:14Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2157: Pooling in 2400009 wikitext text/x-wiki pgvu88ydsujaygkqqu7uqg0ocfcj3n7 2400010 2400009 2026-04-09T09:58:57Z Stashbot 7414 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2400010 wikitext text/x-wiki 0o60rnj0ep4sy5y30w1d6njwnzueh9a 2400012 2400010 2026-04-09T10:04:59Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_esams - 3.2 upgrade (T421402) 2400012 wikitext text/x-wiki nyyi7cic24hh34hcp4c25jp1cmah1nx 2400013 2400012 2026-04-09T10:07:55Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-fe1019.eqiad.wmnet with OS bullseye 2400013 wikitext text/x-wiki po1uwi2tuva0bkm31sdl59exfmf9cor 2400014 2400013 2026-04-09T10:08:02Z Stashbot 7414 fabfur@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_esams - 3.2 upgrade (T421402) 2400014 wikitext text/x-wiki o6jxuc8jzapngmz7mxpuizp1vliezco 2400015 2400014 2026-04-09T10:08:20Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-fe1019 2400015 wikitext text/x-wiki rvt12b87bwjsfy52i9yi7u2vv8fr89c 2400016 2400015 2026-04-09T10:08:34Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.netbox 2400016 wikitext text/x-wiki ot4admgvhib4vfpvufqdttie7qx9fdx 2400017 2400016 2026-04-09T10:13:07Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe1019 - mvernon@cumin2002" 2400017 wikitext text/x-wiki ro4pa9q5dg3ktm8j0ztc96pw22h80dd 2400018 2400017 2026-04-09T10:13:13Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe1019 - mvernon@cumin2002" 2400018 wikitext text/x-wiki 76yyn5s1o51fk93rsy6axss5q6bzj1x 2400019 2400018 2026-04-09T10:13:14Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2400019 wikitext text/x-wiki 6ws5rsnvqr53mrh4vyxsfavvxfvz0pf 2400020 2400019 2026-04-09T10:13:15Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-fe1019.eqiad.wmnet 92.32.64.10.in-addr.arpa 2.9.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2400020 wikitext text/x-wiki 101pxelfv3fyknx86km5oz8uj2hrim9 2400021 2400020 2026-04-09T10:13:17Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-fe1019.eqiad.wmnet 92.32.64.10.in-addr.arpa 2.9.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2400021 wikitext text/x-wiki fz3xzemgik82c0fwmii2c5w83flbs6s 2400022 2400021 2026-04-09T10:13:18Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-fe1019 2400022 wikitext text/x-wiki fxupdjpnc9nqf3hpxf5chiprfmxvi8g 2400023 2400022 2026-04-09T10:13:42Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-fe1019 2400023 wikitext text/x-wiki 6pj5s1aatrwj8gc5ur7f59sl5u7f40e 2400024 2400023 2026-04-09T10:13:43Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-fe1019 2400024 wikitext text/x-wiki tmll2qjhk6gp9o81gif9ybo8dmdqdnt 2400025 2400024 2026-04-09T10:29:50Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1019.eqiad.wmnet with reason: host reimage 2400025 wikitext text/x-wiki b9vo6u91tocju2z96he0zi0so0jmn2n 2400026 2400025 2026-04-09T10:34:33Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1019.eqiad.wmnet with reason: host reimage 2400026 wikitext text/x-wiki syq8lsmy381ukgbwk7vknd2heijoyjl 2400027 2400026 2026-04-09T10:45:33Z Stashbot 7414 moritzm: upgrade debdeploy-server on cumin2002 to 0.0.99.14-1+deb12u1+exp2 (temporary build with Cumin 6 compat before we have Cumin 6 universally) 2400027 wikitext text/x-wiki 1tqukkqsb41biczd0gyk3hn9qlxxhye 2400028 2400027 2026-04-09T10:47:26Z Stashbot 7414 moritzm: installing openssl security updates 2400028 wikitext text/x-wiki rt528nd2zugr27rkxzyop8rz7m7456h 2400029 2400028 2026-04-09T10:54:16Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1019.eqiad.wmnet with OS bullseye 2400029 wikitext text/x-wiki 0gmqqe7c32t9s7xpeuc2pgb1l0wtkye 2400030 2400029 2026-04-09T10:55:34Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on P{ms-fe[1009-1018,1020-1024].eqiad.wmnet} and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) 2400030 wikitext text/x-wiki l4qtthq98jjbdx9hgbggugk6138ni3t 2400031 2400030 2026-04-09T11:01:03Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2400031 wikitext text/x-wiki irtt81k572jlzkhjtwrtwkt9mir5oiw 2400032 2400031 2026-04-09T11:01:12Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2400032 wikitext text/x-wiki 7hy8gsvkogx154la44d94nkrperwlgs 2400033 2400032 2026-04-09T11:01:26Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2400033 wikitext text/x-wiki 42at96c11o7th2dd9vvsqdx5yu7czoy 2400034 2400033 2026-04-09T11:01:43Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2400034 wikitext text/x-wiki 3vuho8995siqp7nae4ka9e6hvbmsxid 2400035 2400034 2026-04-09T11:04:16Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on P{ms-fe[1009-1018,1020-1024].eqiad.wmnet} and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) 2400035 wikitext text/x-wiki 31n1z5hlj3ukvyix6x9r89ie9b1g3rv 2400036 2400035 2026-04-09T11:04:54Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2400036 wikitext text/x-wiki cs34psohq0s7h2qseolucpsx9mzeda3 2400037 2400036 2026-04-09T11:05:00Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2400037 wikitext text/x-wiki i88bzjoq6greqjl5loq16h0z7tdz6yu 2400038 2400037 2026-04-09T11:16:53Z Stashbot 7414 moritzm: installing tiff security updates 2400038 wikitext text/x-wiki jsuh5mlum70jj6ux2rh451io7yahaw5 2400039 2400038 2026-04-09T11:25:53Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-fe1020.eqiad.wmnet with OS bullseye 2400039 wikitext text/x-wiki i515q6fej17baxzwammon1lgd634ecr 2400040 2400039 2026-04-09T11:26:24Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-fe1020 2400040 wikitext text/x-wiki ml88jzypjouza4wpommpe5gzu4rf7us 2400041 2400040 2026-04-09T11:27:04Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.netbox 2400041 wikitext text/x-wiki eyrtej4n7rdkvm2voba0zdfj8v7armf 2400042 2400041 2026-04-09T11:27:29Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry 2400042 wikitext text/x-wiki jlsju5u05gfeqq1bs5k1ny5pfhw6gmy 2400043 2400042 2026-04-09T11:28:50Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2400043 wikitext text/x-wiki cgzgdt5qmlupj4mnp2eeo2b2tqajde8 2400044 2400043 2026-04-09T11:29:00Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2400044 wikitext text/x-wiki 42xx5o8ozc9scv67pbue4xyiqwycanz 2400045 2400044 2026-04-09T11:29:52Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry 2400045 wikitext text/x-wiki brcn4xg9y3hfiaj49i0w7vsju7zjxva 2400046 2400045 2026-04-09T11:30:56Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe1020 - mvernon@cumin2002" 2400046 wikitext text/x-wiki 7q9eawufo66fkkpi5opvhuo87zwoje4 2400047 2400046 2026-04-09T11:31:01Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe1020 - mvernon@cumin2002" 2400047 wikitext text/x-wiki il0ep3twpfhbwc7as18yluctrcwh2b3 2400048 2400047 2026-04-09T11:31:02Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2400048 wikitext text/x-wiki bmh1plg1fas0va5xubguoh8766bltwm 2400049 2400048 2026-04-09T11:31:03Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-fe1020.eqiad.wmnet 113.48.64.10.in-addr.arpa 3.1.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2400049 wikitext text/x-wiki nol8bla878s65x358aoxj4eq9vmoqoe 2400050 2400049 2026-04-09T11:31:05Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-fe1020.eqiad.wmnet 113.48.64.10.in-addr.arpa 3.1.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2400050 wikitext text/x-wiki qqg504nx4285d2rg8x1ibsflyiy6iq6 2400051 2400050 2026-04-09T11:31:06Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-fe1020 2400051 wikitext text/x-wiki fxqu1c5sbtdi5abhx25uvy3bh2pyox9 2400052 2400051 2026-04-09T11:44:52Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-fe1020 2400052 wikitext text/x-wiki ha05z4eqg36pziwfmavcnbslzhidfno 2400053 2400052 2026-04-09T11:44:53Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-fe1020 2400053 wikitext text/x-wiki eb4c3xs9nyeg9mwh8bkmnavqh9s9rzf 2400054 2400053 2026-04-09T11:51:30Z Stashbot 7414 moritzm: installing nginx security updates 2400054 wikitext text/x-wiki imdphxtaie84gvv4p5x2rvlgwnlzt7r 0 17290 2399860 2399554 2026-04-08T17:54:51Z Stashbot 7414 bd808: Unblock 167.56.0.0/13 (T422721) 2399860 wikitext text/x-wiki g1i2rrrc812m91k3i1g5stswhe49ald 2399889 2399860 2026-04-08T19:26:50Z Stashbot 7414 brennen: gitlab-webhooks: building & deploying https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/37 - hitting some build tooling stuff, trying a fix per instructions in the error log 2399889 wikitext text/x-wiki lqf465gqbqulgjdywhqgdmoj2rwhzfu 2399918 2399889 2026-04-08T22:01:34Z Stashbot 7414 bd808: Unblock 95.216.12.170/32 (T422751) 2399918 wikitext text/x-wiki 2gy0a4gl7lr7nh5xwr9e1chhypby7ql 2399920 2399918 2026-04-08T22:19:03Z Stashbot 7414 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1269068 2399920 wikitext text/x-wiki hhzudl0en13zovhrx8v894exeopi8ly 2399972 2399920 2026-04-09T07:48:46Z Stashbot 7414 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1268559 "Zuul: use standalone jobs for GrowthExperiments Cypress tests" | T417412 2399972 wikitext text/x-wiki 1tii4s9ttntt4ulg1wxjo2xsmc378kg 12 18681 2399971 2354315 2026-04-09T07:38:57Z Novem Linguae 23682 /* Latest supported pre-built images */ bypass redirect, add description 2399971 wikitext text/x-wiki muvcllyqz782tbws7p9f7tn9ei27kz0 0 20365 2399819 2395166 2026-04-08T16:56:27Z Martin Urbanec 6634 /* Debugging a maintenance script */ document how this can be done in k8s 2399819 wikitext text/x-wiki bwtei54qwyqn5as1u8ecie5r8ypp04m 2399823 2399819 2026-04-08T17:03:19Z Martin Urbanec 6634 /* Debugging a maintenance script */ update for clarity 2399823 wikitext text/x-wiki 9iruozl7mzza9wttt1m915wgv36qxkq 2399910 2399823 2026-04-08T21:27:55Z BryanDavis 1604 /* Debugging a maintenance script */ add syntaxhighlight lang 2399910 wikitext text/x-wiki 71cgsp5pt0h5tatms888cb15fg4q8ew 498 30942 2399673 2399672 2026-04-08T11:59:39Z Stashbot 7414 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for service: project,designate 2399673 wikitext text/x-wiki 1j6222gqh45ndgs1emhnhns1f75y9n2 2399708 2399673 2026-04-08T13:35:03Z Stashbot 7414 godog: leave designate processes up only on cloudcontrol1006 to ease debugging - T422646 2399708 wikitext text/x-wiki kufe23xly8n9xvkeumkzvyr3yar7cy5 2399714 2399708 2026-04-08T13:48:33Z Stashbot 7414 godog: stop designate and memcached on all cloudcontrol1* 2399714 wikitext text/x-wiki c00bmr0g0inn8trg78p8j3hd9gd0e0f 2399715 2399714 2026-04-08T14:03:17Z Stashbot 7414 godog: bounce nova on cloudcontrol1006 2399715 wikitext text/x-wiki nekbd2syoneqf5wl30y1l7m9rdmxo8p 2399865 2399715 2026-04-08T18:17:34Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services 2399865 wikitext text/x-wiki mxm3l1184u9sxawhcpx6wsvxarqc9wq 2399866 2399865 2026-04-08T18:17:56Z Stashbot 7414 andrewbogott: restarting openstack services in eqiad1 before digging into a tofu failure 2399866 wikitext text/x-wiki 4ebr33h76ufzda1jjtj9oe213r94kg6 2399871 2399866 2026-04-08T18:30:38Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services 2399871 wikitext text/x-wiki 2isjl1g4bj66uvr0cgc6i4kl9rx8jmo 1 237771 2399899 2399671 2026-04-08T20:32:02Z BryanDavis 1604 /* #wikimedia-operations on telegram */ Reply 2399899 wikitext text/x-wiki h32iwge756zcxio8jfqol9tgf4oircz 12 440873 2399970 2395846 2026-04-09T07:28:09Z Novem Linguae 23682 Restart the webserver when you make changes to this file. 2399970 wikitext text/x-wiki 1dkz5qxhocd2oiqo4rqeg0dat7c2s5x 498 444707 2399807 2398682 2026-04-08T16:06:26Z Stashbot 7414 mutante: - upgrading openssl, libssl3 on gitlab-1002 2399807 wikitext text/x-wiki fm1g8zgj67wblha9sy4vn8toof57ynn 12 449113 2399818 2395336 2026-04-08T16:52:08Z DCaro (WMF) 18271 /* Internal log storage */ 2399818 wikitext text/x-wiki pttr6j9ggqbv2ntxva3v788lcmzocfv 0 449160 2399924 2399528 2026-04-09T00:00:34Z Dexbot 30554 Bot: Updating the report 2399924 wikitext text/x-wiki d9awol7f1ix7p8d85jgy9fb2hu0dtm1 2399983 2399924 2026-04-09T08:28:30Z Dexbot 30554 Bot: Updating the report 2399983 wikitext text/x-wiki 144vz8oud16oohofnfnjch3j3mjov34 12 451664 2399814 2399361 2026-04-08T16:29:16Z DCaro (WMF) 18271 /* Example: Python web service */ 2399814 wikitext text/x-wiki ncto6lah2mnkz51tzzxp2y5cers18ow 2399945 2399814 2026-04-09T05:48:53Z DCaro (WMF) 18271 /* User upgrade notes for the upgrade started on 2026-03-31 */ 2399945 wikitext text/x-wiki hvxvcy23gyvbt0f6fkt6tkumdn5ygrt 0 453101 2399840 2352229 2026-04-08T17:33:57Z Jgreen (wmf) 18 /* Service Administration */ 2399840 wikitext text/x-wiki or0e7fbnf7ax9eb0nkv7qfce9vw83yh 0 453597 2399816 2399061 2026-04-08T16:43:37Z DCaro (WMF) 18271 /* 2026 */ 2399816 wikitext text/x-wiki 02flc49pj2dtxal99hl9zw07hz4bwx3 0 459560 2399692 2395233 2026-04-08T12:59:15Z DAdedoyin (WMF) 39150 Test 2399692 wikitext text/x-wiki l6aq5nqo9pqk95j4qqm7gth3nm6enhi 0 459699 2399758 2399534 2026-04-08T15:12:48Z Stashbot 7414 ejegg: payments-wiki upgraded from 1ad85e6c to 064a770e 2399758 wikitext text/x-wiki f728125z1gdomhb235pi8th9ng249lr 2399856 2399758 2026-04-08T17:44:06Z Stashbot 7414 ejegg: fundraising civicrm upgraded from 4ee0b5e8 to e60321bb 2399856 wikitext text/x-wiki bmtofl23orxjzku9hgmx7jtnbb2r3uj 2399932 2399856 2026-04-09T01:20:43Z Stashbot 7414 ejegg: fundraising civicrm upgraded from e60321bb to d8d3871c 2399932 wikitext text/x-wiki 2clkzqk5r4yuv5ui4v9r1zdzh2kbs18 2400055 2399932 2026-04-09T11:53:30Z Stashbot 7414 jgleeson: SmashPig upgraded from 5c083891 to 100101fb 2400055 wikitext text/x-wiki sbtgukugwx0s58uqu5a9dvbueish434 0 459783 2399691 2398661 2026-04-08T12:52:27Z Duesentrieb 52342 /* Configuration and Implementation */ fix link 2399691 wikitext text/x-wiki os1bcuk2farrophjh3rdwfaf7pxcob2 2400011 2399691 2026-04-09T09:59:18Z Duesentrieb 52342 /* Configuration and Implementation */ fix links 2400011 wikitext text/x-wiki 91oi5b2azb8aysd86v1v67p9dyq619z 498 459970 2399804 2399336 2026-04-08T15:56:51Z Stashbot 7414 wmbot~difronzo@tools-bastion-15: deployed v0.7.7 (by DiFronzo on GitHub) 2399804 wikitext text/x-wiki pcbn516x2949ic7rzgm0o8qzv94smlv 2 460033 2399809 2399533 2026-04-08T16:09:12Z EEvans (WMF) 5035 more expansion 2399809 wikitext text/x-wiki bdumnhrp9atdyxmsfu4bhach5gn1a7x 2399810 2399809 2026-04-08T16:12:44Z EEvans (WMF) 5035 /* Configuration */ link to example gerrit 2399810 wikitext text/x-wiki qdtsrhz3e0xspxt68kdalslp1gacxix 2399817 2399810 2026-04-08T16:50:40Z EEvans (WMF) 5035 /* Running Puppet Locally */ formatting 2399817 wikitext text/x-wiki ssx6dk3tg5fzjy43zagz33sormc67ib 2399868 2399817 2026-04-08T18:25:43Z EEvans (WMF) 5035 /* Running Puppet Locally */ nits 2399868 wikitext text/x-wiki 2jttq5ej8vgi2o1oh7xpwfnp22829b8 2399869 2399868 2026-04-08T18:27:23Z EEvans (WMF) 5035 /* Run Puppet Everywhere Else */ nit 2399869 wikitext text/x-wiki thp7hmh95cpkyewau6k7gm5z163e9u6 2399870 2399869 2026-04-08T18:29:41Z EEvans (WMF) 5035 /* Begin the bootstrap(s) */ formatting 2399870 wikitext text/x-wiki iisazgfqaj9nncuaiavtt1t53k0p27e 2399878 2399870 2026-04-08T18:56:03Z EEvans (WMF) 5035 /* Disk Setup */ flesh out section 2399878 wikitext text/x-wiki 0nmihydiyrxaczug2cjlg7ec3c9fc6z 2399881 2399878 2026-04-08T19:00:06Z EEvans (WMF) 5035 /* Downtime Host */ more deets 2399881 wikitext text/x-wiki 6l1l560jp0v5uxfj0tbr9b84b4wkxm4 2399883 2399881 2026-04-08T19:01:56Z EEvans (WMF) 5035 /* Downtime Host */ nit 2399883 wikitext text/x-wiki f1aw6g6zl1vt0jr5ezmorxexqpuuip7 2399885 2399883 2026-04-08T19:10:32Z EEvans (WMF) 5035 /* Adjusting Cluster Stream Throughput */ more deets 2399885 wikitext text/x-wiki 4rpk2wy8wg0o5cfteufivc7l2jo0219 2399888 2399885 2026-04-08T19:24:50Z EEvans (WMF) 5035 /* Adjusting Cluster Stream Throughput */ more detail 2399888 wikitext text/x-wiki 41y3x4ak8rwj86z7k5cwnoghnb2hgqh 2399891 2399888 2026-04-08T19:35:18Z EEvans (WMF) 5035 /* Begin the bootstrap(s) */ more deets 2399891 wikitext text/x-wiki jwn48zd2b1rhc8y5ls8sgvqd4norjvy 2399921 2399891 2026-04-08T22:40:11Z EEvans (WMF) 5035 /* Begin the bootstrap(s) */ add email to invocation 2399921 wikitext text/x-wiki s7tpsidyrtx05lvrn8az8d1s9wdbmoy 2399922 2399921 2026-04-08T23:31:55Z EEvans (WMF) 5035 /* Remove the Downtime */ formatting 2399922 wikitext text/x-wiki h1rx526594t4daudde7wy9x4hlirev3 2399923 2399922 2026-04-08T23:33:41Z EEvans (WMF) 5035 formatting 2399923 wikitext text/x-wiki g6c8k0olu3m3ngr0m3d7f7xo7b77d19 2399927 2399923 2026-04-09T00:22:45Z EEvans (WMF) 5035 /* Remove the Downtime */ typo 2399927 wikitext text/x-wiki ntgc8mbu7fqakkxt3vgqkc03q21t9zs 3 460035 2399806 2026-04-08T16:01:40Z StrikerBot 8475 Welcome to Toolforge! 2399806 wikitext text/x-wiki 8pzirp74dv8nfec329i6ow7rqhbqrco