Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.45.0-wmf.6 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2314623 2314275 2025-06-18T14:23:37Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer 2314623 wikitext text/x-wiki n4jq8zt2sp2px3lv9qkwl047198nmhn 2314632 2314623 2025-06-18T14:35:49Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer 2314632 wikitext text/x-wiki g1kjle6vlb0a8z2p8suax7i82tvcmr4 0 4039 2314780 2302078 2025-06-18T17:34:43Z SBassett (WMF) 13019 /* Security mitigations */ s/1/l/ 2314780 wikitext text/x-wiki eqje0giaadvhloswjf4azpwmxx1381t 0 4108 2314651 2314499 2025-06-18T15:06:38Z ScheduleDeploymentBot 37566 Add [[gerrit:1160797]] to Thursday, June 19 UTC morning backport window 2314651 wikitext text/x-wiki ixxolr33idgjjpwn8g9264s1xmdocpm 2314678 2314651 2025-06-18T15:54:52Z ScheduleDeploymentBot 37566 Add [[gerrit:1160858]] to Wednesday, June 18 UTC late backport window 2314678 wikitext text/x-wiki 093jezoc1iv9toem7mgqct4vu5z20p0 2314827 2314678 2025-06-18T18:07:49Z ScheduleDeploymentBot 37566 Add [[gerrit:838270]] to Wednesday, June 18 UTC late backport window 2314827 wikitext text/x-wiki c0hgafa751xyg4x7rflaqvfcqc6k676 2314828 2314827 2025-06-18T18:08:02Z ScheduleDeploymentBot 37566 Add [[gerrit:838271]] to Wednesday, June 18 UTC late backport window 2314828 wikitext text/x-wiki cuv9xejw5cjenrdlker7j5x7u6or8ff 2314881 2314828 2025-06-18T19:19:40Z ScheduleDeploymentBot 37566 Add [[gerrit:1153722]] to Wednesday, June 18 UTC late backport window 2314881 wikitext text/x-wiki 5twpjk6plhyl3q7om8803neg4hbi7h1 2315037 2314881 2025-06-19T05:27:47Z ScheduleDeploymentBot 37566 Add [[gerrit:1161182]] to Thursday, June 19 UTC morning backport window 2315037 wikitext text/x-wiki gbtv12vh305ivaqk206sz321ni2nxpt 498 5034 2314676 2314267 2025-06-18T15:51:42Z BryanDavis 1604 /* Cherry-picking a patch from gerrit */ simplify PCC help link 2314676 wikitext text/x-wiki gdx85gkaigpumk88roo9l3boo7xf837 498 6726 2314604 2313890 2025-06-18T14:09:27Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer 2314604 wikitext text/x-wiki 5ekxec9a8niw1qx8zf1jdcsipstoys5 2314622 2314604 2025-06-18T14:23:00Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer 2314622 wikitext text/x-wiki s5z0mfmdzx58zl3hb6kgg59t7fiehdf 0 6762 2314521 2228614 2025-06-18T12:46:58Z JCrespo (WMF) 5427 /* Configured Pools */ director is now backup1014 2314521 wikitext text/x-wiki rujfo6l52vxkd4k015hhjk0xwkytxgl 2314522 2314521 2025-06-18T12:49:35Z JCrespo (WMF) 5427 update old references to backup1001 2314522 wikitext text/x-wiki dlmvl0t0y8rw42qysckzkod9b1aa6h8 0 7919 2314506 2314505 2025-06-18T12:01:19Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P78350 and previous config saved to /var/cache/conftool/dbconfig/20250618-120117-marostegui.json 2314506 wikitext text/x-wiki t423ytrj0gkg1zzh3r5p200ohk99432 2314508 2314506 2025-06-18T12:15:02Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply 2314508 wikitext text/x-wiki 0h7mdpee59j29u9omv4ffgnzv1cen0h 2314509 2314508 2025-06-18T12:15:48Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply 2314509 wikitext text/x-wiki heydavz1p2zktdczxxwso5717bi2bli 2314510 2314509 2025-06-18T12:16:27Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T396130)', diff saved to https://phabricator.wikimedia.org/P78351 and previous config saved to /var/cache/conftool/dbconfig/20250618-121624-marostegui.json 2314510 wikitext text/x-wiki aejeg49xqh29mxffnix5nk60mtgnnas 2314511 2314510 2025-06-18T12:16:41Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1218.eqiad.wmnet with reason: Maintenance 2314511 wikitext text/x-wiki 7i37szwb9dn47035j05fmqrskoi36zy 2314512 2314511 2025-06-18T12:16:49Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1218 (T396130)', diff saved to https://phabricator.wikimedia.org/P78352 and previous config saved to /var/cache/conftool/dbconfig/20250618-121646-marostegui.json 2314512 wikitext text/x-wiki icy9jgkwwbgp7l5w6fydpiuasiksz82 2314513 2314512 2025-06-18T12:32:45Z Stashbot 7414 jnuche@deploy1003: Installing scap version "4.178.0" for 183 host(s) 2314513 wikitext text/x-wiki f9efdtkfjojd0chw0lgxgn1t6kpzieq 2314514 2314513 2025-06-18T12:33:34Z Stashbot 7414 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1193.eqiad.wmnet with reason: Maintenance 2314514 wikitext text/x-wiki 70dfks5t99mnkqgbe9c8dzwkoubbmr6 2314515 2314514 2025-06-18T12:34:41Z Stashbot 7414 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2161.codfw.wmnet with reason: Maintenance 2314515 wikitext text/x-wiki hhft81bvduc2c5eyadpxal40pgs4mhw 2314516 2314515 2025-06-18T12:37:01Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T396130)', diff saved to https://phabricator.wikimedia.org/P78353 and previous config saved to /var/cache/conftool/dbconfig/20250618-123658-marostegui.json 2314516 wikitext text/x-wiki rh85dmdnbg9u4b5i5xddbouqok3d568 2314517 2314516 2025-06-18T12:43:29Z Stashbot 7414 elukey: drop old Thanos Swift's Tegola tile cache containers - T396584 2314517 wikitext text/x-wiki p0gy12h6ookh7zuu714h7dt0dajqf8p 2314518 2314517 2025-06-18T12:43:45Z Stashbot 7414 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet 2314518 wikitext text/x-wiki gfd2pavxxhjj6na312zg4421apb5dm8 2314519 2314518 2025-06-18T12:45:37Z Stashbot 7414 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet 2314519 wikitext text/x-wiki bo2v70wv5uh2tgzgfq5yezz72xg563r 2314523 2314519 2025-06-18T12:52:08Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P78354 and previous config saved to /var/cache/conftool/dbconfig/20250618-125206-marostegui.json 2314523 wikitext text/x-wiki 9browb3art4r94vgy8zs6dndufa5gx5 2314524 2314523 2025-06-18T12:54:13Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply 2314524 wikitext text/x-wiki 8gxcbgb7hx4iydao8vxdv1wl5err93s 2314525 2314524 2025-06-18T12:55:13Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply 2314525 wikitext text/x-wiki e8p9tl5ez9a29hzixjgwo71wnrjjtfn 2314527 2314525 2025-06-18T13:00:42Z Stashbot 7414 jynus: bacula director migration finalized, backup1014 is the new bacula director. backup1001 should no longer be used. T387892 2314527 wikitext text/x-wiki eiuj655cfu6gt7nm5yyszig22bet4bx 2314531 2314527 2025-06-18T13:07:15Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P78355 and previous config saved to /var/cache/conftool/dbconfig/20250618-130713-marostegui.json 2314531 wikitext text/x-wiki b51cg8acibsnwrf692dlbalt04zao67 2314532 2314531 2025-06-18T13:07:32Z Stashbot 7414 jnuche@deploy1003: Installing scap version "4.178.1" for 4 host(s) 2314532 wikitext text/x-wiki 90qvtprvk9q9jfas6hafhz7i9w9w9t0 2314533 2314532 2025-06-18T13:08:23Z Stashbot 7414 sukhe@cumin1002: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:esams or A:drmrs and A:cp - 9.2.10 upgrade (T390912) 2314533 wikitext text/x-wiki 686w6s7r18dayieacwltyd07bemt3iv 2314535 2314533 2025-06-18T13:10:24Z Stashbot 7414 jnuche@deploy1003: Installation of scap version "4.178.1" completed for 4 hosts 2314535 wikitext text/x-wiki h3ysdhx3f77xsj0wwhppjcan6ghhecc 2314537 2314535 2025-06-18T13:13:44Z Stashbot 7414 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.admin depooling P{lvs5004.eqsin.wmnet} and A:liberica (T396561) 2314537 wikitext text/x-wiki 6c5ksfizvu9oiu7y3v19jobfjytukyg 2314538 2314537 2025-06-18T13:14:09Z Stashbot 7414 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs5004.eqsin.wmnet} and A:liberica (T396561) 2314538 wikitext text/x-wiki berwfbexkie48z2ucabetbg98tw0cbz 2314539 2314538 2025-06-18T13:14:13Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host cloudcephosd2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314539 wikitext text/x-wiki c5v2i9krscp2553tmcuqkg04j47li3t 2314540 2314539 2025-06-18T13:14:26Z Stashbot 7414 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1160740|Enable the Contribute menu in 8th group of Wikipedias (T395084)]] 2314540 wikitext text/x-wiki h5zju1i5xb4pilx202aro5n56qg6fkq 2314541 2314540 2025-06-18T13:14:42Z Stashbot 7414 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs5004.eqsin.wmnet with reason: switching to katran 2314541 wikitext text/x-wiki 7cany4g5gnywysc1kad13ksi4zv8e5m 2314554 2314541 2025-06-18T13:15:45Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply 2314554 wikitext text/x-wiki 0bm89a4ezqwesiiqsgcwa17rracxdj3 2314555 2314554 2025-06-18T13:15:49Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314555 wikitext text/x-wiki 0ur5svk9rxbf5dd0tb56b9qormuz683 2314556 2314555 2025-06-18T13:16:44Z Stashbot 7414 kartik@deploy1003: kartik: Backport for [[gerrit:1160740|Enable the Contribute menu in 8th group of Wikipedias (T395084)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2314556 wikitext text/x-wiki 3vq669v9jdmdl1qc5es4ez6u2pi3fez 2314557 2314556 2025-06-18T13:18:04Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host cloudcephosd2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314557 wikitext text/x-wiki 81cu5ij8fbibv8fg5e78ckvqd2a9fiq 2314558 2314557 2025-06-18T13:19:22Z Stashbot 7414 kartik@deploy1003: kartik: Continuing with sync 2314558 wikitext text/x-wiki q9hi57d0p1mv5ho2i97m12r9w0l49m6 2314559 2314558 2025-06-18T13:19:39Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply 2314559 wikitext text/x-wiki szegq4sa3kq1haqbfn4kmrsliw2y6zl 2314560 2314559 2025-06-18T13:22:23Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T396130)', diff saved to https://phabricator.wikimedia.org/P78356 and previous config saved to /var/cache/conftool/dbconfig/20250618-132220-marostegui.json 2314560 wikitext text/x-wiki grz0seuo3205mgysf3o8w5s70885f2w 2314561 2314560 2025-06-18T13:22:37Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1219.eqiad.wmnet with reason: Maintenance 2314561 wikitext text/x-wiki 89itrbn3597cnnzhihpv3r1lmds3nrg 2314562 2314561 2025-06-18T13:22:46Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1219 (T396130)', diff saved to https://phabricator.wikimedia.org/P78357 and previous config saved to /var/cache/conftool/dbconfig/20250618-132242-marostegui.json 2314562 wikitext text/x-wiki j609y5a5plyfdvq4xm8u746kjzmdb7m 2314563 2314562 2025-06-18T13:26:23Z Stashbot 7414 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1160740|Enable the Contribute menu in 8th group of Wikipedias (T395084)]] (duration: 11m 57s) 2314563 wikitext text/x-wiki 555l35psm27vofx78hwowhro68xm7ds 2314564 2314563 2025-06-18T13:29:33Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314564 wikitext text/x-wiki ssjxelx04cgz11vfpq2ci3pdt6a8gw0 2314565 2314564 2025-06-18T13:29:59Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host cloudcephosd2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314565 wikitext text/x-wiki jwq4omck3uypwy6n9t01v0rpb163t1f 2314566 2314565 2025-06-18T13:30:46Z Stashbot 7414 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs5004.eqsin.wmnet 2314566 wikitext text/x-wiki 852zcl3kiw96togcx1s5821dnit0vu9 2314567 2314566 2025-06-18T13:30:49Z Stashbot 7414 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs5004.eqsin.wmnet 2314567 wikitext text/x-wiki gmdcw7fhl7hlk9tf2rimftf7i6rolro 2314568 2314567 2025-06-18T13:36:27Z Stashbot 7414 jnuche@deploy1003: Installing scap version "4.178.2" for 4 host(s) 2314568 wikitext text/x-wiki m4w2gqqrf2n5as0z9zkxetwt61a7822 2314569 2314568 2025-06-18T13:37:11Z Stashbot 7414 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs5004.eqsin.wmnet} and A:liberica (T396561) 2314569 wikitext text/x-wiki it0fzt4z94p5flg19jgadrs64q6aibf 2314570 2314569 2025-06-18T13:37:16Z Stashbot 7414 vgutierrez: repool lvs5004 (text) using katran - T396561 2314570 wikitext text/x-wiki 9umt2maebz32ksgun1eya9czhvhc8rx 2314571 2314570 2025-06-18T13:37:31Z Stashbot 7414 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs5004.eqsin.wmnet} and A:liberica (T396561) 2314571 wikitext text/x-wiki ecjy7f8l9fwuccqo2bo9q8bvs6ynv1f 2314572 2314571 2025-06-18T13:39:16Z Stashbot 7414 jnuche@deploy1003: Installation of scap version "4.178.2" completed for 4 hosts 2314572 wikitext text/x-wiki pjy0p7jlpqteh3nv6x0q4hj54tahiks 2314573 2314572 2025-06-18T13:42:32Z Stashbot 7414 moritzm: installing net-tools regression updates on Bullseye 2314573 wikitext text/x-wiki firmvbivpxnf4o8794z1mlv5wmpv83p 2314574 2314573 2025-06-18T13:43:10Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T396130)', diff saved to https://phabricator.wikimedia.org/P78358 and previous config saved to /var/cache/conftool/dbconfig/20250618-134307-marostegui.json 2314574 wikitext text/x-wiki 76t9g1yxaw1h627uqigrlhdpocffnf3 2314576 2314574 2025-06-18T13:47:08Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314576 wikitext text/x-wiki o5n48wgkmpwdyqd97163nqzi3hc33t7 2314577 2314576 2025-06-18T13:47:38Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host cloudcephosd2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314577 wikitext text/x-wiki s68b0kj209xgvm2wvgtk3819zhgxr0q 2314578 2314577 2025-06-18T13:54:19Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2314578 wikitext text/x-wiki 2mk3hmjz6wjc2mxflvblyij0ko4agrl 2314579 2314578 2025-06-18T13:54:32Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2314579 wikitext text/x-wiki n82wsje2p9bt0x0rhs1cttif15d8qua 2314581 2314579 2025-06-18T13:56:54Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host cloudcephosd2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314581 wikitext text/x-wiki lyyludkd175tazbhq02el18nlqd4ujt 2314582 2314581 2025-06-18T13:58:17Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P78359 and previous config saved to /var/cache/conftool/dbconfig/20250618-135814-marostegui.json 2314582 wikitext text/x-wiki pe5a9cvn0rw4c2r3gshc1dufnpgne8d 2314583 2314582 2025-06-18T13:59:27Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd2005'] 2314583 wikitext text/x-wiki kjrz0viulpyyhzaz8nieafmoskcw89m 2314584 2314583 2025-06-18T13:59:38Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudcephosd2005'] 2314584 wikitext text/x-wiki ijz4pkwyc37o67g4n9h9eyyith1es6p 2314585 2314584 2025-06-18T14:00:01Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2314585 wikitext text/x-wiki 9arf18a0inl2pvd4n9ulopg3qhljwj6 2314586 2314585 2025-06-18T14:00:06Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host cloudcephosd2005.codfw.wmnet with OS bullseye 2314586 wikitext text/x-wiki kplwvrdwbtsbyy8kvtb54rxshw51ugt 2314588 2314586 2025-06-18T14:03:22Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2314588 wikitext text/x-wiki d98h2efmc60bk7gzhkwpr4p19wqg2qd 2314589 2314588 2025-06-18T14:03:56Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2314589 wikitext text/x-wiki rucni42ats07a0g8bz2gplhuojwxrtk 2314590 2314589 2025-06-18T14:04:03Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314590 wikitext text/x-wiki tl8szdg901asl9kjwmzqrfrb2g4ztd5 2314593 2314590 2025-06-18T14:05:33Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd2006'] 2314593 wikitext text/x-wiki anhq2vw0rpe3oa0evi5rxsonuhnowe2 2314594 2314593 2025-06-18T14:05:43Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudcephosd2006'] 2314594 wikitext text/x-wiki jv4kl2bvsi1q03bytdav6bke93qaiom 2314596 2314594 2025-06-18T14:06:18Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host cloudcephosd2006.codfw.wmnet with OS bullseye 2314596 wikitext text/x-wiki gup3zwxulsg02xsyoicghnzt293bgle 2314601 2314596 2025-06-18T14:08:18Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2314601 wikitext text/x-wiki 6a6ftdsga9wdsb3rx041kvvsaidtic9 2314602 2314601 2025-06-18T14:08:58Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2314602 wikitext text/x-wiki bvtqtvch8izurru30p6ir6m514vr4ob 2314603 2314602 2025-06-18T14:09:01Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2314603 wikitext text/x-wiki pelijt586nblkg9yxkv9i3yuyy2a2hw 2314605 2314603 2025-06-18T14:09:42Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2314605 wikitext text/x-wiki i7v7lv57ifwabpjzd9nhuqwq2dtlgxw 2314606 2314605 2025-06-18T14:10:31Z Stashbot 7414 jnuche@deploy1003: Installing scap version "4.178.3" for 4 host(s) 2314606 wikitext text/x-wiki jtwjm8bfc71wttv0els9f1l03po4h4d 2314607 2314606 2025-06-18T14:13:23Z Stashbot 7414 jnuche@deploy1003: Installation of scap version "4.178.3" completed for 4 hosts 2314607 wikitext text/x-wiki f28tmvlaohyjtreldx694zttj3fwgbp 2314608 2314607 2025-06-18T14:13:26Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P78360 and previous config saved to /var/cache/conftool/dbconfig/20250618-141322-marostegui.json 2314608 wikitext text/x-wiki g6wav9d8nu6rhvxdwr3ifzs8d8vb6hm 2314610 2314608 2025-06-18T14:13:48Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2314610 wikitext text/x-wiki npjorcykgal2stc08ti07b7pjcx7mc1 2314611 2314610 2025-06-18T14:15:26Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2314611 wikitext text/x-wiki 5inrftzf9svw19n9mst89dy2psbyi05 2314612 2314611 2025-06-18T14:17:21Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2314612 wikitext text/x-wiki 64cjcgcrxq24d0gu5wv2120te0dd2l7 2314613 2314612 2025-06-18T14:17:53Z Stashbot 7414 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2314613 wikitext text/x-wiki lds8slstmhz1glz4lpc9n75uncvyf8b 2314614 2314613 2025-06-18T14:18:06Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2314614 wikitext text/x-wiki j2ndh8uupmjrwu7zv0eb4dhuyy77a0q 2314615 2314614 2025-06-18T14:18:50Z Stashbot 7414 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314615 wikitext text/x-wiki 4gmsiu8t9x1zzxm9dab3m1rkilsr8da 2314616 2314615 2025-06-18T14:18:53Z Stashbot 7414 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2314616 wikitext text/x-wiki cc3mwq0f52annai6br74f32rqghwvn9 2314617 2314616 2025-06-18T14:19:21Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd2007'] 2314617 wikitext text/x-wiki qb18z4gdb3q8yf8af5raidrh5yvtlit 2314618 2314617 2025-06-18T14:19:34Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudcephosd2007'] 2314618 wikitext text/x-wiki dupabyg3dnk0dxe5rebif2pubu7vxcn 2314619 2314618 2025-06-18T14:20:28Z Stashbot 7414 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host cloudcephosd2007.codfw.wmnet with OS bullseye 2314619 wikitext text/x-wiki 5ctsids2jibmy2o0kjxzjn3krn0c48v 2314620 2314619 2025-06-18T14:21:44Z Stashbot 7414 bking@deploy1003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply 2314620 wikitext text/x-wiki 103cxdbb5gnmsk62z1lyutthfuy5bf6 2314621 2314620 2025-06-18T14:22:00Z Stashbot 7414 bking@deploy1003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply 2314621 wikitext text/x-wiki e03655jiibu5fx001r46mkxyzghhnf7 2314624 2314621 2025-06-18T14:24:15Z Stashbot 7414 bking@deploy1003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply 2314624 wikitext text/x-wiki jny2l1uxm27vlot2w2xp7xfvx266s0j 2314625 2314624 2025-06-18T14:24:23Z Stashbot 7414 bking@deploy1003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply 2314625 wikitext text/x-wiki 6snh1dfqwapwpinnw9v599u61dfdz90 2314626 2314625 2025-06-18T14:24:33Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2314626 wikitext text/x-wiki fy5m81dbi48rc6a071vwqw0dh1fcmgu 2314627 2314626 2025-06-18T14:26:56Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2314627 wikitext text/x-wiki c6b65gvkr1002vjfznv0gdiexyjbvgj 2314628 2314627 2025-06-18T14:28:33Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T396130)', diff saved to https://phabricator.wikimedia.org/P78363 and previous config saved to /var/cache/conftool/dbconfig/20250618-142829-marostegui.json 2314628 wikitext text/x-wiki i1bba91evth2cnuhj7yr3p42j3rsacp 2314629 2314628 2025-06-18T14:28:46Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1232.eqiad.wmnet with reason: Maintenance 2314629 wikitext text/x-wiki 4say2rlg46kjynrjp3817i8zq467b4w 2314630 2314629 2025-06-18T14:28:55Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1232 (T396130)', diff saved to https://phabricator.wikimedia.org/P78364 and previous config saved to /var/cache/conftool/dbconfig/20250618-142852-marostegui.json 2314630 wikitext text/x-wiki 5u6camw9f7td1iqp9br7sd5tdrp0i2w 2314633 2314630 2025-06-18T14:40:28Z Stashbot 7414 James_F: Running `mwscript-k8s --php_version=8.1 -f -- extensions/WikiLambda/maintenance/updateSecondaryTables.php --wiki=wikifunctionswiki --cache --verbose --zType Z8` for T396449 2314633 wikitext text/x-wiki 9ulyw9om6lgew3dc0zuvya8cduv5yla 2314634 2314633 2025-06-18T14:41:34Z Stashbot 7414 bking@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply 2314634 wikitext text/x-wiki nq5pwugdxx61hv85yterk0kmphxrciz 2314635 2314634 2025-06-18T14:41:43Z Stashbot 7414 bking@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply 2314635 wikitext text/x-wiki l3dmqz40si5bpv54npzgtll7qpl5tgw 2314636 2314635 2025-06-18T14:45:21Z Stashbot 7414 bking@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply 2314636 wikitext text/x-wiki 340n2gp48a66uwkg5tmz976qxy6z8jo 2314637 2314636 2025-06-18T14:45:26Z Stashbot 7414 bking@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply 2314637 wikitext text/x-wiki 19w5u04adxu180acw2eyjvczi3zol5s 2314638 2314637 2025-06-18T14:47:07Z Stashbot 7414 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet 2314638 wikitext text/x-wiki ji965118j571sszkrum1lodvyq10u0g 2314639 2314638 2025-06-18T14:48:14Z Stashbot 7414 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet 2314639 wikitext text/x-wiki g314nnhwr2d0kre5jrwuivp6lc05kyq 2314640 2314639 2025-06-18T14:49:06Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T396130)', diff saved to https://phabricator.wikimedia.org/P78365 and previous config saved to /var/cache/conftool/dbconfig/20250618-144903-marostegui.json 2314640 wikitext text/x-wiki kyuk4qlyyf91bhuzagf3e52myvlluci 2314641 2314640 2025-06-18T14:50:28Z Stashbot 7414 swfrench-wmf: reprepro included conftool 5.3.0 in apt.wikimedia.org - T395696 2314641 wikitext text/x-wiki 8ovljs5sz0i52lxusc1vb1sauoqds5d 2314642 2314641 2025-06-18T14:51:34Z Stashbot 7414 jmm@puppetserver1001: conftool action : set/pooled=no; selector: name=ncredir7002.magru.wmnet 2314642 wikitext text/x-wiki tpximy8dy4y1echfjks7ei811slzs9g 2314643 2314642 2025-06-18T14:51:38Z Stashbot 7414 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ncredir7002.magru.wmnet 2314643 wikitext text/x-wiki db85c96q5o7cblp0i4u4muc8uw54l23 2314646 2314643 2025-06-18T14:59:40Z Stashbot 7414 ejegg: civicrm upgraded from 63302c18 to 10eac2f8 2314646 wikitext text/x-wiki jpn6kt9k6ee7zlsbohvkwa6qn8ir1ih 2314647 2314646 2025-06-18T15:01:15Z Stashbot 7414 dancy@deploy1003: Started scap sync-world: Testing T396166 2314647 wikitext text/x-wiki 3n2lsk1cxvvn69albw828l4wccmzyui 2314649 2314647 2025-06-18T15:03:09Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-coord1003.eqiad.wmnet 2314649 wikitext text/x-wiki nm5as3gb0qown7xr89euexau5en1kcu 2314650 2314649 2025-06-18T15:04:12Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P78366 and previous config saved to /var/cache/conftool/dbconfig/20250618-150410-marostegui.json 2314650 wikitext text/x-wiki 84mw9h1b577lgadki1ham69tghgxc9q 2314652 2314650 2025-06-18T15:09:53Z Stashbot 7414 dancy@deploy1003: Finished scap sync-world: Testing T396166 (duration: 08m 37s) 2314652 wikitext text/x-wiki aexaf6ca9erg7p4knlmv6e548tv51qs 2314653 2314652 2025-06-18T15:19:20Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P78367 and previous config saved to /var/cache/conftool/dbconfig/20250618-151918-marostegui.json 2314653 wikitext text/x-wiki a4fnlp72tvkc2cs630ys0atwqth0mas 2314655 2314653 2025-06-18T15:21:31Z Stashbot 7414 ejegg: enabled queue consumer to send donor portal login links 2314655 wikitext text/x-wiki lxfaj4454qwmtj4bgpto7bukku3jkwl 2314656 2314655 2025-06-18T15:27:25Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply 2314656 wikitext text/x-wiki 0raczr0xykrppw4tab8ue59w2dby4v8 2314657 2314656 2025-06-18T15:29:02Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply 2314657 wikitext text/x-wiki ddfldt8qowzstngmpzcgtwi42fgxko4 2314658 2314657 2025-06-18T15:29:30Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply 2314658 wikitext text/x-wiki 0edpms63wch6harhr775ei1jbi4ryqn 2314659 2314658 2025-06-18T15:29:59Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply 2314659 wikitext text/x-wiki lyosdth7sfbu16shae61xx49svtv49a 2314660 2314659 2025-06-18T15:30:26Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply 2314660 wikitext text/x-wiki 3tk80x5170tkozbk28nrvebtdrmuv5f 2314661 2314660 2025-06-18T15:30:37Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply 2314661 wikitext text/x-wiki 7tllesd20gmb64yqldpmzpyx4ao222z 2314662 2314661 2025-06-18T15:30:58Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply 2314662 wikitext text/x-wiki 37s4ifzbecdzxubs5x64as46sddn4xl 2314663 2314662 2025-06-18T15:31:25Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply 2314663 wikitext text/x-wiki qlr6px71j76osm5x95elxtvyv0pj4xl 2314664 2314663 2025-06-18T15:31:41Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply 2314664 wikitext text/x-wiki pyi2sistbu58ingtmyqilfpu9l9s3rj 2314665 2314664 2025-06-18T15:34:30Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T396130)', diff saved to https://phabricator.wikimedia.org/P78368 and previous config saved to /var/cache/conftool/dbconfig/20250618-153425-marostegui.json 2314665 wikitext text/x-wiki ebyhj9o4a30xgcrj7pitbdgwb8qi0zr 2314666 2314665 2025-06-18T15:34:43Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1234.eqiad.wmnet with reason: Maintenance 2314666 wikitext text/x-wiki 5xp0n0jqxtkuwuzll9o3wraz3q7wajg 2314667 2314666 2025-06-18T15:34:52Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1234 (T396130)', diff saved to https://phabricator.wikimedia.org/P78369 and previous config saved to /var/cache/conftool/dbconfig/20250618-153448-marostegui.json 2314667 wikitext text/x-wiki 382rxmk563fy4j9temjjl5tll45j045 2314668 2314667 2025-06-18T15:35:17Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd2006.codfw.wmnet with OS bullseye 2314668 wikitext text/x-wiki 8hkfx7022eysian24tym2lc2ig4mygf 2314669 2314668 2025-06-18T15:37:04Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd2007.codfw.wmnet with OS bullseye 2314669 wikitext text/x-wiki r69gy3yexietnze954kor6dwvcatv17 2314670 2314669 2025-06-18T15:38:19Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephosd2007.codfw.wmnet with OS bullseye 2314670 wikitext text/x-wiki c53dp4vlmkf83g7ycb3k6aa720dv9ui 2314671 2314670 2025-06-18T15:39:35Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply 2314671 wikitext text/x-wiki kl9kmclwxvmnbt0ir717igau5jqx5lp 2314672 2314671 2025-06-18T15:41:05Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-codfw and A:cp - Fix VSLbs() assert error and upgrade libvmod-wmfuniq to 0.2.0 (T396581) 2314672 wikitext text/x-wiki cb4xjvhneh27busypoe2570pfap18kj 2314673 2314672 2025-06-18T15:44:31Z Stashbot 7414 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd2005.codfw.wmnet with OS bullseye 2314673 wikitext text/x-wiki bby4lwzpyl2tg5v38hjobe4od9bzxu9 2314674 2314673 2025-06-18T15:45:44Z Stashbot 7414 brett: Depooling cp7001 for firmware upgrades re: thermal support ticket - T386959 2314674 wikitext text/x-wiki ss9mfj5f1ogbmmwtmobqd953lwroeeq 2314675 2314674 2025-06-18T15:45:48Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp7001.* 2314675 wikitext text/x-wiki qm6lv7x7t7706wvswf8c18zfnk7a5t4 2314677 2314675 2025-06-18T15:52:45Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2314677 wikitext text/x-wiki pkfs9u38j1r25rqr5qosxvgvb38kusg 2314679 2314677 2025-06-18T15:54:59Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T396130)', diff saved to https://phabricator.wikimedia.org/P78370 and previous config saved to /var/cache/conftool/dbconfig/20250618-155455-marostegui.json 2314679 wikitext text/x-wiki kmaav8g1kr5rh7vik32uenae49cqhzi 2314680 2314679 2025-06-18T15:57:59Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2314680 wikitext text/x-wiki 4euvhoyf437zsgnrgpycokg6pta9sf6 2314681 2314680 2025-06-18T15:58:14Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts an-coord1003.eqiad.wmnet 2314681 wikitext text/x-wiki 3qlmkx11dkw8pfvmmekq4xo8lc92ifv 2314682 2314681 2025-06-18T15:59:01Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2314682 wikitext text/x-wiki 698aea8qvyk5t7fw7g6sc0yonn5nd2g 2314683 2314682 2025-06-18T15:59:34Z Stashbot 7414 swfrench-wmf: deployed conftool 5.3.0 to all bullseye and bookworm hosts - T395696 2314683 wikitext text/x-wiki 594luhl3ybnc9scamp74xtl2buw0dm6 2314684 2314683 2025-06-18T16:02:51Z Stashbot 7414 btullis@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on an-mariadb1002.eqiad.wmnet with reason: Upgrading SSD firmware 2314684 wikitext text/x-wiki 17uctsa54yo5uodzqdiinpjstai3tyo 2314685 2314684 2025-06-18T16:03:13Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314685 wikitext text/x-wiki 2pjvb4y2xwrdylj8uqgpnmy9p59law6 2314686 2314685 2025-06-18T16:03:33Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-mariadb1002.eqiad.wmnet 2314686 wikitext text/x-wiki rpbreeuceposx3zlky2i1har18596ds 2314688 2314686 2025-06-18T16:10:05Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P78371 and previous config saved to /var/cache/conftool/dbconfig/20250618-161003-marostegui.json 2314688 wikitext text/x-wiki myh742eqxzi2xcfgmd99wz6bufezlfr 2314689 2314688 2025-06-18T16:10:18Z Stashbot 7414 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cp7001.magru.wmnet with reason: BIOS upgrades 2314689 wikitext text/x-wiki ckg986nqosd6tlc9p7wwz40vr2m163c 2314690 2314689 2025-06-18T16:10:30Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314690 wikitext text/x-wiki 9qu5gf50a5gozhba2p33bpy6m3tjlhe 2314691 2314690 2025-06-18T16:10:43Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet 2314691 wikitext text/x-wiki l6dxekvxv18cd550v513pz76ymze6or 2314692 2314691 2025-06-18T16:10:47Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314692 wikitext text/x-wiki d8h907hwgnrwkmr3zrbj7zveqelry1t 2314693 2314692 2025-06-18T16:11:45Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314693 wikitext text/x-wiki px9ft436igifgc4xdufa1b443txw7eu 2314694 2314693 2025-06-18T16:15:48Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314694 wikitext text/x-wiki p4eg4bnbcgj2h6mkkuqof5pdhr5yqst 2314695 2314694 2025-06-18T16:16:49Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314695 wikitext text/x-wiki 98rfkwuxwtpn44o407wei6256u89h7a 2314696 2314695 2025-06-18T16:19:41Z Stashbot 7414 hnowlan@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2314696 wikitext text/x-wiki ixgch0ob1efs35hj0qzwebxraqasscd 2314697 2314696 2025-06-18T16:19:58Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephosd2005.codfw.wmnet with OS bullseye 2314697 wikitext text/x-wiki 24vc7bwxsa7r9u1m01v3ox362nflx2h 2314698 2314697 2025-06-18T16:20:07Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet 2314698 wikitext text/x-wiki qt9lmcrwaaeydf5777bnhq0fi35ecbx 2314699 2314698 2025-06-18T16:20:10Z Stashbot 7414 btullis@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts an-mariadb1002.eqiad.wmnet 2314699 wikitext text/x-wiki luinhe2acc1vqu5qdr2pa9yplp5vvcj 2314700 2314699 2025-06-18T16:20:49Z Stashbot 7414 hnowlan@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2314700 wikitext text/x-wiki jknn3flju5d65iokf49kjphyqxgw8ns 2314701 2314700 2025-06-18T16:21:38Z Stashbot 7414 hnowlan@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2314701 wikitext text/x-wiki 09x63zn7tuzslbjrbxluknhlmsqdo4k 2314702 2314701 2025-06-18T16:21:55Z Stashbot 7414 sukhe@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:esams or A:drmrs and A:cp - 9.2.10 upgrade (T390912) 2314702 wikitext text/x-wiki 7ldh8dqgc8qkup3kgrbvofol9b6trva 2314703 2314702 2025-06-18T16:23:49Z Stashbot 7414 hnowlan@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2314703 wikitext text/x-wiki g5thdo5er7nmsypcssgstfbi712k40u 2314704 2314703 2025-06-18T16:25:13Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P78372 and previous config saved to /var/cache/conftool/dbconfig/20250618-162511-marostegui.json 2314704 wikitext text/x-wiki 7gb3pfhb0blxuogl65jfhqjq2x3ibwj 2314705 2314704 2025-06-18T16:30:34Z Stashbot 7414 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2314705 wikitext text/x-wiki 2zmzarq9l1vdps2ombirvtraa4pt1yy 2314706 2314705 2025-06-18T16:31:59Z Stashbot 7414 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2314706 wikitext text/x-wiki 4egvyb55akfqt0hqylxfna9q4n6iwgb 2314707 2314706 2025-06-18T16:33:47Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2314707 wikitext text/x-wiki khuhfqe8vzhne50ojk6ce6cydxqtakf 2314708 2314707 2025-06-18T16:34:12Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2314708 wikitext text/x-wiki o463w1gu8ms428txb60qt0y1sch9ead 2314709 2314708 2025-06-18T16:34:32Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply 2314709 wikitext text/x-wiki afeslbw3mqnd3jbh3ghewgyt7e3jnq6 2314710 2314709 2025-06-18T16:34:44Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply 2314710 wikitext text/x-wiki 6tt9678sct3msw243io0j0ya7b6zibz 2314711 2314710 2025-06-18T16:40:22Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T396130)', diff saved to https://phabricator.wikimedia.org/P78373 and previous config saved to /var/cache/conftool/dbconfig/20250618-164019-marostegui.json 2314711 wikitext text/x-wiki ivwemlxhf4wnhfwpzpppgzryhy64uze 2314712 2314711 2025-06-18T16:40:36Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1235.eqiad.wmnet with reason: Maintenance 2314712 wikitext text/x-wiki t019mfrf880ggcmcoc9aw2hwlb59jo5 2314713 2314712 2025-06-18T16:40:44Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1235 (T396130)', diff saved to https://phabricator.wikimedia.org/P78374 and previous config saved to /var/cache/conftool/dbconfig/20250618-164041-marostegui.json 2314713 wikitext text/x-wiki hl1cfh5c7gbhee6z5e14nu5b6uvu675 2314714 2314713 2025-06-18T16:47:04Z Stashbot 7414 ChrisDobbins901_: cdobbins@cumin2002:~$ sudo -i cookbook sre.cdn.roll-upgrade-ats --query 'A:cp-eqsin' --task-id T390912 --reason '9.2.10 upgrade' 2314714 wikitext text/x-wiki 1n9vnywqjz6apjrv4ogu659103d6utt 2314715 2314714 2025-06-18T16:47:29Z Stashbot 7414 cdobbins@cumin2002: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-eqsin and A:cp - 9.2.10 upgrade (T390912) 2314715 wikitext text/x-wiki ldv2p5l4f140hvpsfm2uq8kfn4arvoz 2314716 2314715 2025-06-18T16:50:05Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd2005.codfw.wmnet with OS bullseye 2314716 wikitext text/x-wiki r5kxu0zn6i8od7roc2uojgd99c58nvk 2314717 2314716 2025-06-18T16:50:34Z Stashbot 7414 dancy@deploy1003: Started scap sync-world: Backport for [[gerrit:1136044|Update entries on https://www.mediawiki.org/keys/keys.html (T364694)]] 2314717 wikitext text/x-wiki l9m192unk7fmg85k1z0rptmudnsy9ic 2314718 2314717 2025-06-18T16:52:48Z Stashbot 7414 dancy@deploy1003: dancy, aklapper: Backport for [[gerrit:1136044|Update entries on https://www.mediawiki.org/keys/keys.html (T364694)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2314718 wikitext text/x-wiki nd1yl1dvpz65my9lponjaw87dro5equ 2314719 2314718 2025-06-18T16:53:42Z Stashbot 7414 dancy@deploy1003: dancy, aklapper: Continuing with sync 2314719 wikitext text/x-wiki ox9j3e9yi0dmi4ocohwrnvg7kuqy4v6 2314720 2314719 2025-06-18T16:55:51Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.dns.netbox 2314720 wikitext text/x-wiki 2rqwz4hydtri6g1mae763eaq1x4bog3 2314721 2314720 2025-06-18T16:56:52Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd2007.codfw.wmnet with OS bullseye 2314721 wikitext text/x-wiki cl5iljw92xenzszhpvdhdlgucdjka7w 2314722 2314721 2025-06-18T16:59:22Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2314722 wikitext text/x-wiki 3r3qxpecgk7cb158j1t7mgex4xyxs60 2314723 2314722 2025-06-18T17:00:43Z Stashbot 7414 dancy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1136044|Update entries on https://www.mediawiki.org/keys/keys.html (T364694)]] (duration: 10m 09s) 2314723 wikitext text/x-wiki mxc8dntwutuoz1hfzp7wtkz91s0mwdd 2314724 2314723 2025-06-18T17:00:47Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding rdb2011 to codfw - jhancock@cumin1003" 2314724 wikitext text/x-wiki mreplmr5v150mqg3ky3gjrmn89cnmmj 2314725 2314724 2025-06-18T17:01:02Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding rdb2011 to codfw - jhancock@cumin1003" 2314725 wikitext text/x-wiki dmag32kz0uuiw2ptyxyg6mndn7dc5nu 2314726 2314725 2025-06-18T17:01:05Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2314726 wikitext text/x-wiki k6muz8910vfgkt60723xaiv8uwz9zft 2314727 2314726 2025-06-18T17:01:09Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host aux-k8s-worker2006 2314727 wikitext text/x-wiki ancm0ad0sqnc6yi9mjoh373st4tfj7f 2314728 2314727 2025-06-18T17:01:13Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T396130)', diff saved to https://phabricator.wikimedia.org/P78375 and previous config saved to /var/cache/conftool/dbconfig/20250618-170109-marostegui.json 2314728 wikitext text/x-wiki cz8edzm2z8rqtlxxze6va3q7jwp5bhn 2314729 2314728 2025-06-18T17:01:17Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host aux-k8s-worker2006 2314729 wikitext text/x-wiki 1c1tm6kcl7hhfe6388hpacfp48oxv9x 2314730 2314729 2025-06-18T17:01:20Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host aux-k8s-worker2007 2314730 wikitext text/x-wiki ssrvjys8b7kzcmnozohqbe58gbeeqvj 2314731 2314730 2025-06-18T17:01:30Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host aux-k8s-worker2007 2314731 wikitext text/x-wiki anp67tqi8fbj2l5tsr0uydy58lp0sxn 2314732 2314731 2025-06-18T17:01:33Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host aux-k8s-worker2008 2314732 wikitext text/x-wiki g3393sac864pa6zrjn2mb047e7b9koh 2314733 2314732 2025-06-18T17:01:42Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host aux-k8s-worker2008 2314733 wikitext text/x-wiki 6mqc8cuj90ijzqqkwz10m5waq38xw7y 2314734 2314733 2025-06-18T17:01:45Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host aux-k8s-worker2009 2314734 wikitext text/x-wiki 6vi0e8984qqahnx94lytqlw6ntakzzr 2314735 2314734 2025-06-18T17:01:56Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host aux-k8s-worker2009 2314735 wikitext text/x-wiki sj9fw3so3b8st4q85vvba4e2fn5yj4t 2314736 2314735 2025-06-18T17:01:59Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host build2003 2314736 wikitext text/x-wiki l1aphnqu1nk7zvkysinyncqtnx6rxlw 2314737 2314736 2025-06-18T17:02:08Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host build2003 2314737 wikitext text/x-wiki 75lay3d8owvzex5s2lkmi5xzd63mqf1 2314738 2314737 2025-06-18T17:02:13Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host rdb2011 2314738 wikitext text/x-wiki seuv5ig5kkql86kwq01jlwd3115gv0w 2314739 2314738 2025-06-18T17:02:21Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host rdb2011 2314739 wikitext text/x-wiki igd1zvj2ylrlmy6p0qup964r5fiydo3 2314740 2314739 2025-06-18T17:02:28Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host rdb2012 2314740 wikitext text/x-wiki 548cewqyzykdhyy0s0s6y6ye19yrboc 2314741 2314740 2025-06-18T17:02:42Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host rdb2012 2314741 wikitext text/x-wiki 69w5yortefl635ec51ux6thjjdbvr3t 2314742 2314741 2025-06-18T17:02:45Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host sretest2003 2314742 wikitext text/x-wiki t8sgr1dki1uqy3ilmrvnc6ltqlctp8k 2314743 2314742 2025-06-18T17:02:54Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host sretest2003 2314743 wikitext text/x-wiki 14e4dejsj6nvagsxb3olgrxyv2t76d6 2314744 2314743 2025-06-18T17:03:16Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host sretest2004 2314744 wikitext text/x-wiki jb4idjv0qoon6nvjwzf6ro6mqa3rhit 2314745 2314744 2025-06-18T17:03:25Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host sretest2004 2314745 wikitext text/x-wiki 5gg94zh0dbvw0l06gtpyzpxxr8tq6pq 2314746 2314745 2025-06-18T17:03:29Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host sretest2006 2314746 wikitext text/x-wiki q6v6v08pf3tnevwp80c1b8ue0ljarpb 2314747 2314746 2025-06-18T17:03:43Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host sretest2006 2314747 wikitext text/x-wiki bi130ns00nrkfi8zp0dh8jyhplurby2 2314748 2314747 2025-06-18T17:03:47Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host sretest2009 2314748 wikitext text/x-wiki 18fxki0lqoh69arznxo56f6ozlj13n3 2314749 2314748 2025-06-18T17:03:55Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host sretest2009 2314749 wikitext text/x-wiki jcos44d8tavp4mk4o55u7nivk72tkf3 2314750 2314749 2025-06-18T17:04:09Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.dns.netbox 2314750 wikitext text/x-wiki 3wlozo289aoqjjw6rs0jx1zhd8hsz67 2314751 2314750 2025-06-18T17:06:43Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2314751 wikitext text/x-wiki oxl9624eo4dpizpg1uewil39zrcf9qr 2314752 2314751 2025-06-18T17:09:21Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host rdb2011.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314752 wikitext text/x-wiki hx7l1v24hug436o4am46k2vi1858vo9 2314753 2314752 2025-06-18T17:09:41Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host rdb2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314753 wikitext text/x-wiki pw24rfg3iwyzkx27mv0n25iq7dc19e1 2314754 2314753 2025-06-18T17:09:44Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2314754 wikitext text/x-wiki b8e4qcanj7uihw2ljnvn0vuk5l0gmwq 2314755 2314754 2025-06-18T17:11:40Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2314755 wikitext text/x-wiki k7goiyvqphrhv6so8nlj1hgo6y4p8yq 2314756 2314755 2025-06-18T17:15:52Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply 2314756 wikitext text/x-wiki 169f0ba7tcaboqgr185f3esbnmecw0s 2314757 2314756 2025-06-18T17:16:12Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply 2314757 wikitext text/x-wiki sroa74xwid72dlex4bf154wx9ig5fez 2314758 2314757 2025-06-18T17:16:19Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P78376 and previous config saved to /var/cache/conftool/dbconfig/20250618-171617-marostegui.json 2314758 wikitext text/x-wiki qzbgrq2t92r7otqxgb6e13b04p7msvj 2314759 2314758 2025-06-18T17:17:09Z Stashbot 7414 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply 2314759 wikitext text/x-wiki cprl586vt89yo1hjf66r0lfvtuzril2 2314760 2314759 2025-06-18T17:17:41Z Stashbot 7414 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply 2314760 wikitext text/x-wiki hu6ekahrm0pe9dg5eom6ms7j1hexg87 2314761 2314760 2025-06-18T17:22:10Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2314761 wikitext text/x-wiki leawcu3jfcr3endrmxhog8aztx58v6r 2314762 2314761 2025-06-18T17:27:54Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply 2314762 wikitext text/x-wiki f91y39zdfll9e09n6z9h93ac5kqh411 2314763 2314762 2025-06-18T17:28:22Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply 2314763 wikitext text/x-wiki 35t7jzqis4xvxgfcrvmg8kp2zo7486a 2314764 2314763 2025-06-18T17:28:25Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply 2314764 wikitext text/x-wiki iydk4f8ye0a0p3ihlmxbg4dzkurdgbx 2314765 2314764 2025-06-18T17:28:33Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-eqiad and A:cp - 9.2.10 upgrade (T390912) 2314765 wikitext text/x-wiki b1tkbwm5x7u5kfqoqf14uy2n0jyoukt 2314766 2314765 2025-06-18T17:28:36Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply 2314766 wikitext text/x-wiki m2yf3t8awy613dfkvjoj1yzrdxz6r20 2314767 2314766 2025-06-18T17:28:39Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply 2314767 wikitext text/x-wiki k7tz6iq4mht1msxadtqku53ow1uhnaf 2314768 2314767 2025-06-18T17:28:47Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply 2314768 wikitext text/x-wiki 93fulv9wkf5q28qk6ip9s2bx7d82tcn 2314769 2314768 2025-06-18T17:28:50Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply 2314769 wikitext text/x-wiki ak8gj5ojavp86azwt23tsj9xop3ytlx 2314770 2314769 2025-06-18T17:28:56Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2314770 wikitext text/x-wiki kma0gmoftv7j4nbp8meod7j4t34k3qt 2314771 2314770 2025-06-18T17:28:59Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply 2314771 wikitext text/x-wiki 4wunou1z0dhaiudsa6d0wnvcbvyw8wk 2314772 2314771 2025-06-18T17:29:15Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply 2314772 wikitext text/x-wiki 9xg4575u5iscgalv00xsu3saxgogq2e 2314773 2314772 2025-06-18T17:29:18Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply 2314773 wikitext text/x-wiki l877041n2hs440uwh3tz9ktdiidvgxj 2314774 2314773 2025-06-18T17:29:38Z Stashbot 7414 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply 2314774 wikitext text/x-wiki hk9uil8tnj9300okx0xe2c5g598zqca 2314775 2314774 2025-06-18T17:31:26Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P78377 and previous config saved to /var/cache/conftool/dbconfig/20250618-173124-marostegui.json 2314775 wikitext text/x-wiki kp14p0drktzebkdo1ptr8esdilwy7p0 2314776 2314775 2025-06-18T17:32:42Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply 2314776 wikitext text/x-wiki ef47mr60ujs1jx6y8uuq9llt68e068o 2314777 2314776 2025-06-18T17:33:28Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply 2314777 wikitext text/x-wiki ksk69u97wt69jvxgl18r2t4wgpspw83 2314778 2314777 2025-06-18T17:34:00Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply 2314778 wikitext text/x-wiki 3ubd7amhd6lvctp4fsyi37jww0wcyjt 2314779 2314778 2025-06-18T17:34:19Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply 2314779 wikitext text/x-wiki 65mrcolkawai1pqhey1z387wg5g3tep 2314781 2314779 2025-06-18T17:34:50Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply 2314781 wikitext text/x-wiki 0wvezfj6ie6ifrbuwxkx96jdiv1dci3 2314782 2314781 2025-06-18T17:35:06Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply 2314782 wikitext text/x-wiki duaoaq319n5xo6x3uf1rrwruvw5j7me 2314783 2314782 2025-06-18T17:35:36Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7001.* 2314783 wikitext text/x-wiki 7k268ds4xk0c26dw9otfipid5cki0vo 2314784 2314783 2025-06-18T17:35:39Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply 2314784 wikitext text/x-wiki 7o3wb8bs1d90gll6a9vrg82q72eez65 2314785 2314784 2025-06-18T17:35:42Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2314785 wikitext text/x-wiki 3tn02c4o9ei9ylp22siicit23t6mj7h 2314786 2314785 2025-06-18T17:36:11Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply 2314786 wikitext text/x-wiki 343y6mhn5wd5fvc5c9bln0697ygjyvm 2314787 2314786 2025-06-18T17:36:35Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply 2314787 wikitext text/x-wiki os35iaeewuwp5lr4tj1zgrz0bj4x3so 2314788 2314787 2025-06-18T17:37:07Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply 2314788 wikitext text/x-wiki t9k98i1kto6c5cer1gvtb3vyzlof9ve 2314790 2314788 2025-06-18T17:37:33Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply 2314790 wikitext text/x-wiki t5am0iaxi3u6in2fu9ximcic7dumc5d 2314791 2314790 2025-06-18T17:39:33Z Stashbot 7414 swfrench-wmf: migrated all shellbox instances to bookworm-based httpd images in codfw - T378128 2314791 wikitext text/x-wiki cmgcre6kfjwdvy6arjmcnvh3g8ywd2f 2314792 2314791 2025-06-18T17:39:56Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host rdb2011.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314792 wikitext text/x-wiki 0uddzntqc66axtorkjq97f1m8a1pkl9 2314793 2314792 2025-06-18T17:40:27Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host rdb2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314793 wikitext text/x-wiki r9wqrout2wxewnuaaca47hzbyogcz5u 2314794 2314793 2025-06-18T17:43:36Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host rdb2011.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314794 wikitext text/x-wiki 81v4wc0ox5oeutqzuxrhdyhykjtyth4 2314795 2314794 2025-06-18T17:43:50Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host rdb2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314795 wikitext text/x-wiki ito1aotu9ffv8txhpyxhtxlhgnwm6cr 2314796 2314795 2025-06-18T17:46:35Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T396130)', diff saved to https://phabricator.wikimedia.org/P78378 and previous config saved to /var/cache/conftool/dbconfig/20250618-174632-marostegui.json 2314796 wikitext text/x-wiki 7dfgek9mfnesck9mx5fyk1tq9cpyf8k 2314797 2314796 2025-06-18T17:46:49Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1239.eqiad.wmnet with reason: Maintenance 2314797 wikitext text/x-wiki ni554aadb5dt5ew3nbedyb3cta080zp 2314798 2314797 2025-06-18T17:48:59Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host rdb2011.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314798 wikitext text/x-wiki r2per62ejadiy4nyrfmxzph9z14otv9 2314799 2314798 2025-06-18T17:49:17Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host rdb2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314799 wikitext text/x-wiki 1gdw9mwfgoz0l1pxx2c1isz8pohc6vh 2314800 2314799 2025-06-18T17:49:53Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host rdb2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314800 wikitext text/x-wiki 7n3fd00k4clmmeaqrwgbkncc0o5ws74 2314801 2314800 2025-06-18T17:50:08Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host rdb2011.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314801 wikitext text/x-wiki 0qoypigm8jaz96hod62hve23qwf8lux 2314802 2314801 2025-06-18T17:50:22Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host rdb2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314802 wikitext text/x-wiki niejtetnbp25jtpm9qzy0sajpwsngo7 2314803 2314802 2025-06-18T17:50:30Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host rdb2011.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314803 wikitext text/x-wiki sw3rrvkndib8jfgqsdjwpllf3aw6lb0 2314805 2314803 2025-06-18T17:52:10Z Stashbot 7414 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool db2155 for queries (T385167)', diff saved to https://phabricator.wikimedia.org/P78379 and previous config saved to /var/cache/conftool/dbconfig/20250618-175206-ladsgroup.json 2314805 wikitext text/x-wiki kacanvxp1dwko2krpn7fbbk7tp1ekn3 2314806 2314805 2025-06-18T17:53:29Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply 2314806 wikitext text/x-wiki f0ucj0tqiock70v50ebjecaiis0znyi 2314808 2314806 2025-06-18T17:54:14Z Stashbot 7414 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Running queries (T385167) 2314808 wikitext text/x-wiki 23vempcgvs0x1mz0rtkyow7ehs6tk6c 2314809 2314808 2025-06-18T17:54:16Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply 2314809 wikitext text/x-wiki nbkc7j2fcsyuneihrt3sebtewwda1er 2314810 2314809 2025-06-18T17:54:45Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply 2314810 wikitext text/x-wiki fgcn8uyy6fusxor3q3509q7vuy8ftb0 2314811 2314810 2025-06-18T17:55:00Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply 2314811 wikitext text/x-wiki ctfddm3neiiqdyq1h0fa84kag7irklb 2314812 2314811 2025-06-18T17:55:32Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply 2314812 wikitext text/x-wiki 5rqj4oky4889oruyyzx7djidcuv7ceo 2314813 2314812 2025-06-18T17:55:47Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply 2314813 wikitext text/x-wiki 56aminhraalejnsokhustt8br1k5dtb 2314814 2314813 2025-06-18T17:56:18Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply 2314814 wikitext text/x-wiki 5xz6dc8u1l1an5kmzomcud4s3refjrk 2314815 2314814 2025-06-18T17:56:21Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply 2314815 wikitext text/x-wiki i3s3agd0sbr2pbnnbthxqylbdtdefc9 2314816 2314815 2025-06-18T17:56:51Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply 2314816 wikitext text/x-wiki oi649670l6nwj56idm3s07hu20t0k0f 2314817 2314816 2025-06-18T17:57:14Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply 2314817 wikitext text/x-wiki 73ca4z1wuyg2czd6fdo5jpavcpzzaff 2314818 2314817 2025-06-18T17:57:45Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply 2314818 wikitext text/x-wiki 39vfjpnd5arlwsowkibn048rlkbh8h0 2314819 2314818 2025-06-18T17:58:16Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply 2314819 wikitext text/x-wiki 81vt04yubyw0zga64r4py36175o3nvk 2314820 2314819 2025-06-18T17:58:56Z Stashbot 7414 swfrench-wmf: migrated all shellbox instances to bookworm-based httpd images in eqiad - T378128 2314820 wikitext text/x-wiki m53rys8esr62tx7kifd2tar43wap0hs 2314821 2314820 2025-06-18T17:59:41Z Stashbot 7414 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet 2314821 wikitext text/x-wiki rbugdvb2rfz5q6ai5n03przawups36l 2314822 2314821 2025-06-18T18:03:23Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host aux-k8s-worker2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314822 wikitext text/x-wiki rqqjdsblvfkmgz4by3c9i1xzhx0dfuo 2314823 2314822 2025-06-18T18:03:38Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host aux-k8s-worker2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314823 wikitext text/x-wiki 5h2hrs2eq4xw2118p2jg1250ppmerbn 2314824 2314823 2025-06-18T18:03:55Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host aux-k8s-worker2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314824 wikitext text/x-wiki i3rqor3cfqgylc2wh9cq7quj1zwpojs 2314825 2314824 2025-06-18T18:04:10Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host aux-k8s-worker2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314825 wikitext text/x-wiki cbmcd82mos2elgbuxjpka01jwg68h73 2314826 2314825 2025-06-18T18:05:13Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1240.eqiad.wmnet with reason: Maintenance 2314826 wikitext text/x-wiki 6q2qg82gqygb7dxxr95q8ijr2n0x5g8 2314829 2314826 2025-06-18T18:13:01Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Deploy arclamp 2314829 wikitext text/x-wiki 6i7gq47vr8i5050cuq5tefcvpk0xlfp 2314830 2314829 2025-06-18T18:13:23Z Stashbot 7414 ladsgroup@deploy1003: sync-world aborted: Deploy arclamp (duration: 00m 33s) 2314830 wikitext text/x-wiki npac7gm32stjs1uyy4wu6b5aewywrpz 2314831 2314830 2025-06-18T18:14:12Z Stashbot 7414 ladsgroup@deploy1003: Started deploy [performance/arc-lamp@76afb89]: Deploy arclamp 2314831 wikitext text/x-wiki 8s137i7fg8f6pdwnxy7yprgdc4dll60 2314832 2314831 2025-06-18T18:14:21Z Stashbot 7414 ladsgroup@deploy1003: Finished deploy [performance/arc-lamp@76afb89]: Deploy arclamp (duration: 00m 08s) 2314832 wikitext text/x-wiki inac4il3ltbjodhw7xzzbebs0dnkxtf 2314833 2314832 2025-06-18T18:15:26Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aux-k8s-worker2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314833 wikitext text/x-wiki m9yssjugirx9j47zxopdm9f4fknc9v5 2314834 2314833 2025-06-18T18:16:12Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aux-k8s-worker2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314834 wikitext text/x-wiki onjuv5lixih84dk8gp5ivymoyefg2dq 2314835 2314834 2025-06-18T18:16:57Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host aux-k8s-worker2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314835 wikitext text/x-wiki k5t9qpz3zb3hk8lvm95cbt77tkksq77 2314836 2314835 2025-06-18T18:17:08Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aux-k8s-worker2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314836 wikitext text/x-wiki a55rjwfjye9d6af4rf3xv8ahndbzcz0 2314837 2314836 2025-06-18T18:17:23Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host aux-k8s-worker2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314837 wikitext text/x-wiki hytbz139dw5x2gq5gguwf7ycrcntmo2 2314838 2314837 2025-06-18T18:17:52Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aux-k8s-worker2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314838 wikitext text/x-wiki nzcxr2z7ce0bipuwsa16gj6uizi2ixm 2314839 2314838 2025-06-18T18:17:55Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host aux-k8s-worker2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314839 wikitext text/x-wiki frj207n66mbk4tcbljt4l9kmferxpfj 2314840 2314839 2025-06-18T18:18:43Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host aux-k8s-worker2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314840 wikitext text/x-wiki pv9ki9omryzzqha1ziqghzacdcvwe59 2314842 2314840 2025-06-18T18:20:32Z Stashbot 7414 jgleeson: civicrm rolled back from 10eac2f8 to 63302c18 2314842 wikitext text/x-wiki 4agwdahvc255gnexfn96exvo95vm7br 2314843 2314842 2025-06-18T18:22:21Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aux-k8s-worker2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314843 wikitext text/x-wiki kcx74dwp1x4zdeeve9ng658siyl1rzi 2314844 2314843 2025-06-18T18:23:08Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1251.eqiad.wmnet with reason: Maintenance 2314844 wikitext text/x-wiki f2t5w4xx5j4jiyt405ljxaae5rlpqv6 2314845 2314844 2025-06-18T18:23:17Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1251 (T396130)', diff saved to https://phabricator.wikimedia.org/P78381 and previous config saved to /var/cache/conftool/dbconfig/20250618-182313-marostegui.json 2314845 wikitext text/x-wiki so9soksrjsbembgzt8psbkcquymcrx1 2314846 2314845 2025-06-18T18:23:20Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aux-k8s-worker2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314846 wikitext text/x-wiki q21fkpmnqecms4x1dyvrewbcwwrlodg 2314847 2314846 2025-06-18T18:24:06Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aux-k8s-worker2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314847 wikitext text/x-wiki 60xrmn3kgb2lx3hcmnxn2xyc2hwfdrz 2314848 2314847 2025-06-18T18:25:59Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host aux-k8s-worker2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314848 wikitext text/x-wiki a3fh60fb9dfd2gash5o1354bqk4hm16 2314849 2314848 2025-06-18T18:26:16Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host aux-k8s-worker2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314849 wikitext text/x-wiki 3x1vbjuadf2p3jb2h43vb7x69s802p0 2314850 2314849 2025-06-18T18:26:23Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aux-k8s-worker2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314850 wikitext text/x-wiki ni67l5646mc3y9pxxgaxy3jumbjpj7z 2314851 2314850 2025-06-18T18:26:36Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host aux-k8s-worker2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314851 wikitext text/x-wiki qk9osxlphzvx05ks4xnvqs2ulickr4r 2314852 2314851 2025-06-18T18:26:39Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aux-k8s-worker2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314852 wikitext text/x-wiki 41z1jz6n9v8z64b0re2hfm3e88ff6n1 2314853 2314852 2025-06-18T18:27:00Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aux-k8s-worker2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314853 wikitext text/x-wiki 63v5lnql28tef2pjlty74krhtt5fvrc 2314854 2314853 2025-06-18T18:27:44Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aux-k8s-worker2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314854 wikitext text/x-wiki 57j4cbp27qizofzvwrwoh96vr4grkyb 2314855 2314854 2025-06-18T18:27:56Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host aux-k8s-worker2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314855 wikitext text/x-wiki h53t7q82dbtujdwcbuehz03dzbs0vzq 2314856 2314855 2025-06-18T18:28:21Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aux-k8s-worker2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART 2314856 wikitext text/x-wiki 7c92zjyl6k7p2tyvcr5i06jykejhntw 2314857 2314856 2025-06-18T18:29:01Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1152853|etcd: Remove ES clusters from "write clusters" if section is RO (T395696)]] 2314857 wikitext text/x-wiki tvh43rtm2rm1yy0pmenmnv23ac80ku1 2314858 2314857 2025-06-18T18:31:16Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1152853|etcd: Remove ES clusters from "write clusters" if section is RO (T395696)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2314858 wikitext text/x-wiki 9p5bd6ltp0cbe1ypjndqp8gmz55kb0k 2314859 2314858 2025-06-18T18:38:40Z Stashbot 7414 cdobbins@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-eqsin and A:cp - 9.2.10 upgrade (T390912) 2314859 wikitext text/x-wiki d0j0m8c0oyyultfuoamtimkwj17u55u 2314861 2314859 2025-06-18T18:43:29Z Stashbot 7414 ladsgroup@cumin1002: dbctl commit (dc=all): 'Testing T395696', diff saved to https://phabricator.wikimedia.org/P78382 and previous config saved to /var/cache/conftool/dbconfig/20250618-184325-ladsgroup.json 2314861 wikitext text/x-wiki siecmgg0cio6pafvthn4qu5izg5o0qp 2314862 2314861 2025-06-18T18:45:41Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1251 (T396130)', diff saved to https://phabricator.wikimedia.org/P78383 and previous config saved to /var/cache/conftool/dbconfig/20250618-184538-marostegui.json 2314862 wikitext text/x-wiki 75qjbjdjm0jhhlqus9h7l9y7xsqf67x 2314863 2314862 2025-06-18T18:48:01Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-eqiad and A:cp - 9.2.10 upgrade (T390912) 2314863 wikitext text/x-wiki cbb7h9f59htryowuxoaa9oasv3grgs5 2314864 2314863 2025-06-18T18:49:07Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with sync 2314864 wikitext text/x-wiki inposcfzyrc9rxnyr4c11mkkxtr2nan 2314865 2314864 2025-06-18T18:55:56Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152853|etcd: Remove ES clusters from "write clusters" if section is RO (T395696)]] (duration: 26m 55s) 2314865 wikitext text/x-wiki gi7d5c8g39ikovw27ie48redwqrylf3 2314866 2314865 2025-06-18T18:56:20Z Stashbot 7414 ryankemper@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on 6 hosts with reason: T395772 hosts not serving production traffic 2314866 wikitext text/x-wiki lx7h87ubkbk5arhzncpzhs60fzefksm 2314867 2314866 2025-06-18T18:57:23Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1160990|etcd: Check for array key (T395696)]] 2314867 wikitext text/x-wiki 4lr2nvx9o2cyhhw18a2t45f24q5q9rq 2314869 2314867 2025-06-18T18:59:45Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1160990|etcd: Check for array key (T395696)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2314869 wikitext text/x-wiki 5ycuby9m3kz322bkcfmmwmo35ss7ql7 2314870 2314869 2025-06-18T19:00:32Z Stashbot 7414 wfan: payments-wiki upgraded from aa102260 to f56db8e6 2314870 wikitext text/x-wiki i8tm5pvqqpcox7umtf6eva0hq3uni6b 2314871 2314870 2025-06-18T19:00:47Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P78384 and previous config saved to /var/cache/conftool/dbconfig/20250618-190045-marostegui.json 2314871 wikitext text/x-wiki i2z6trsztoc04iuwrwsze9i2emx4rqs 2314873 2314871 2025-06-18T19:03:03Z Stashbot 7414 ladsgroup@deploy1003: ladsgroup: Continuing with sync 2314873 wikitext text/x-wiki iv18wnhf3bjdrej4ib4fpyjxo7mzx1t 2314875 2314873 2025-06-18T19:05:19Z Stashbot 7414 ChrisDobbins901_: cdobbins@cumin2002:~$ sudo -i cookbook sre.cdn.roll-upgrade-ats --query 'A:cp-codfw' --task-id T390912 --reason '9.2.10 upgrade' 2314875 wikitext text/x-wiki 197bl5nylkovaatovgoj3zlqdnvyopp 2314876 2314875 2025-06-18T19:05:25Z Stashbot 7414 cdobbins@cumin2002: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-codfw and A:cp - 9.2.10 upgrade (T390912) 2314876 wikitext text/x-wiki kfyunmbq5rec2gekze3yuc4698gp8zb 2314877 2314876 2025-06-18T19:07:56Z Stashbot 7414 ejegg: civicrm upgraded from 63302c18 to 670b3f6b 2314877 wikitext text/x-wiki odu420q5div2b0s8onxquu5r2457iaz 2314878 2314877 2025-06-18T19:10:02Z Stashbot 7414 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1160990|etcd: Check for array key (T395696)]] (duration: 12m 39s) 2314878 wikitext text/x-wiki h6m5zpgjor0r2ywgovb5r5k82evtfzl 2314879 2314878 2025-06-18T19:14:44Z Stashbot 7414 ladsgroup@cumin1002: dbctl commit (dc=all): 'Testing T395696', diff saved to https://phabricator.wikimedia.org/P78385 and previous config saved to /var/cache/conftool/dbconfig/20250618-191440-ladsgroup.json 2314879 wikitext text/x-wiki tj52n18zvi9wxwuzqff9z98ily6ta69 2314880 2314879 2025-06-18T19:15:55Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P78386 and previous config saved to /var/cache/conftool/dbconfig/20250618-191553-marostegui.json 2314880 wikitext text/x-wiki fw9zmrw3tu96bevdn039em42kkqwgfb 2314882 2314880 2025-06-18T19:30:44Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host aux-k8s-worker2006.codfw.wmnet with OS bookworm 2314882 wikitext text/x-wiki et3hy61vc6a9vr2l20iqzn1bdta9wyu 2314883 2314882 2025-06-18T19:31:02Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host aux-k8s-worker2007.codfw.wmnet with OS bookworm 2314883 wikitext text/x-wiki jjledfe8u7ige5imcg3zz3qdehtqbgm 2314884 2314883 2025-06-18T19:31:06Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1251 (T396130)', diff saved to https://phabricator.wikimedia.org/P78387 and previous config saved to /var/cache/conftool/dbconfig/20250618-193101-marostegui.json 2314884 wikitext text/x-wiki 21656i3lcv2gl8kljogu2t6uvlqwm9c 2314885 2314884 2025-06-18T19:31:16Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host aux-k8s-worker2008.codfw.wmnet with OS bookworm 2314885 wikitext text/x-wiki 8n39ls72i9sw1gqynzvtbwimz0n2xgc 2314886 2314885 2025-06-18T19:31:19Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance 2314886 wikitext text/x-wiki 6zlkua8cgrogg9gf55up2j91owqt352 2314887 2314886 2025-06-18T19:31:33Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host aux-k8s-worker2009.codfw.wmnet with OS bookworm 2314887 wikitext text/x-wiki 1znbmbjfa9y1ayeh2vs6fzt7njogja8 2314888 2314887 2025-06-18T19:32:47Z Stashbot 7414 ryankemper: T393966 Ran puppet on `titan1001` following merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1155335. Puppet looks happy and I see the new recording rules getting created 2314888 wikitext text/x-wiki jn02jo0o723tkpf3oxemejcnxdd79s5 2314889 2314888 2025-06-18T19:43:29Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-worker2006.codfw.wmnet with reason: host reimage 2314889 wikitext text/x-wiki 9tetwnems3brpd2ht9d23fwv6myxlx9 2314890 2314889 2025-06-18T19:43:43Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-worker2007.codfw.wmnet with reason: host reimage 2314890 wikitext text/x-wiki bw6uhchhf1cyfx7yzuys1ghxildk1rl 2314891 2314890 2025-06-18T19:44:01Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-worker2008.codfw.wmnet with reason: host reimage 2314891 wikitext text/x-wiki 8hoqdro8pv00vx2jndyteg7iwepacw3 2314892 2314891 2025-06-18T19:44:10Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-worker2009.codfw.wmnet with reason: host reimage 2314892 wikitext text/x-wiki huaz8acqiiuitejhjxh6p67uo1q1nr2 2314893 2314892 2025-06-18T19:47:17Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-worker2006.codfw.wmnet with reason: host reimage 2314893 wikitext text/x-wiki pd46mlcreindkam0rjazq1vg36kj4ef 2314894 2314893 2025-06-18T19:50:52Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-worker2009.codfw.wmnet with reason: host reimage 2314894 wikitext text/x-wiki er26q82yfuyxdx7lzmc91837qatjtr0 2314895 2314894 2025-06-18T19:54:43Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.179.0" for 2 host(s) 2314895 wikitext text/x-wiki puykthsd42arujqj4lm1j4jhmt72cmf 2314896 2314895 2025-06-18T19:54:53Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-worker2007.codfw.wmnet with reason: host reimage 2314896 wikitext text/x-wiki 26rbopf610trkl38vb64qtln0vr8s2l 2314897 2314896 2025-06-18T19:55:25Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on aux-k8s-worker2008.codfw.wmnet with reason: host reimage 2314897 wikitext text/x-wiki j9jhj9r87lpd2pz0fbr1dqit88s6hsa 2314898 2314897 2025-06-18T19:56:32Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.179.0" completed for 2 hosts 2314898 wikitext text/x-wiki 5lctwepnu3394rstyt8ej6q75mct3xb 2314902 2314898 2025-06-18T19:59:47Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.dns.netbox 2314902 wikitext text/x-wiki 3vyfzxo5g7rvhoz0wp2jpdhln3zb14k 2314903 2314902 2025-06-18T20:02:21Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2314903 wikitext text/x-wiki pne2q8g42re683xh12jq14irz3v0214 2314905 2314903 2025-06-18T20:02:54Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd2005-dev 2314905 wikitext text/x-wiki kh0ztace3almobuwdb2f5h55rubd8d6 2314906 2314905 2025-06-18T20:03:04Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd2005-dev 2314906 wikitext text/x-wiki sebs7b4bgyf62bi023u4k43y2repkq5 2314907 2314906 2025-06-18T20:03:09Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd2006-dev 2314907 wikitext text/x-wiki tl21eoy6uhl0fqsk28uq2an9l5xvguo 2314908 2314907 2025-06-18T20:03:12Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003" 2314908 wikitext text/x-wiki iu6oc9t1ug3g01v17o7dg2fs405ghj5 2314909 2314908 2025-06-18T20:03:19Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd2006-dev 2314909 wikitext text/x-wiki kfh19f63ozagn5xejxn3n4si7vmwrjs 2314910 2314909 2025-06-18T20:03:26Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd2007-dev 2314910 wikitext text/x-wiki q5mb7fjud13gi8jsjjuxeb3g0idgmbh 2314911 2314910 2025-06-18T20:03:36Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd2007-dev 2314911 wikitext text/x-wiki frs21p0z5fuqwerv2ucqjynprt9hr0q 2314912 2314911 2025-06-18T20:03:43Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003" 2314912 wikitext text/x-wiki f3f3d1fiiuu8nbimupei984cvnyeb8c 2314913 2314912 2025-06-18T20:03:47Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-worker2006.codfw.wmnet with OS bookworm 2314913 wikitext text/x-wiki njiuh3v5xl38sxo8m315990ojrk2mp4 2314914 2314913 2025-06-18T20:04:24Z Stashbot 7414 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1160858|Revert "Enable new mobile search experience everywhere (not including empty search recommendations)"]] 2314914 wikitext text/x-wiki a0t2javpxrvxcs0k9aiapqbyjmy0yxs 2314915 2314914 2025-06-18T20:06:26Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003" 2314915 wikitext text/x-wiki 5v84ixl14bguwbdqeagocwttosec1p8 2314916 2314915 2025-06-18T20:06:36Z Stashbot 7414 ebernhardson@deploy1003: ebernhardson, ksarabia: Backport for [[gerrit:1160858|Revert "Enable new mobile search experience everywhere (not including empty search recommendations)"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2314916 wikitext text/x-wiki d3c6yggxrmvvzlzszhz17zagbhevatp 2314917 2314916 2025-06-18T20:07:16Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003" 2314917 wikitext text/x-wiki n7kvuwslgrwlpsoaeojdv13t91ea50h 2314918 2314917 2025-06-18T20:07:19Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-worker2009.codfw.wmnet with OS bookworm 2314918 wikitext text/x-wiki dw737we65s7f9n2kctbbc0r2bsxkm7l 2314919 2314918 2025-06-18T20:08:15Z Stashbot 7414 ebernhardson@deploy1003: ebernhardson, ksarabia: Continuing with sync 2314919 wikitext text/x-wiki luhcve2ub58wgre9csud36tone454qj 2314920 2314919 2025-06-18T20:10:44Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003" 2314920 wikitext text/x-wiki t41e9p14nt1mgdruxct43uxe2uazxob 2314921 2314920 2025-06-18T20:11:06Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003" 2314921 wikitext text/x-wiki e949flsy0qxsf0t0j2lxuf9r6vaqtre 2314922 2314921 2025-06-18T20:11:10Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-worker2007.codfw.wmnet with OS bookworm 2314922 wikitext text/x-wiki t0tyi2ip9tk38jwgzswpzf8xs23s4cl 2314923 2314922 2025-06-18T20:12:16Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003" 2314923 wikitext text/x-wiki 84v2nre8tbciyaoyadvlnft1wr1r9je 2314924 2314923 2025-06-18T20:13:13Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003" 2314924 wikitext text/x-wiki 6kjwaik92o3xawuusca0r28w9klpx8q 2314925 2314924 2025-06-18T20:13:16Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-worker2008.codfw.wmnet with OS bookworm 2314925 wikitext text/x-wiki hb2qsnrf5quskqugrtxtjh97xk1vopb 2314926 2314925 2025-06-18T20:15:09Z Stashbot 7414 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1160858|Revert "Enable new mobile search experience everywhere (not including empty search recommendations)"]] (duration: 10m 45s) 2314926 wikitext text/x-wiki r7qw46btot996m6a5thyfzbjp1knphz 2314927 2314926 2025-06-18T20:18:32Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.179.1" for 2 host(s) 2314927 wikitext text/x-wiki 622t80ap8ve174b2lxfbosg8yrrshwu 2314928 2314927 2025-06-18T20:20:21Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.179.1" completed for 2 hosts 2314928 wikitext text/x-wiki 75ws3pxv8j38sfgmn7deqlldzlmlg7r 2314930 2314928 2025-06-18T20:26:51Z Stashbot 7414 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:838270|cirrus: Add services for read operations (T143553)]] 2314930 wikitext text/x-wiki s7sgvumijhpo0d9qk2pd1l3307y1ozq 2314932 2314930 2025-06-18T20:29:09Z Stashbot 7414 ebernhardson@deploy1003: ebernhardson: Backport for [[gerrit:838270|cirrus: Add services for read operations (T143553)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2314932 wikitext text/x-wiki bl1y7rgukx7trcllj4zssehs8mfo7hm 2314933 2314932 2025-06-18T20:31:15Z Stashbot 7414 cdobbins@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-codfw and A:cp - 9.2.10 upgrade (T390912) 2314933 wikitext text/x-wiki 3y0954jd5d2sznjqk18wrb0xtzh4lrd 2314934 2314933 2025-06-18T20:31:19Z Stashbot 7414 ebernhardson@deploy1003: ebernhardson: Continuing with sync 2314934 wikitext text/x-wiki 5lf2mm80hqn0iuxhx2r8ur3djft303f 2314936 2314934 2025-06-18T20:37:23Z Stashbot 7414 hashar: gerrit: deleted bunch of obsoletes references under `refs/users/*` accross all repositories. See T397317 (private) 2314936 wikitext text/x-wiki 4hw5ai3367fjat0y1zsi4t7q8cqbfcn 2314937 2314936 2025-06-18T20:38:02Z Stashbot 7414 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:838270|cirrus: Add services for read operations (T143553)]] (duration: 11m 11s) 2314937 wikitext text/x-wiki 7fcn50s5r7ysiozvvsj9lfzqociq23x 2314938 2314937 2025-06-18T20:39:52Z Stashbot 7414 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1153722|Set category collation to "uca-az" for Azerbaijani projects (T395896)]] 2314938 wikitext text/x-wiki qufxxwr51j8unbzjz6ocdn3tfgsvek4 2314940 2314938 2025-06-18T20:42:08Z Stashbot 7414 ebernhardson@deploy1003: nmw03, ebernhardson: Backport for [[gerrit:1153722|Set category collation to "uca-az" for Azerbaijani projects (T395896)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2314940 wikitext text/x-wiki nwrbnv8883p5och99t7yymg3vrjgswd 2314941 2314940 2025-06-18T20:43:47Z Stashbot 7414 volans@cumin2002: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: Release v0.10.1 - volans@cumin2002 2314941 wikitext text/x-wiki 6y7n49fresppxzthjsyljis3mfzyjs9 2314942 2314941 2025-06-18T20:44:07Z Stashbot 7414 ebernhardson@deploy1003: nmw03, ebernhardson: Continuing with sync 2314942 wikitext text/x-wiki 88kkz15xliu1etnvmospmt0n9fqw8g2 2314943 2314942 2025-06-18T20:44:36Z Stashbot 7414 volans@cumin2002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet with reason: Release v0.10.1 - volans@cumin2002 2314943 wikitext text/x-wiki 00ew66tkblmz54n1vzlh5al5ofk0cag 2314944 2314943 2025-06-18T20:44:55Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp7001.* 2314944 wikitext text/x-wiki ql58sott4rhq14j7ogxjwgmhh2egfdb 2314945 2314944 2025-06-18T20:45:34Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cloudcephosd2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314945 wikitext text/x-wiki eoigqecn0rshz3dputf9iuhmi5ss8l8 2314946 2314945 2025-06-18T20:46:01Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cloudcephosd2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314946 wikitext text/x-wiki fp3i8ail8rjjc63und7o5cs9j5rtckb 2314947 2314946 2025-06-18T20:46:34Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cloudcephosd2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314947 wikitext text/x-wiki ll4xqj3rs9h85j8kmc6i3lrvm6p4szk 2314948 2314947 2025-06-18T20:50:58Z Stashbot 7414 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153722|Set category collation to "uca-az" for Azerbaijani projects (T395896)]] (duration: 11m 06s) 2314948 wikitext text/x-wiki iqh3cnxc4pfl60z19x1u9f3vaqzw9a4 2314949 2314948 2025-06-18T20:52:17Z Stashbot 7414 ebernhardson: running updateCollation.php for azwikibooks, azwikiquote, azwikisource, and azwiktionary 2314949 wikitext text/x-wiki kjo7sqibectbtkq50ma6ia7jycsxoa2 2314950 2314949 2025-06-18T20:54:43Z Stashbot 7414 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:838271|Use discovery dns for elasticsearch read traffic (T143553)]] 2314950 wikitext text/x-wiki cdto4ng3x452g41a287g625qsomkwx5 2314951 2314950 2025-06-18T20:56:55Z Stashbot 7414 ebernhardson@deploy1003: ebernhardson: Backport for [[gerrit:838271|Use discovery dns for elasticsearch read traffic (T143553)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2314951 wikitext text/x-wiki kwdf7x7ghpjb3ka72ug5vzx1hmqkjee 2314952 2314951 2025-06-18T20:57:58Z Stashbot 7414 ebernhardson@deploy1003: ebernhardson: Continuing with sync 2314952 wikitext text/x-wiki ea1ydnyoox03a5us9r4wg5487ew5qta 2314953 2314952 2025-06-18T20:59:26Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314953 wikitext text/x-wiki t97ot6e2tok47bw65d8soa1t76lu6c5 2314954 2314953 2025-06-18T20:59:30Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314954 wikitext text/x-wiki cfy3he53bdjhcvgigfzxyjzgyzcaegi 2314955 2314954 2025-06-18T20:59:33Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314955 wikitext text/x-wiki klczhpq5i3iuceqirnwjihipqglgsvg 2314956 2314955 2025-06-18T21:00:01Z Stashbot 7414 ebernhardson: updateCollation.php for azwikibooks, azwikiquote, azwikisource, and azwiktionary completed 2314956 wikitext text/x-wiki 7gf4ek5w8jc71ort0w59v32t4q3hd86 2314957 2314956 2025-06-18T21:03:11Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.dns.netbox 2314957 wikitext text/x-wiki pjn0mrieenf7c1g0rlivot4hkm8mbun 2314958 2314957 2025-06-18T21:04:55Z Stashbot 7414 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:838271|Use discovery dns for elasticsearch read traffic (T143553)]] (duration: 10m 14s) 2314958 wikitext text/x-wiki 6eogw1g80y3dkh5lz5ionfqyaj8kdk8 2314959 2314958 2025-06-18T21:05:26Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.180.0" for 2 host(s) 2314959 wikitext text/x-wiki ltx1k59ow11a3urlwcqt5e1e9yzogky 2314960 2314959 2025-06-18T21:05:41Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2314960 wikitext text/x-wiki 5t7ef4ek252olwji4kwmo77jpnwx4a0 2314961 2314960 2025-06-18T21:06:21Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cloudcephosd2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314961 wikitext text/x-wiki 6khzxp1ymb38u21vla4y6jej3howzxp 2314962 2314961 2025-06-18T21:06:51Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cloudcephosd2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314962 wikitext text/x-wiki htrm1tnlwuvcdo4hi8uj3q01yucs9v3 2314963 2314962 2025-06-18T21:07:16Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.180.0" completed for 2 hosts 2314963 wikitext text/x-wiki a6dosubncrbnmpb1mz0k4hy35swomnv 2314964 2314963 2025-06-18T21:07:30Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cloudcephosd2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314964 wikitext text/x-wiki 39quccdz1tkmuz1b1o75t7jxkzzep34 2314966 2314964 2025-06-18T21:22:26Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314966 wikitext text/x-wiki s84oc6lwfajf2wet6y6bw1iknyyf93a 2314967 2314966 2025-06-18T21:22:32Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314967 wikitext text/x-wiki cdpopeevpv2ft9txwddspdjsdsx9a63 2314968 2314967 2025-06-18T21:22:44Z Stashbot 7414 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314968 wikitext text/x-wiki 7e1zmvubpd16j0uszx743t4i3nqxo6i 2314969 2314968 2025-06-18T21:23:56Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephosd2005.codfw.wmnet with OS bullseye 2314969 wikitext text/x-wiki k3ds07k4etbylcu43gvaj7tpefj0xku 2314970 2314969 2025-06-18T21:30:50Z Stashbot 7414 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply 2314970 wikitext text/x-wiki 0559dhivhb4f1xtz2l4mkib2cfak92o 2314971 2314970 2025-06-18T21:31:51Z Stashbot 7414 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply 2314971 wikitext text/x-wiki 6sm6dcmhpozdftxbb8bdr08778pdpwd 2314973 2314971 2025-06-18T21:36:36Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7001.* 2314973 wikitext text/x-wiki hq16ab5y6xsf534pyjt196q3mpmpzxc 2314974 2314973 2025-06-18T21:39:53Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp3072.* 2314974 wikitext text/x-wiki b18riszqjgoiff8bm8pq277uhw7j80c 2314975 2314974 2025-06-18T21:39:57Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp3072.* 2314975 wikitext text/x-wiki o2u3983uuj8s28qnjl919hj7vvwzxbn 2314976 2314975 2025-06-18T21:40:08Z Stashbot 7414 brett: Depooling cp3072 to upgrade bios 2314976 wikitext text/x-wiki luga2gjoo2t3upl2xqvfbtqua4epyyu 2314977 2314976 2025-06-18T21:41:45Z Stashbot 7414 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cp3072.esams.wmnet with reason: BIOS upgrades 2314977 wikitext text/x-wiki bj5je0lnlw074axwrlwijz9xuil0t2a 2314979 2314977 2025-06-18T21:59:36Z Stashbot 7414 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cloudcephosd2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314979 wikitext text/x-wiki qmrdbb1qxax395r3f2c2y21k11fxjxi 2314980 2314979 2025-06-18T22:01:08Z Stashbot 7414 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2314980 wikitext text/x-wiki le3ib057di52dlxxgn87tnl7nd2hamr 2314982 2314980 2025-06-18T22:04:58Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for cp3072.esams.wmnet 2314982 wikitext text/x-wiki ldmmx1ia34ug8tvbv7aikz4wt36f7iw 2314983 2314982 2025-06-18T22:05:01Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp3072.esams.wmnet 2314983 wikitext text/x-wiki jjxfbtenhskq5fkwvhstjxrdhkqwdfj 2314984 2314983 2025-06-18T22:05:04Z Stashbot 7414 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp3072.* 2314984 wikitext text/x-wiki 2zzrja4f9g713n5hm8731nnegjyyazz 2314986 2314984 2025-06-18T22:09:54Z Stashbot 7414 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-codfw and A:cp - Fix VSLbs() assert error and upgrade libvmod-wmfuniq to 0.2.0 (T396581) 2314986 wikitext text/x-wiki 98l3izyjgdq70kby3h2mcg4nrkw4aav 2314988 2314986 2025-06-18T22:12:31Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@f8d7b38]: no-op deploy to phab1005 2314988 wikitext text/x-wiki kk281nm6uqllb60xbv4yck8v5v6pvkh 2314989 2314988 2025-06-18T22:12:38Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@f8d7b38]: no-op deploy to phab1005 (duration: 00m 07s) 2314989 wikitext text/x-wiki mqy6adz0s7n3xhdnrr9sigpqbh415zk 2314990 2314989 2025-06-18T22:14:22Z Stashbot 7414 brennen@deploy1003: Started deploy [phabricator/deployment@6af4bb7]: merge-phorge-2024.35 deploy to phab1005 (T390034) 2314990 wikitext text/x-wiki n5jljdqror5kobt5q3xlg4hlhpipzzw 2314991 2314990 2025-06-18T22:14:49Z Stashbot 7414 brennen@deploy1003: Finished deploy [phabricator/deployment@6af4bb7]: merge-phorge-2024.35 deploy to phab1005 (T390034) (duration: 00m 26s) 2314991 wikitext text/x-wiki m0rwayiwyp9wzr3vmdfaspskg8r8bz3 2314992 2314991 2025-06-18T22:19:28Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply 2314992 wikitext text/x-wiki m01hpzz2q7f71l5us52z2ftz25y623u 2314993 2314992 2025-06-18T22:19:53Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply 2314993 wikitext text/x-wiki to859h905tunhabxaoqwzxybtypai4r 2314994 2314993 2025-06-18T22:44:07Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.dns.netbox 2314994 wikitext text/x-wiki baqnwjcuzk1e6yeps4tx3sfxa2wyf4e 2314995 2314994 2025-06-18T22:47:33Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added mgmt for aux-k8s-worker100[6-9] - jclark@cumin1002" 2314995 wikitext text/x-wiki q15o5jv11fte56qinbyc9d1pr9i2zix 2314996 2314995 2025-06-18T22:47:38Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added mgmt for aux-k8s-worker100[6-9] - jclark@cumin1002" 2314996 wikitext text/x-wiki oic0lgs3bwzmdxii0yxufql79mdq930 2314997 2314996 2025-06-18T22:47:41Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2314997 wikitext text/x-wiki jslzovfzia8piv9bpczss3d8h2f8xre 2314998 2314997 2025-06-18T22:52:55Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host aux-k8s-worker1006 2314998 wikitext text/x-wiki 5jodyoaqbvj5iv67sb4lqvim6h2u95j 2314999 2314998 2025-06-18T22:54:03Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host aux-k8s-worker1006 2314999 wikitext text/x-wiki mknue25xc9zbasno4kbepa99di8f75n 2315000 2314999 2025-06-18T22:54:07Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host aux-k8s-worker1007 2315000 wikitext text/x-wiki 0mwsj1gpdbnjnexokklu16h5oxxt3tu 2315001 2315000 2025-06-18T22:55:15Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host aux-k8s-worker1007 2315001 wikitext text/x-wiki jux88y4iqf5t3jcbn7m7bp0lwf7en21 2315002 2315001 2025-06-18T22:55:19Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host aux-k8s-worker1008 2315002 wikitext text/x-wiki 0lhex8p3wfs6agrulwlyqufstafvbr0 2315003 2315002 2025-06-18T22:56:37Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host aux-k8s-worker1008 2315003 wikitext text/x-wiki ls662qy57xrzqdub3vwlf7gfs1or4vd 2315004 2315003 2025-06-18T22:56:43Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host aux-k8s-worker1009 2315004 wikitext text/x-wiki pk4p4yz831im74qnsw4nt7xrpajwjo3 2315005 2315004 2025-06-18T22:58:01Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host aux-k8s-worker1009 2315005 wikitext text/x-wiki jnouos5io5yhtk90xfx96tg1gfg89l4 2315008 2315005 2025-06-19T00:21:20Z Stashbot 7414 ejegg: civicrm upgraded from 670b3f6b to c6225a10 2315008 wikitext text/x-wiki 41wje234amp2h9a3fbj41e2dzf5qi4i 2315009 2315008 2025-06-19T00:39:17Z Stashbot 7414 eileen: config revision changed from 3521b9fe to a5546244 2315009 wikitext text/x-wiki pviblqqa1zuklafidj9op66u97gqce8 2315012 2315009 2025-06-19T01:12:47Z Stashbot 7414 eileen: civicrm upgraded from c6225a10 to e22e242f 2315012 wikitext text/x-wiki f3w19g64hxc7kv9p0ep7cvps0k0y2tu 2315018 2315012 2025-06-19T01:46:46Z Stashbot 7414 eileen: civicrm upgraded from e22e242f to 3288293e 2315018 wikitext text/x-wiki f55vx59piga76q7g7li02nban6p1uxq 2315019 2315018 2025-06-19T01:55:47Z Stashbot 7414 ejegg: re-enabled recurring donation charge job 2315019 wikitext text/x-wiki ay6coxxktgsi82vsjppayzmoglyftcc 2315023 2315019 2025-06-19T05:05:25Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1163.eqiad.wmnet with reason: Maintenance 2315023 wikitext text/x-wiki 4m9rtrc98kf0bnzw1hizcwyexxrkybd 2315024 2315023 2025-06-19T05:07:29Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depool pc2 T378715', diff saved to https://phabricator.wikimedia.org/P78392 and previous config saved to /var/cache/conftool/dbconfig/20250619-050725-root.json 2315024 wikitext text/x-wiki 9d6ujsuvtgnyxey25d5oaplucszvu1m 2315025 2315024 2025-06-19T05:08:46Z Stashbot 7414 marostegui@cumin1003: DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1 day, 0:00:00 on pc1012.eqiad.wmnet with reason: Maintenance 2315025 wikitext text/x-wiki fvfxxyo74s9lc23f6ba91z3v452408h 2315027 2315025 2025-06-19T05:09:04Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2012.codfw.wmnet,pc1012.eqiad.wmnet with reason: Maintenance 2315027 wikitext text/x-wiki 2xiaqagp1qr3rp8fx710n57b9lknf4j 2315038 2315027 2025-06-19T05:34:28Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2150.codfw.wmnet with reason: Maintenance 2315038 wikitext text/x-wiki sggl2105wzh61br2ic5v3zi1h0zj3mt 2315039 2315038 2025-06-19T05:34:37Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2150 (T396130)', diff saved to https://phabricator.wikimedia.org/P78393 and previous config saved to /var/cache/conftool/dbconfig/20250619-053433-marostegui.json 2315039 wikitext text/x-wiki blbpg5mlu6iszdnxqk6bidxmsecih6f 2315041 2315039 2025-06-19T05:38:03Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2186.codfw.wmnet with reason: Maintenance 2315041 wikitext text/x-wiki 9oc7d04elzfsskqii0ps2o3ok7qpgo9 2315042 2315041 2025-06-19T05:38:28Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2186', diff saved to https://phabricator.wikimedia.org/P78394 and previous config saved to /var/cache/conftool/dbconfig/20250619-053826-root.json 2315042 wikitext text/x-wiki 9032tgvrdxorzp4n3fv25b50er8p81l 2315043 2315042 2025-06-19T05:44:21Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db2186 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P78395 and previous config saved to /var/cache/conftool/dbconfig/20250619-054418-root.json 2315043 wikitext text/x-wiki phhbhk5etanq2zwfioeukryca0bx8ow 2315044 2315043 2025-06-19T05:55:26Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T396130)', diff saved to https://phabricator.wikimedia.org/P78396 and previous config saved to /var/cache/conftool/dbconfig/20250619-055522-marostegui.json 2315044 wikitext text/x-wiki 1zo0zgy8hnfz3lbxdtm5nr3bn5l14xl 2315045 2315044 2025-06-19T05:59:26Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db2186 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P78397 and previous config saved to /var/cache/conftool/dbconfig/20250619-055924-root.json 2315045 wikitext text/x-wiki hdmygrke2261zmbtm271phmzcyqx26y 2315046 2315045 2025-06-19T06:10:32Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P78398 and previous config saved to /var/cache/conftool/dbconfig/20250619-061030-marostegui.json 2315046 wikitext text/x-wiki f5x3yk6stu7kxp1ayqowayeod7cf709 2315047 2315046 2025-06-19T06:14:32Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db2186 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P78399 and previous config saved to /var/cache/conftool/dbconfig/20250619-061430-root.json 2315047 wikitext text/x-wiki 7b2vmvrj7pla57x2c47yhxkmli0te4w 2315048 2315047 2025-06-19T06:25:39Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P78400 and previous config saved to /var/cache/conftool/dbconfig/20250619-062537-marostegui.json 2315048 wikitext text/x-wiki ow05upo84vgr07l1jim28snzu5mwjrr 2315049 2315048 2025-06-19T06:29:38Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db2186 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P78401 and previous config saved to /var/cache/conftool/dbconfig/20250619-062936-root.json 2315049 wikitext text/x-wiki bx2u5vpr2ujh6bfs0p2werwjx3caj77 2315050 2315049 2025-06-19T06:38:00Z Stashbot 7414 stevemunene@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on an-worker[1149-1153].eqiad.wmnet with reason: Upgrade an-worker hard drives from 4TB to 8TB group 10 2315050 wikitext text/x-wiki h1rf2a61co4yd9jdvg8h76apyxgh6a4 2315051 2315050 2025-06-19T06:38:25Z Stashbot 7414 stevemunene@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on an-worker1175.eqiad.wmnet with reason: Upgrade an-worker hard drives from 4TB to 8TB group 10 2315051 wikitext text/x-wiki koojg5zv5ohbmu8absjdj3xi4so4scu 2315052 2315051 2025-06-19T06:39:17Z Stashbot 7414 stevemunene@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on an-worker1154.eqiad.wmnet with reason: Upgrade an-worker hard drives from 4TB to 8TB group 9 2315052 wikitext text/x-wiki eguv1xpvs97bd3avt786t1lex1fnjml 2315053 2315052 2025-06-19T06:39:44Z Stashbot 7414 stevemunene@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on an-worker1176.eqiad.wmnet with reason: Upgrade an-worker hard drives from 4TB to 8TB group 9 2315053 wikitext text/x-wiki 4f0e49ota3vvu3uz8ji4j22zeybn9cz 2315054 2315053 2025-06-19T06:40:48Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T396130)', diff saved to https://phabricator.wikimedia.org/P78402 and previous config saved to /var/cache/conftool/dbconfig/20250619-064045-marostegui.json 2315054 wikitext text/x-wiki pr4rdw0bc5bernj9dld83gygtppkx7f 2315055 2315054 2025-06-19T06:41:02Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2159.codfw.wmnet with reason: Maintenance 2315055 wikitext text/x-wiki mzxqlnauz8msjr32b5qh9ba7v2biel3 2315056 2315055 2025-06-19T06:41:11Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2159 (T396130)', diff saved to https://phabricator.wikimedia.org/P78403 and previous config saved to /var/cache/conftool/dbconfig/20250619-064108-marostegui.json 2315056 wikitext text/x-wiki 8gp6pv2crs3vyid0hr3l1op1lyvuyva 2315057 2315056 2025-06-19T07:01:50Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T396130)', diff saved to https://phabricator.wikimedia.org/P78404 and previous config saved to /var/cache/conftool/dbconfig/20250619-070146-marostegui.json 2315057 wikitext text/x-wiki 83u3b582q69lreeyqxjxqpkmoeauigs 2315058 2315057 2025-06-19T07:04:06Z Stashbot 7414 gkyziridis@deploy1003: Started scap sync-world: Backport for [[gerrit:1160797|ores-extension: enable extension with revertrisk filter for azwiki (T395824)]] 2315058 wikitext text/x-wiki 67x1mk29zc198muf92i5yfxyemisad2 2315059 2315058 2025-06-19T07:04:52Z Stashbot 7414 moritzm: installing edk2 security updates 2315059 wikitext text/x-wiki mnrj8ld3w3z31037623w0l0lw5lg6eu 2315060 2315059 2025-06-19T07:06:30Z Stashbot 7414 gkyziridis@deploy1003: gkyziridis: Backport for [[gerrit:1160797|ores-extension: enable extension with revertrisk filter for azwiki (T395824)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2315060 wikitext text/x-wiki i322lm7o4fb9qxbmibbdsnvdoic1w7k 2315061 2315060 2025-06-19T07:09:00Z Stashbot 7414 gkyziridis@deploy1003: gkyziridis: Continuing with sync 2315061 wikitext text/x-wiki 07vv7g0jjakgdce5xi6hiq1wofaryno 2315062 2315061 2025-06-19T07:15:57Z Stashbot 7414 gkyziridis@deploy1003: Finished scap sync-world: Backport for [[gerrit:1160797|ores-extension: enable extension with revertrisk filter for azwiki (T395824)]] (duration: 11m 50s) 2315062 wikitext text/x-wiki 9nh676tpf447d9qsc76p7oktj36vh2w 2315063 2315062 2025-06-19T07:16:56Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P78405 and previous config saved to /var/cache/conftool/dbconfig/20250619-071654-marostegui.json 2315063 wikitext text/x-wiki p15leab1ylr5utam52dh3zd6ferf20x 2315064 2315063 2025-06-19T07:18:11Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2047.codfw.wmnet 2315064 wikitext text/x-wiki po4ouzmy869s1egzoghkygkmcx82x0n 2315065 2315064 2025-06-19T07:20:13Z Stashbot 7414 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1161182|Enable the Contribute menu in Egyptian Arabic, Igbo, and Uzbek]] 2315065 wikitext text/x-wiki k4ccw98uwqqlmc742ualgvns7odxxk8 2315066 2315065 2025-06-19T07:22:29Z Stashbot 7414 kartik@deploy1003: kartik: Backport for [[gerrit:1161182|Enable the Contribute menu in Egyptian Arabic, Igbo, and Uzbek]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2315066 wikitext text/x-wiki fn1vnoq7sa66648jf557j5cjgipdp84 2315067 2315066 2025-06-19T07:24:23Z Stashbot 7414 kartik@deploy1003: kartik: Continuing with sync 2315067 wikitext text/x-wiki 5clh5er9hpmxhiekw47dmz74t7v5iw7 2315068 2315067 2025-06-19T07:25:03Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2233].codfw.wmnet,db[1164,1217,1250].eqiad.wmnet with reason: Primary switchover m2 T397182 2315068 wikitext text/x-wiki 5pve48jgas5tx86nonoib8ucs7dzzr9 2315069 2315068 2025-06-19T07:25:22Z Stashbot 7414 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - T397300 2315069 wikitext text/x-wiki 5dv327ufs375abau08524bcqrmt3t8z 2315070 2315069 2025-06-19T07:25:32Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2047.codfw.wmnet 2315070 wikitext text/x-wiki e5abzdnkptvqh7hv65mkje1rjowmnp5 2315071 2315070 2025-06-19T07:26:56Z Stashbot 7414 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - T397300 2315071 wikitext text/x-wiki sznrg96bcn3g5otssc1v629n7p9gzq2 2315072 2315071 2025-06-19T07:29:16Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2048.codfw.wmnet 2315072 wikitext text/x-wiki dghsg67l5onmwyog9n5o15qdcsc2zew 2315073 2315072 2025-06-19T07:31:13Z Stashbot 7414 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1161182|Enable the Contribute menu in Egyptian Arabic, Igbo, and Uzbek]] (duration: 10m 59s) 2315073 wikitext text/x-wiki ft7rie3nb8nv7lapkfx2eggmqsxeii5 2315074 2315073 2025-06-19T07:32:03Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P78407 and previous config saved to /var/cache/conftool/dbconfig/20250619-073201-marostegui.json 2315074 wikitext text/x-wiki rooowdcdggya31m3gkpw7bh8p1f0r4u 2315075 2315074 2025-06-19T07:33:54Z Stashbot 7414 marostegui: Failover m2 from db1164 to db1250 - T397182 2315075 wikitext text/x-wiki khjqgvmqvzuqvmwe8p133kjvj5venp6 2315076 2315075 2025-06-19T07:36:36Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2048.codfw.wmnet 2315076 wikitext text/x-wiki 9pxbyip3zbcup6xsqr4pgtzer783rdr 2315078 2315076 2025-06-19T07:37:47Z Stashbot 7414 jynus: just started es read only backup regeneration T387892 2315078 wikitext text/x-wiki js82nxtjm0ypms3qpegsjqpylrwpjnq 2315079 2315078 2025-06-19T07:39:18Z Stashbot 7414 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2047.codfw.wmnet to cluster codfw and group B 2315079 wikitext text/x-wiki pwwhv38w825j3osou8we3zdkqj7w2db 2315080 2315079 2025-06-19T07:41:35Z Stashbot 7414 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2047.codfw.wmnet to cluster codfw and group B 2315080 wikitext text/x-wiki skkow9yprk242phiyfrjvya58z0hss0 2315081 2315080 2025-06-19T07:41:41Z Stashbot 7414 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti2048.codfw.wmnet to cluster codfw and group B 2315081 wikitext text/x-wiki 8juk3fluvfnkcooote52s92wmno5yqw 2315082 2315081 2025-06-19T07:41:54Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[1164,1217].eqiad.wmnet with reason: Maintenance 2315082 wikitext text/x-wiki ntunevh3wliny194ii9fwzry9l5ujsd 2315083 2315082 2025-06-19T07:47:11Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T396130)', diff saved to https://phabricator.wikimedia.org/P78409 and previous config saved to /var/cache/conftool/dbconfig/20250619-074708-marostegui.json 2315083 wikitext text/x-wiki 9ctad2bdfpp0ily9m81c27p3aqvrct5 2315084 2315083 2025-06-19T07:47:25Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2168.codfw.wmnet with reason: Maintenance 2315084 wikitext text/x-wiki cl47790y7a5nvrdsy9ok5hkrm8ebgw9 2315085 2315084 2025-06-19T07:47:34Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2168 (T396130)', diff saved to https://phabricator.wikimedia.org/P78410 and previous config saved to /var/cache/conftool/dbconfig/20250619-074731-marostegui.json 2315085 wikitext text/x-wiki 2keqtkxofjtpziarwicv9c7bz8156mt 2315086 2315085 2025-06-19T07:50:31Z Stashbot 7414 moritzm: installing glib2.0 security updates 2315086 wikitext text/x-wiki qbqywrmdqbjura3t5blsiw9nk1mrc9h 2315087 2315086 2025-06-19T07:55:50Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1179', diff saved to https://phabricator.wikimedia.org/P78411 and previous config saved to /var/cache/conftool/dbconfig/20250619-075548-root.json 2315087 wikitext text/x-wiki bnxthunra4ct296d5g17pdgvbralsph 2315088 2315087 2025-06-19T07:56:20Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1179.eqiad.wmnet with reason: Maintenance 2315088 wikitext text/x-wiki t3a4epn3q7s0dgua6j22pei51cngp7l 2315105 2315088 2025-06-19T08:07:22Z Stashbot 7414 moritzm: installing python-tornado security updates 2315105 wikitext text/x-wiki ji6a6l5cfn0gfjdd2tbrpjsn5sqzoe3 2315106 2315105 2025-06-19T08:08:15Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168 (T396130)', diff saved to https://phabricator.wikimedia.org/P78412 and previous config saved to /var/cache/conftool/dbconfig/20250619-080812-marostegui.json 2315106 wikitext text/x-wiki fj5p5sz1o0aej3el11hbslko5inoy5l 2315107 2315106 2025-06-19T08:08:23Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db1179 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P78413 and previous config saved to /var/cache/conftool/dbconfig/20250619-080820-root.json 2315107 wikitext text/x-wiki 6hg9fxzr2vbjc09t2b9b70yp0k9sqlj 2315108 2315107 2025-06-19T08:10:24Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.hosts.decommission for hosts thanos-be[1001-1004].eqiad.wmnet 2315108 wikitext text/x-wiki 3trwni166xu8ve1jnjs584guwi3qbd8 2315109 2315108 2025-06-19T08:12:01Z Stashbot 7414 hashar@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.6 refs T392176 2315109 wikitext text/x-wiki gp5erx1k1xnpd05lrv8dobghz4iln1q 2315116 2315109 2025-06-19T08:17:09Z Stashbot 7414 Ammar: Ran fixStuckGlobalRename.php for T397384 T397219 T397218 2315116 wikitext text/x-wiki nroy36v4heom9cekrzlgaqh77zhjuy5 2315117 2315116 2025-06-19T08:23:22Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P78414 and previous config saved to /var/cache/conftool/dbconfig/20250619-082320-marostegui.json 2315117 wikitext text/x-wiki 95rgpkby79dt7o3ugn6po107q0w3w6y 2315118 2315117 2025-06-19T08:23:29Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db1179 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P78415 and previous config saved to /var/cache/conftool/dbconfig/20250619-082326-root.json 2315118 wikitext text/x-wiki o6aeeyfx8mfibd34fyn2ma06vy8rn0e 2315119 2315118 2025-06-19T08:23:35Z Stashbot 7414 akosiaris@cumin1003: START - Cookbook sre.hosts.provision for host aux-k8s-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2315119 wikitext text/x-wiki bg3tpqsvueo2f456t6dy016jaxtnena 2315120 2315119 2025-06-19T08:25:09Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.dns.netbox 2315120 wikitext text/x-wiki otbiwkq2kfzjtzx1s5rrzii3eu59d2k 2315121 2315120 2025-06-19T08:28:39Z Stashbot 7414 akosiaris@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aux-k8s-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2315121 wikitext text/x-wiki 2e6dr7ezjxq5mz8d8aqurh3sc121ivd 2315122 2315121 2025-06-19T08:29:16Z Stashbot 7414 mvernon@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: thanos-be[1001-1004].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - mvernon@cumin1003" 2315122 wikitext text/x-wiki ds89zs8hm8td4sz4gosv8mo5mzmrtbo 2315123 2315122 2025-06-19T08:31:13Z Stashbot 7414 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2048.codfw.wmnet to cluster codfw and group B 2315123 wikitext text/x-wiki cl3mjphjod1paakb6vop5dcxl2tg7a0 2315124 2315123 2025-06-19T08:31:50Z Stashbot 7414 moritzm: installing modsecurity-apache security updates 2315124 wikitext text/x-wiki hpwuumhfpowr65ypxgpfi8lbnjntini 2315125 2315124 2025-06-19T08:33:04Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: thanos-be[1001-1004].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - mvernon@cumin1003" 2315125 wikitext text/x-wiki al0tox6kj7e9zd3yas00h96n7avs83z 2315126 2315125 2025-06-19T08:33:08Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2315126 wikitext text/x-wiki iczdhd3e7eyp082vlslrv8gkqap0m98 2315127 2315126 2025-06-19T08:33:11Z Stashbot 7414 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts thanos-be[1001-1004].eqiad.wmnet 2315127 wikitext text/x-wiki r0lxhs9vf7zfd1xscip407axwt7w01a 2315128 2315127 2025-06-19T08:38:29Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P78417 and previous config saved to /var/cache/conftool/dbconfig/20250619-083827-marostegui.json 2315128 wikitext text/x-wiki hbkfbqqu5zj1xofl0qxt0ggldc23buk 2315129 2315128 2025-06-19T08:38:34Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db1179 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P78418 and previous config saved to /var/cache/conftool/dbconfig/20250619-083832-root.json 2315129 wikitext text/x-wiki cyl1fipfmv7dwakrun9a1k2lb09oumv 2315130 2315129 2025-06-19T08:47:04Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2235.codfw.wmnet with reason: Maintenance 2315130 wikitext text/x-wiki dugst5gxjx27abiq1p6en6vq2haniaz 2315131 2315130 2025-06-19T08:50:43Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db[2160,2235].codfw.wmnet with reason: Maintenance 2315131 wikitext text/x-wiki thwkqcqf4bmhce3jq6ivok0ljdxhf30 2315132 2315131 2025-06-19T08:50:47Z Stashbot 7414 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2021.codfw.wmnet 2315132 wikitext text/x-wiki mspvsxn8yzorlnnnjij7gvnythxe3m2 2315133 2315132 2025-06-19T08:52:33Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.rename from kafka-stretch2001 to dse-k8s-worker2001 2315133 wikitext text/x-wiki 8gs46np9xro8dlkzclqn7793g0kgw86 2315134 2315133 2025-06-19T08:52:55Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.netbox 2315134 wikitext text/x-wiki 141bit6piqza7ei6la1rx3alj7fl6mz 2315135 2315134 2025-06-19T08:53:37Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168 (T396130)', diff saved to https://phabricator.wikimedia.org/P78419 and previous config saved to /var/cache/conftool/dbconfig/20250619-085334-marostegui.json 2315135 wikitext text/x-wiki 6ehvx3ufdvv70a0r8mgkcypjnycajjf 2315136 2315135 2025-06-19T08:53:46Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db1179 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P78420 and previous config saved to /var/cache/conftool/dbconfig/20250619-085344-root.json 2315136 wikitext text/x-wiki tuxywk09vqkhlc5j4szrwazshyg9alw 2315137 2315136 2025-06-19T08:53:52Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2182.codfw.wmnet with reason: Maintenance 2315137 wikitext text/x-wiki 6wv80kxghiqjknl6i5wxubi998nmuww 2315138 2315137 2025-06-19T08:54:00Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2182 (T396130)', diff saved to https://phabricator.wikimedia.org/P78421 and previous config saved to /var/cache/conftool/dbconfig/20250619-085357-marostegui.json 2315138 wikitext text/x-wiki j6fpgmrd4q7nrhwl2g375qdr5kakt22 2315139 2315138 2025-06-19T08:55:07Z Stashbot 7414 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2021.codfw.wmnet 2315139 wikitext text/x-wiki mf7t31jnt4yk31zqty1ufywo4eaj8xa 2315140 2315139 2025-06-19T08:56:07Z Stashbot 7414 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2021.codfw.wmnet 2315140 wikitext text/x-wiki alhf5016h0auxouw4dp9j47g48s7u8c 2315141 2315140 2025-06-19T08:56:42Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kafka-stretch2001 to dse-k8s-worker2001 - btullis@cumin1003" 2315141 wikitext text/x-wiki 50rtetj9h7ayadqxeo9y86i4mcz5h3t 2315142 2315141 2025-06-19T08:58:38Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kafka-stretch2001 to dse-k8s-worker2001 - btullis@cumin1003" 2315142 wikitext text/x-wiki hqqy3rggyqpj8fljae6hy6ppv32cmcp 2315143 2315142 2025-06-19T08:58:41Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2315143 wikitext text/x-wiki enhnele4h659dllahc7k6gonapfko6d 2315144 2315143 2025-06-19T08:58:44Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2001 on all recursors 2315144 wikitext text/x-wiki 7hx4md0bpvjpelksdze1y9g5ee9bpjo 2315145 2315144 2025-06-19T08:58:47Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2001 on all recursors 2315145 wikitext text/x-wiki 643bouumgsfeaxgtc4e6iljm7paf3kn 2315146 2315145 2025-06-19T08:58:50Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2001 2315146 wikitext text/x-wiki 04839jknv4i8w0055i94ntdjixcadbx 2315147 2315146 2025-06-19T08:58:55Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2001 2315147 wikitext text/x-wiki qto1rcrp2hl3srqyd87p8jdzdw07tko 2315148 2315147 2025-06-19T08:58:58Z Stashbot 7414 urbanecm@deploy1003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply 2315148 wikitext text/x-wiki 2knm0a9tci2kvqe50vco7wunrenfmop 2315149 2315148 2025-06-19T08:59:02Z Stashbot 7414 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s8 T397164 2315149 wikitext text/x-wiki axjo531hbfw4emfxoyunrm61wnzzyp4 2315150 2315149 2025-06-19T08:59:26Z Stashbot 7414 urbanecm@deploy1003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply 2315150 wikitext text/x-wiki f6y27yqrq4reo7mb6boijfpsoc7g49h 2315151 2315150 2025-06-19T08:59:33Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kafka-stretch2001 to dse-k8s-worker2001 2315151 wikitext text/x-wiki n9yrnmorb9pxtixalqq9lhc4290rz21 2315152 2315151 2025-06-19T09:00:00Z Stashbot 7414 urbanecm@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply 2315152 wikitext text/x-wiki d4iir6kkpvxe8kfdymkwf4m47rqs1z4 2315153 2315152 2025-06-19T09:00:50Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.rename from kafka-stretch2002 to dse-k8s-worker2002 2315153 wikitext text/x-wiki 3cxfzv0vemsafj1gaw5p1ehvsb9b1d4 2315154 2315153 2025-06-19T09:01:09Z Stashbot 7414 urbanecm@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply 2315154 wikitext text/x-wiki 76ts1bspgyaebrj4ra950ud05r97zy0 2315155 2315154 2025-06-19T09:01:13Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.netbox 2315155 wikitext text/x-wiki 5tfosc9k8859v4vpqwsfzyw4404v3x6 2315156 2315155 2025-06-19T09:01:18Z Stashbot 7414 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply 2315156 wikitext text/x-wiki 93qizdxveqnco8xlsfp48cb0koggbij 2315157 2315156 2025-06-19T09:02:13Z Stashbot 7414 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply 2315157 wikitext text/x-wiki 2yvvtb6ksh072dwis465txznkbol7kb 2315161 2315157 2025-06-19T09:08:53Z Stashbot 7414 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host krb2002.codfw.wmnet 2315161 wikitext text/x-wiki tr7v8p7ny7le5yla0ezy3fj8dcckd7u 2315164 2315161 2025-06-19T09:10:41Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kafka-stretch2002 to dse-k8s-worker2002 - btullis@cumin1003" 2315164 wikitext text/x-wiki rhj4zittosima4x7gebs1pq0wv2s9yd 2315165 2315164 2025-06-19T09:14:17Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kafka-stretch2002 to dse-k8s-worker2002 - btullis@cumin1003" 2315165 wikitext text/x-wiki iz2h9w0hkk4moob4l7z9f3n656s8a46 2315166 2315165 2025-06-19T09:14:20Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2315166 wikitext text/x-wiki g5b74makrx28xshhaz1631i9oy38bjh 2315167 2315166 2025-06-19T09:14:23Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2002 on all recursors 2315167 wikitext text/x-wiki jexd76mttnnk6x5eb7lrh1gab75mswx 2315168 2315167 2025-06-19T09:14:26Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2002 on all recursors 2315168 wikitext text/x-wiki dr2xz0uus6g3abisjpbbbwzo7ru1tid 2315169 2315168 2025-06-19T09:14:29Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2002 2315169 wikitext text/x-wiki oyv98spxmy792ghp0yiut7hm4eiijj2 2315170 2315169 2025-06-19T09:14:32Z Stashbot 7414 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb2002.codfw.wmnet 2315170 wikitext text/x-wiki no5ln7dpw4rpyh9rphk9hduxxod0fup 2315171 2315170 2025-06-19T09:15:34Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2196', diff saved to https://phabricator.wikimedia.org/P78422 and previous config saved to /var/cache/conftool/dbconfig/20250619-091532-root.json 2315171 wikitext text/x-wiki thjmq1vqkm3fe7bf979uj9bxapfg7e6 2315172 2315171 2025-06-19T09:15:43Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T396130)', diff saved to https://phabricator.wikimedia.org/P78423 and previous config saved to /var/cache/conftool/dbconfig/20250619-091539-marostegui.json 2315172 wikitext text/x-wiki 0c5bmmb391ln8h4t13rkcuh9w0d7fp6 2315173 2315172 2025-06-19T09:16:03Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2196.codfw.wmnet with reason: Maintenance 2315173 wikitext text/x-wiki c7nj928frs6gr88v2du5rmo4sg7k33l 2315174 2315173 2025-06-19T09:16:51Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2002 2315174 wikitext text/x-wiki pnzwn0on8r4zk4hwtews5u273jfp0e4 2315175 2315174 2025-06-19T09:17:30Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kafka-stretch2002 to dse-k8s-worker2002 2315175 wikitext text/x-wiki jhzxwi796pk7slt7m0s55oeqxhs3cwh 2315177 2315175 2025-06-19T09:19:13Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host aux-k8s-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2315177 wikitext text/x-wiki aw6k8mw0hqk10mka2jyhg0wgki2qv2w 2315178 2315177 2025-06-19T09:19:17Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aux-k8s-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2315178 wikitext text/x-wiki 4spclozurx2xy84xwmw07gzupaupf6r 2315179 2315178 2025-06-19T09:22:09Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host aux-k8s-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2315179 wikitext text/x-wiki fuxyryx300y39x921234wkd6xpob1wf 2315180 2315179 2025-06-19T09:22:13Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aux-k8s-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2315180 wikitext text/x-wiki d31uwx0t049i8yardc2238pu5iln00n 2315181 2315180 2025-06-19T09:25:53Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host aux-k8s-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2315181 wikitext text/x-wiki 8dumdr3q8eyneyawhayz699lhowtrax 2315182 2315181 2025-06-19T09:25:56Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aux-k8s-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2315182 wikitext text/x-wiki snon4t6sfg1jzomvcklt33a13xy007q 2315183 2315182 2025-06-19T09:28:03Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db2196 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P78424 and previous config saved to /var/cache/conftool/dbconfig/20250619-092801-root.json 2315183 wikitext text/x-wiki 2hzkp854boqcflhs530hhrmldpfxkj5 2315184 2315183 2025-06-19T09:28:42Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host aux-k8s-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2315184 wikitext text/x-wiki l14ixoa09bltuep8q01dwxbh4mbwcw1 2315185 2315184 2025-06-19T09:30:28Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2001.codfw.wmnet with OS bookworm 2315185 wikitext text/x-wiki 2etn3s4j8lxa3wfnp9g7cj8jz3cimgj 2315186 2315185 2025-06-19T09:30:38Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.move-vlan for host dse-k8s-worker2001 2315186 wikitext text/x-wiki 4j9ismgmbdlmpti03fg5zmdxpoaksor 2315187 2315186 2025-06-19T09:30:49Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P78425 and previous config saved to /var/cache/conftool/dbconfig/20250619-093047-marostegui.json 2315187 wikitext text/x-wiki ew127owppgkwtem2hmdoyd5a857bl8n 2315188 2315187 2025-06-19T09:31:23Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.netbox 2315188 wikitext text/x-wiki ex7sbp77b3ydo50a9ytgwu74yohtxdv 2315189 2315188 2025-06-19T09:33:16Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2315189 wikitext text/x-wiki 2c50pax3onzljoskmjmmu0arvb60ujv 2315190 2315189 2025-06-19T09:33:57Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aux-k8s-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2315190 wikitext text/x-wiki 3q1nviqcyumesli5rur1ismzaj3oixh 2315194 2315190 2025-06-19T09:36:25Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply 2315194 wikitext text/x-wiki gpug8nl3dvw1rui2o8gfik9n1occfys 2315195 2315194 2025-06-19T09:36:33Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply 2315195 wikitext text/x-wiki 5ntnc9s2qvebwh5er2xw1uraffq6cc2 2315197 2315195 2025-06-19T09:39:50Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host dse-k8s-worker2001 - btullis@cumin1003" 2315197 wikitext text/x-wiki j1h3k1c3xmu9kdz4s4j2tlm170izkjk 2315198 2315197 2025-06-19T09:39:54Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host dse-k8s-worker2001 - btullis@cumin1003" 2315198 wikitext text/x-wiki mw6xautwn8pkay50kgyc8ltymp1rx11 2315199 2315198 2025-06-19T09:39:57Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2315199 wikitext text/x-wiki 2hc2mvk59xqh25co9b5u101emdmlq14 2315200 2315199 2025-06-19T09:40:00Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2001.codfw.wmnet 126.32.192.10.in-addr.arpa 6.2.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2315200 wikitext text/x-wiki fn137mnf7ph6vr0etsfrqeztl90been 2315201 2315200 2025-06-19T09:40:03Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2001.codfw.wmnet 126.32.192.10.in-addr.arpa 6.2.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2315201 wikitext text/x-wiki 665gmyhqymuq07i3dfenjwl2op2z1ns 2315202 2315201 2025-06-19T09:40:07Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2001 2315202 wikitext text/x-wiki oscgmk85rafkwehkbpghx3a81mht8hn 2315203 2315202 2025-06-19T09:40:26Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2001 2315203 wikitext text/x-wiki 813cv92g3cb8cs0kxyx64zufl20sjee 2315204 2315203 2025-06-19T09:40:29Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host dse-k8s-worker2001 2315204 wikitext text/x-wiki qu2wbmiqa6rgtsz36rxrw7rf38nsdes 2315205 2315204 2025-06-19T09:43:09Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db2196 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P78426 and previous config saved to /var/cache/conftool/dbconfig/20250619-094306-root.json 2315205 wikitext text/x-wiki 2hyzrjar9uc4amg8f6kncphztv6c8oj 2315206 2315205 2025-06-19T09:43:34Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2315206 wikitext text/x-wiki o6jylm73l1cfkkdljaoebxr9sgxvh5z 2315207 2315206 2025-06-19T09:44:27Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "New api endpoints for the requestctl client - oblivian@cumin1003" 2315207 wikitext text/x-wiki jp2ioucan9gxykwd9ehfah03ghoiqkx 2315208 2315207 2025-06-19T09:44:30Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: New api endpoints for the requestctl client - oblivian@cumin1003 2315208 wikitext text/x-wiki lno1bypocu3kunitqh0bo16v0ru7gva 2315209 2315208 2025-06-19T09:45:01Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: New api endpoints for the requestctl client - oblivian@cumin1003 2315209 wikitext text/x-wiki doglhrxlgsajjkflydnlqwfiep2s0mh 2315210 2315209 2025-06-19T09:45:04Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "New api endpoints for the requestctl client - oblivian@cumin1003" 2315210 wikitext text/x-wiki phhzwrb586m8o58lifuacoiqc678kf6 2315211 2315210 2025-06-19T09:45:56Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P78427 and previous config saved to /var/cache/conftool/dbconfig/20250619-094554-marostegui.json 2315211 wikitext text/x-wiki lk1v2dkwlj5cxaqgx97rzfi3u5pt6ge 2315213 2315211 2025-06-19T09:52:23Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Homer release to add wikikube-worker-exp - cmooney@cumin1003 2315213 wikitext text/x-wiki 48pw84od5kqzd6mpivzu2odwj3rxc62 2315214 2315213 2025-06-19T09:54:49Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Homer release to add wikikube-worker-exp - cmooney@cumin1003 2315214 wikitext text/x-wiki avqq3bzr8c71i4mwd868jsuzxqmftbo 2315215 2315214 2025-06-19T09:57:07Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2001.codfw.wmnet with reason: host reimage 2315215 wikitext text/x-wiki 4ap1lter7dk0k4lkizookenrdxvogob 2315216 2315215 2025-06-19T09:58:13Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db2196 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P78428 and previous config saved to /var/cache/conftool/dbconfig/20250619-095811-root.json 2315216 wikitext text/x-wiki fa282nusogh0hx7zprvet4mhdttyqys 2315217 2315216 2025-06-19T10:01:05Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T396130)', diff saved to https://phabricator.wikimedia.org/P78429 and previous config saved to /var/cache/conftool/dbconfig/20250619-100102-marostegui.json 2315217 wikitext text/x-wiki lnqnlqg8ao1crpxk110fylwlndtala8 2315218 2315217 2025-06-19T10:01:19Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2198.codfw.wmnet with reason: Maintenance 2315218 wikitext text/x-wiki 5aji5k1wnp0q5a6py0nvhaxtoj8ip8l 2315219 2315218 2025-06-19T10:02:48Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2001.codfw.wmnet with reason: host reimage 2315219 wikitext text/x-wiki 0y81qdo6y1ducn2hqoirw2sk7q2h27j 2315220 2315219 2025-06-19T10:04:08Z Stashbot 7414 akosiaris@cumin1003: START - Cookbook sre.hosts.reimage for host aux-k8s-worker2006.codfw.wmnet with OS bookworm 2315220 wikitext text/x-wiki bvw6bw3wif60bzw3wxlsgpnqwdt34in 2315221 2315220 2025-06-19T10:12:39Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/changeprop: apply 2315221 wikitext text/x-wiki 48s3u05odlzpz9mng56kdhjiqh3kirt 2315222 2315221 2025-06-19T10:12:46Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/changeprop: apply 2315222 wikitext text/x-wiki hhk5aq5hane29f6cpij2h2r6gk414s3 2315223 2315222 2025-06-19T10:12:49Z Stashbot 7414 godog: powercycle netmon1003 2315223 wikitext text/x-wiki qncfqv5f1xly01czopnfzxg8tsrdei8 2315224 2315223 2025-06-19T10:13:19Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db2196 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P78430 and previous config saved to /var/cache/conftool/dbconfig/20250619-101317-root.json 2315224 wikitext text/x-wiki fgogcvpjgjvl68gsc49by6l4c6q5ty9 2315225 2315224 2025-06-19T10:14:26Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply 2315225 wikitext text/x-wiki 9cqgqhf1i62nymi2oky3idj15n1h0z9 2315226 2315225 2025-06-19T10:14:35Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply 2315226 wikitext text/x-wiki os7huyk6fesgapoggew9ykwkzaahnzk 2315227 2315226 2025-06-19T10:14:44Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: apply 2315227 wikitext text/x-wiki cqjhdng8g8lmx85hr2s1cnligla3tc3 2315228 2315227 2025-06-19T10:14:49Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply 2315228 wikitext text/x-wiki lj4fwau54awwgf1bjw5u089zlp65aw7 2315229 2315228 2025-06-19T10:16:40Z Stashbot 7414 akosiaris@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-worker2006.codfw.wmnet with reason: host reimage 2315229 wikitext text/x-wiki n6pvqmucknh6xj55d5tikrhyo54sqos 2315230 2315229 2025-06-19T10:16:44Z Stashbot 7414 akosiaris@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on aux-k8s-worker2006.codfw.wmnet with reason: host reimage 2315230 wikitext text/x-wiki s2e2p46v6qpkguhubsnm65jwhrabrnv 2315231 2315230 2025-06-19T10:17:12Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/changeprop: apply 2315231 wikitext text/x-wiki kxob4ub2n6gikejbf5murm6wx8rlk88 2315232 2315231 2025-06-19T10:17:38Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2200.codfw.wmnet with reason: Maintenance 2315232 wikitext text/x-wiki muuhtbqkmrdvf1ed18q0f6r8w1c80df 2315233 2315232 2025-06-19T10:18:32Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/changeprop: apply 2315233 wikitext text/x-wiki rzo43xyvltwhgua95psotm31vem79de 2315234 2315233 2025-06-19T10:19:12Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2001.codfw.wmnet with OS bookworm 2315234 wikitext text/x-wiki kpflj1kqcud4x21rovwxdcax55oivv7 2315235 2315234 2025-06-19T10:19:39Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2002.codfw.wmnet with OS bookworm 2315235 wikitext text/x-wiki sr8ksgbibmrumq2f15c9i7yxg2tewgl 2315236 2315235 2025-06-19T10:19:48Z Stashbot 7414 Emperor: depool / restart / repool ms-fe1009 [some idle timeouts] 2315236 wikitext text/x-wiki 1ht8ekv80g9n21ym9ucewnvlxkykcie 2315237 2315236 2025-06-19T10:19:51Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.move-vlan for host dse-k8s-worker2002 2315237 wikitext text/x-wiki tmwt1i8bu064tg1gmznfvvydr9bp2h8 2315238 2315237 2025-06-19T10:19:54Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.netbox 2315238 wikitext text/x-wiki r0ngr3z4vckkq2tqvesfmec0e889st9 2315239 2315238 2025-06-19T10:20:52Z Stashbot 7414 Amir1: dropping searchindex table in itwiki (T397367) 2315239 wikitext text/x-wiki r5ol1i8hxd2ifav1rclx5o0qju8tblg 2315240 2315239 2025-06-19T10:23:08Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/changeprop: apply 2315240 wikitext text/x-wiki gbhko9o9qmnahizpy9mobh2cguvmcyc 2315241 2315240 2025-06-19T10:23:11Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Fix api token loading - oblivian@cumin1003" 2315241 wikitext text/x-wiki iyicnzu3fuo2ukspk6m92hx0kk4vp1b 2315242 2315241 2025-06-19T10:23:15Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix api token loading - oblivian@cumin1003 2315242 wikitext text/x-wiki 848u8u77ugjr0rzmfxl5ywj4qfoqtcl 2315243 2315242 2025-06-19T10:23:43Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix api token loading - oblivian@cumin1003 2315243 wikitext text/x-wiki 3z7t3bbcq4jmayt64ylneguzd1itiyy 2315244 2315243 2025-06-19T10:23:46Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Fix api token loading - oblivian@cumin1003" 2315244 wikitext text/x-wiki 7enx4kf3f2xbeaf5r2zvqs95lzfxpkl 2315245 2315244 2025-06-19T10:23:49Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/changeprop: apply 2315245 wikitext text/x-wiki ll2qgi67vubzjjmj851g0oxrpgh00us 2315246 2315245 2025-06-19T10:23:54Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply 2315246 wikitext text/x-wiki 0eoxzj1i8fke2hq1ofibfm8bn8vo14l 2315247 2315246 2025-06-19T10:24:00Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply 2315247 wikitext text/x-wiki hf6hafrf35wzg06ugy2kj976masy48a 2315248 2315247 2025-06-19T10:25:26Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host dse-k8s-worker2002 - btullis@cumin1003" 2315248 wikitext text/x-wiki te82ug6pn60rgt6zzikrkwki87iejly 2315249 2315248 2025-06-19T10:25:30Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host dse-k8s-worker2002 - btullis@cumin1003" 2315249 wikitext text/x-wiki jf4gbxvm2r20hosik59duzkwmo3f8b9 2315250 2315249 2025-06-19T10:25:33Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2315250 wikitext text/x-wiki t6jaqquqo3wj12llzcdvx81tl8bqp3w 2315251 2315250 2025-06-19T10:25:36Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2002.codfw.wmnet 86.48.192.10.in-addr.arpa 6.8.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2315251 wikitext text/x-wiki 4rknmv4nx0von6w1vwfapvz78d0606e 2315252 2315251 2025-06-19T10:25:39Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2002.codfw.wmnet 86.48.192.10.in-addr.arpa 6.8.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2315252 wikitext text/x-wiki 791cffn58xfwkcn6qahp9kfpxzydjvt 2315253 2315252 2025-06-19T10:25:42Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2002 2315253 wikitext text/x-wiki ocoq3eg081jotnjmsj3pwooszjvxdej 2315254 2315253 2025-06-19T10:25:46Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2002 2315254 wikitext text/x-wiki opc8d1bkbmb8plyayx7lub5cbcs2rl7 2315255 2315254 2025-06-19T10:25:49Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host dse-k8s-worker2002 2315255 wikitext text/x-wiki snrsrx4h9hs4tv28mztuplv8ryuspjs 2315256 2315255 2025-06-19T10:28:58Z Stashbot 7414 moritzm: installing postgresql-13 security updates 2315256 wikitext text/x-wiki h56vmm7q1jyy6hqmgcfmcr3gxe2ne06 2315257 2315256 2025-06-19T10:31:17Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/changeprop: apply 2315257 wikitext text/x-wiki mjqay78ddhst4638b5zbe65vpoz2h77 2315258 2315257 2025-06-19T10:31:30Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/changeprop: apply 2315258 wikitext text/x-wiki 52gsl18wq77iai2xyiwyyieei5vpe7n 2315259 2315258 2025-06-19T10:31:34Z Stashbot 7414 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-cluster depool all services in codfw/codfw: maintenance 2315259 wikitext text/x-wiki ix3nyvs4hnmqi3rb5pdpvi8svoehntg 2315260 2315259 2025-06-19T10:32:18Z Stashbot 7414 moritzm: installing twisted security updates 2315260 wikitext text/x-wiki b4xt6453gbhjg5hbqzpy7j46bcc4ox1 2315261 2315260 2025-06-19T10:32:22Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply 2315261 wikitext text/x-wiki hu3s0cgyp0x7aqncino2nmj0lrfxlis 2315262 2315261 2025-06-19T10:32:25Z Stashbot 7414 akosiaris@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-worker2006.codfw.wmnet with OS bookworm 2315262 wikitext text/x-wiki 6hcgy2ej3vmzufq419u55e18ix1st16 2315263 2315262 2025-06-19T10:33:55Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2208.codfw.wmnet with reason: Maintenance 2315263 wikitext text/x-wiki t4ng5j1lbxyye1mrvd19noacp65lxvf 2315264 2315263 2025-06-19T10:34:03Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2208 (T396130)', diff saved to https://phabricator.wikimedia.org/P78431 and previous config saved to /var/cache/conftool/dbconfig/20250619-103400-marostegui.json 2315264 wikitext text/x-wiki j5ry71cxiyh19ryq9ofzqtwjetvzxet 2315265 2315264 2025-06-19T10:39:54Z Stashbot 7414 moritzm: installing Django security updates 2315265 wikitext text/x-wiki 4ha1x1oqgl6ijalwc9y97r5k5fwicqa 2315266 2315265 2025-06-19T10:42:41Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply 2315266 wikitext text/x-wiki i2rhxwzksllqbkim6a8ggegawmpmzn4 2315267 2315266 2025-06-19T10:42:45Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2002.codfw.wmnet with reason: host reimage 2315267 wikitext text/x-wiki 3ffmf296wcx9hf49xrfm2qp65zbkw6f 2315268 2315267 2025-06-19T10:46:17Z Stashbot 7414 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in codfw/codfw: maintenance 2315268 wikitext text/x-wiki 9v86oafy391dkwya9ogn8wscmwq5sdy 2315269 2315268 2025-06-19T10:46:23Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2002.codfw.wmnet with reason: host reimage 2315269 wikitext text/x-wiki jhul9dfex9sudk9r342mnyfxjt8w6yz 2315270 2315269 2025-06-19T10:47:14Z Stashbot 7414 slyngshede@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - T397300 2315270 wikitext text/x-wiki dk5m6arjq8htllhnwazbnp4mvjrvzrx 2315271 2315270 2025-06-19T10:50:46Z Stashbot 7414 slyngshede@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - T397300 2315271 wikitext text/x-wiki l860j0yd8m8g3lyobnclauqbg264ita 2315272 2315271 2025-06-19T10:53:50Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208 (T396130)', diff saved to https://phabricator.wikimedia.org/P78432 and previous config saved to /var/cache/conftool/dbconfig/20250619-105347-marostegui.json 2315272 wikitext text/x-wiki 9cl456ld0di4mjdtib733605ifyzowu 2315273 2315272 2025-06-19T11:02:41Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2002.codfw.wmnet with OS bookworm 2315273 wikitext text/x-wiki 6iyip8ryuj3c7y208qpp0jinkf26ndb 2315274 2315273 2025-06-19T11:08:56Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P78433 and previous config saved to /var/cache/conftool/dbconfig/20250619-110854-marostegui.json 2315274 wikitext text/x-wiki 8fuwzcxtt36836hw9saie6mn1qvppnw 2315277 2315274 2025-06-19T11:24:03Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P78434 and previous config saved to /var/cache/conftool/dbconfig/20250619-112401-marostegui.json 2315277 wikitext text/x-wiki tw69pphwudxncddzl2jzn1w6xrc8boe 2315280 2315277 2025-06-19T11:39:12Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208 (T396130)', diff saved to https://phabricator.wikimedia.org/P78435 and previous config saved to /var/cache/conftool/dbconfig/20250619-113908-marostegui.json 2315280 wikitext text/x-wiki dymmb2rp96clb02ybt9hoynyo8y5b2a 2315281 2315280 2025-06-19T11:39:26Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2220.codfw.wmnet with reason: Maintenance 2315281 wikitext text/x-wiki ghs0kh51dzf7j2mbfbsl8kt13ehym31 2315282 2315281 2025-06-19T11:39:34Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2220 (T396130)', diff saved to https://phabricator.wikimedia.org/P78436 and previous config saved to /var/cache/conftool/dbconfig/20250619-113931-marostegui.json 2315282 wikitext text/x-wiki ta26pblbs08drlptbrwa4xbg3hg8sg9 2315283 2315282 2025-06-19T11:42:44Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2315283 wikitext text/x-wiki bmuoxklrm2xv3r6v1miuh91r1xlgapc 2315284 2315283 2025-06-19T11:43:27Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply 2315284 wikitext text/x-wiki 3bwkfv5uew2oojnlnyfe0va1tew21oq 2315285 2315284 2025-06-19T11:46:24Z Stashbot 7414 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-cluster pool all services in codfw/codfw: maintenance 2315285 wikitext text/x-wiki mpc88d08vs8maktxsb056wpkmuh4156 2315286 2315285 2025-06-19T11:56:29Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T396130)', diff saved to https://phabricator.wikimedia.org/P78437 and previous config saved to /var/cache/conftool/dbconfig/20250619-115626-marostegui.json 2315286 wikitext text/x-wiki 5r9qt9ry2qugxf55avyqabhxlakeghv 0 14218 2314978 2306597 2025-06-18T21:56:17Z Ladsgroup 1264 /* Setting a external storage section to read only */ 2314978 wikitext text/x-wiki 8lqcf5kznrgb46tq90rm56q8pzv6xmr 2314985 2314978 2025-06-18T22:09:49Z Ladsgroup 1264 /* Temporarily read only */ 2314985 wikitext text/x-wiki 6ul4iy2h2gxc5usjm5366t4o866sn1t 2315014 2314985 2025-06-19T01:16:34Z Neil Shah-Quinn (WMF) 36601 Remove content moved to [[MariaDB/Administration]] 2315014 wikitext text/x-wiki eqp4vfesswau5t6asusmc1flm61nx9r 0 17290 2314530 2314274 2025-06-18T13:06:57Z Stashbot 7414 James_F: Zuul: [mediawiki/extensions/ImageMapEdit] Use bluespice template 2314530 wikitext text/x-wiki spjswif1248kok0razjdq4fsr9zeecb 2314534 2314530 2025-06-18T13:08:30Z Stashbot 7414 James_F: Zuul: Add James E. Blair to CI allowlist 2314534 wikitext text/x-wiki 19g9682x2rht13ttitdtphwotxvh75b 2314536 2314534 2025-06-18T13:10:30Z Stashbot 7414 James_F: Zuul: Add EggRoll97 to CI allowlist 2314536 wikitext text/x-wiki 74bs4ajvtqzjoubxjuap1cn77390h5n 2314841 2314536 2025-06-18T18:19:19Z Stashbot 7414 bd808: `docker system prune --all` on runner-1023.gitlab-runners.eqiad1.wikimedia.cloud 2314841 wikitext text/x-wiki 8dhlztjfooa035x8fg136jqt5sn83ci 2314981 2314841 2025-06-18T22:02:19Z Stashbot 7414 bd808: Blocked 146.174.160.0/19 "Huawei-Cloud-SG" network. (T397378) 2314981 wikitext text/x-wiki br9am8zs6nk0tlyf79yeyz8ck5kbr6w 2314987 2314981 2025-06-18T22:10:18Z Stashbot 7414 bd808: Blocked 202.76.160.0/20 "Huawei-Cloud-SG" network. (T397378) 2314987 wikitext text/x-wiki pwju08a2o4wjgi4jn5djlbl08iowmzk 2315006 2314987 2025-06-18T23:26:39Z Stashbot 7414 bd808: Blocked 128.241.0.0/16 "NTT America" network. (T397378) 2315006 wikitext text/x-wiki s6m5i0vmwij4pn5fnw8hppycuu57idy 2315163 2315006 2025-06-19T09:10:30Z Stashbot 7414 urbanecm: deployment-prep: Update changeprop config per https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1150699 using [[wikitech:Changeprop#To_deployment-prep]] (T394958) 2315163 wikitext text/x-wiki 6oit8utrwuhc5leqftnvmrzygu3fgsg 2315176 2315163 2025-06-19T09:18:54Z Stashbot 7414 urbanecm: deployment-prep: Update changeprop config perhttps://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1161443 using [[wikitech:Changeprop#To_deployment-prep]] (T394958; this time actually changing the beta config) 2315176 wikitext text/x-wiki sc10vhrr1ieaxowbl6jqldn6afnkv4e 2315196 2315176 2025-06-19T09:38:52Z Stashbot 7414 sergi0: deployment-prep: GrowthExperiments config migration `foreachwiki extensions/CommunityConfiguration/maintenance/migrateConfig.php GrowthSuggestedEdits` — T393771 2315196 wikitext text/x-wiki peg6ckqz66n22zpyb1ray1i338fhw2y 498 30942 2314648 2314093 2025-06-18T15:02:12Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph}' 2314648 wikitext text/x-wiki c65p3l9shte0lg18bw0wjuvjvblc8ux 2314789 2314648 2025-06-18T17:37:18Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph}' 2314789 wikitext text/x-wiki gktz2rer5bpunnpjeb16zeyiskd1too 2314931 2314789 2025-06-18T20:27:20Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services 2314931 wikitext text/x-wiki b6lk71ctv9v5835o8guw5qdm6cxsmk4 2314939 2314931 2025-06-18T20:39:57Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services 2314939 wikitext text/x-wiki 6wb013qfq1ygh1m1cc53o7j1il05rym 2315021 2314939 2025-06-19T04:40:20Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services 2315021 wikitext text/x-wiki 56suyrp8fuyzr2c6urt9568alij94nt 2315022 2315021 2025-06-19T04:48:10Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services 2315022 wikitext text/x-wiki rkwu8mhubwv75d5dpxepgukvfsw6qho 10 37907 2315015 2301052 2025-06-19T01:18:45Z Neil Shah-Quinn (WMF) 36601 Add link to new [[MariaDB/Administration]] page 2315015 wikitext text/x-wiki fh1twkigiosg9jprwyeujuolgt1nsl6 2315016 2315015 2025-06-19T01:19:02Z Neil Shah-Quinn (WMF) 36601 Fix typo 2315016 wikitext text/x-wiki dn26g4flfi7qvfyy86q3vzjv9cxgx5k 10 54864 2315102 2305253 2025-06-19T08:03:52Z Taavi-WMF 41365 update link 2315102 wikitext text/x-wiki btupziouuzmr5cl3cxtfbqtlqlmexii 12 101117 2315035 2054774 2025-06-19T05:25:42Z Dexbot 30554 Bot: Bot: Fixing double redirect to [[Help:Toolforge/Jobs Service]] 2315035 wikitext text/x-wiki r5gxjecv5e4phlqd3sqvjs2a07f3v52 2315113 2315035 2025-06-19T08:13:11Z Wikitech double redirect bot 37243 Bot: Fixing double redirect from [[Help:Toolforge/Jobs Service]] to [[Help:Toolforge/Running jobs]] 2315113 wikitext text/x-wiki akgz2xdpbtp0lb31eor7znr54obpcfx 0 443239 2314804 2122486 2025-06-18T17:50:58Z Krinkle 44 /* Deployment */ new section 2314804 wikitext text/x-wiki ipwudjg9gt7ckwbp1z7ylug8vxqxf6e 2315020 2314804 2025-06-19T04:28:58Z Krinkle 44 /* Monitoring */ link to SAL for deployments 2315020 wikitext text/x-wiki gz5afpt2dz77zgj12anj6pse6q53w6a 498 443800 2314860 2311473 2025-06-18T18:42:07Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-13: deployed 8f93390eb6 (&actions= and &title= parameters in /batch/new/pagepile, T397320) 2314860 wikitext text/x-wiki ktveb1u8i7ztedlrb7wec551kiv2qkw 0 445088 2315010 2241545 2025-06-19T01:07:21Z Neil Shah-Quinn (WMF) 36601 Neil Shah-Quinn (WMF) moved page [[MariaDB/monitoring]] to [[MariaDB/Monitoring]]: Use sentence case 2241545 wikitext text/x-wiki lzxatu38wcf7tlcg1wz8tfm784wct3e 0 447004 2314901 2310883 2025-06-18T19:59:32Z Neil Shah-Quinn (WMF) 36601 /* Quarto reports */ Link to publication instructions and copyedit headings 2314901 wikitext text/x-wiki 5phbkdn6wr7bykzl1o0phsl19peou3m 0 447294 2315032 1902387 2025-06-19T05:24:42Z Dexbot 30554 Bot: Bot: Fixing double redirect to [[MariaDB/Monitoring]] 2315032 wikitext text/x-wiki 0e84ufawitdvvghz0kr0elkzsv0hqei 0 447310 2314935 2313370 2025-06-18T20:33:06Z MPopov (WMF) 5891 Added link to experiment-specific instrumentation spec template 2314935 wikitext text/x-wiki 0s5xpz3627saz6uvmns05h6h9zfe7hd 0 448236 2314526 2071847 2025-06-18T12:59:13Z JCrespo (WMF) 5427 /* Fetch backups from bacula */ backup1014 is the new director 2314526 wikitext text/x-wiki l7au9g91ctgis1v1i9x7qnxhd5255kp 0 448850 2314528 2306026 2025-06-18T13:03:08Z DCaro (WMF) 18271 DCaro (WMF) moved page [[Portal:Toolforge/Admin/Jobs framework]] to [[Portal:Toolforge/Admin/Jobs Service]]: Matching all the other services we have 2306026 wikitext text/x-wiki smlqdya5ytar8s222ytwg1v0hz5rode 2314575 2314528 2025-06-18T13:43:31Z DCaro (WMF) 18271 Updated a bit the docs with mentions of the api gateway, working examples and commands, removed outdated references to implementation details and renamed/removed some of the 'framework' references to 'service' where it made sense. 2314575 wikitext text/x-wiki qnrmyv8gbh68tsqes1hapnimpp9txa2 2314599 2314575 2025-06-18T14:06:47Z DCaro (WMF) 18271 2314599 wikitext text/x-wiki rgbp2gc0nmqjnqr0b2tpgux7v0xopy4 12 449113 2314542 2305144 2025-06-18T13:15:02Z DCaro (WMF) 18271 DCaro (WMF) moved page [[Help:Toolforge/Jobs framework]] to [[Help:Toolforge/Jobs Service]]: Matching all the other services we have 2305144 wikitext text/x-wiki r3ffzv3x1hq4dgqnvz6rhded2ljf12e 2314580 2314542 2025-06-18T13:55:01Z DCaro (WMF) 18271 /* See also */ 2314580 wikitext text/x-wiki 06w7jtgikqb57hj6srqrumerz2ocprp 2315090 2314580 2025-06-19T08:03:34Z Taavi-WMF 41365 Taavi-WMF moved page [[Help:Toolforge/Jobs Service]] to [[Help:Toolforge/Running jobs]]: user-facing pages are named after functionality they provide, not the technical implementation 2314580 wikitext text/x-wiki 06w7jtgikqb57hj6srqrumerz2ocprp 12 449126 2315036 1949056 2025-06-19T05:26:02Z Dexbot 30554 Bot: Bot: Fixing double redirect to [[Help:Toolforge/Jobs Service]] 2315036 wikitext text/x-wiki r5gxjecv5e4phlqd3sqvjs2a07f3v52 2315114 2315036 2025-06-19T08:13:21Z Wikitech double redirect bot 37243 Bot: Fixing double redirect from [[Help:Toolforge/Jobs Service]] to [[Help:Toolforge/Running jobs]] 2315114 wikitext text/x-wiki akgz2xdpbtp0lb31eor7znr54obpcfx 0 449160 2314807 2314295 2025-06-18T17:53:52Z Dexbot 30554 Bot: Updating the report 2314807 wikitext text/x-wiki my70ceo6s4abasz8hwxvl7779qsco1h 2315007 2314807 2025-06-19T00:00:46Z Dexbot 30554 Bot: Updating the report 2315007 wikitext text/x-wiki dk5ud85svftt2ljvhk4canzxpojdofz 2315026 2315007 2025-06-19T05:08:58Z Dexbot 30554 Bot: Updating the report 2315026 wikitext text/x-wiki 2tg0lkv48dvtnnxpch52npb5vs0udwb 2315040 2315026 2025-06-19T05:36:02Z Dexbot 30554 Bot: Updating the report 2315040 wikitext text/x-wiki lg92fgw95c2dva2hqjz2kwe8cfnnn7l 0 449207 2315033 1952676 2025-06-19T05:25:02Z Dexbot 30554 Bot: Bot: Fixing double redirect to [[Help:Toolforge/Jobs Service]] 2315033 wikitext text/x-wiki i3ny1peytawk1j1gs0ekp8aj73nt3xt 2315111 2315033 2025-06-19T08:12:52Z Wikitech double redirect bot 37243 Bot: Fixing double redirect from [[Help:Toolforge/Jobs Service]] to [[Help:Toolforge/Running jobs]] 2315111 wikitext text/x-wiki 37ier8m2umihns0yhktfdow7gqqvx5p 13 449208 2314544 2272056 2025-06-18T13:15:02Z DCaro (WMF) 18271 DCaro (WMF) moved page [[Help talk:Toolforge/Jobs framework]] to [[Help talk:Toolforge/Jobs Service]]: Matching all the other services we have 2272056 wikitext text/x-wiki 92j2w39i05o22nzba39zw7acjmizge2 2314609 2314544 2025-06-18T14:13:35Z Taavi-WMF 41365 /* Name */ new section 2314609 wikitext text/x-wiki g123demnya60wnpo36spkct05yj31m2 2315092 2314609 2025-06-19T08:03:35Z Taavi-WMF 41365 Taavi-WMF moved page [[Help talk:Toolforge/Jobs Service]] to [[Help talk:Toolforge/Running jobs]]: user-facing pages are named after functionality they provide, not the technical implementation 2314609 wikitext text/x-wiki g123demnya60wnpo36spkct05yj31m2 2315103 2315092 2025-06-19T08:04:09Z Taavi-WMF 41365 update base archive name 2315103 wikitext text/x-wiki bb3gks45iox1ctfmdb25rae30gvmjpt 0 450748 2315031 2153556 2025-06-19T05:24:22Z Dexbot 30554 Bot: Bot: Fixing double redirect to [[Portal:Toolforge/Admin/Jobs Service]] 2315031 wikitext text/x-wiki 8yowpjgy18ukfwxlrc2rlxrys607h0n 0 451360 2314972 2234020 2025-06-18T21:32:58Z Julievdh 41975 Some name changes, slowly adding experimentation questions 2314972 wikitext text/x-wiki 8jksvs0l7b66t672qofxt9ic2xm4apt 0 451475 2314904 2233825 2025-06-18T20:02:33Z RKemper (WMF) 17058 /* Teams */ Update wrt data platform team structure changes 2314904 wikitext text/x-wiki 3vg4svu88juy0kfik8hfwcddn31w874 0 453382 2315077 2306050 2025-06-19T07:36:52Z AYounsi (WMF) 10040 /* Future improvements */ 2315077 wikitext text/x-wiki jrcogy5lk8b15g3liib5jq1007yfa4u 0 453601 2315028 2119398 2025-06-19T05:23:22Z Dexbot 30554 Bot: Bot: Fixing double redirect to [[Help:Toolforge/Jobs Service]] 2315028 wikitext text/x-wiki r5gxjecv5e4phlqd3sqvjs2a07f3v52 2315110 2315028 2025-06-19T08:12:41Z Wikitech double redirect bot 37243 Bot: Fixing double redirect from [[Help:Toolforge/Jobs Service]] to [[Help:Toolforge/Running jobs]] 2315110 wikitext text/x-wiki akgz2xdpbtp0lb31eor7znr54obpcfx 0 453749 2315115 2171901 2025-06-19T08:16:00Z Taavi-WMF 41365 use double redirect fixing as an example here again 2315115 wikitext text/x-wiki qvp4hv96qpm4492tpu7qb594lqywrvq 116 453906 2314654 2314101 2025-06-18T15:21:14Z Gitlabaccountapprovalbot 37332 ayushkhati1 was rejected. 2314654 wikitext text/x-wiki imxagapy3uh9e2usot57slmc23dx4c4 0 453918 2315034 2137103 2025-06-19T05:25:22Z Dexbot 30554 Bot: Bot: Fixing double redirect to [[Help:Toolforge/Jobs Service]] 2315034 wikitext text/x-wiki r5gxjecv5e4phlqd3sqvjs2a07f3v52 2315112 2315034 2025-06-19T08:13:01Z Wikitech double redirect bot 37243 Bot: Fixing double redirect from [[Help:Toolforge/Jobs Service]] to [[Help:Toolforge/Running jobs]] 2315112 wikitext text/x-wiki akgz2xdpbtp0lb31eor7znr54obpcfx 0 453976 2314645 2247196 2025-06-18T14:51:52Z FCeratto-WMF 39104 Add header in the yaml template 2314645 wikitext text/x-wiki nqaj8bke9ax4kwpdeebb9gtnuez3y7i 0 454165 2315029 2153555 2025-06-19T05:23:42Z Dexbot 30554 Bot: Bot: Fixing double redirect to [[Portal:Toolforge/Admin/Jobs Service]] 2315029 wikitext text/x-wiki 8yowpjgy18ukfwxlrc2rlxrys607h0n 0 454168 2315030 2151371 2025-06-19T05:24:02Z Dexbot 30554 Bot: Bot: Fixing double redirect to [[Portal:Toolforge/Admin/Jobs Service]] 2315030 wikitext text/x-wiki 8yowpjgy18ukfwxlrc2rlxrys607h0n 498 454381 2315089 2310872 2025-06-19T08:02:52Z Stashbot 7414 JJMC89: copypatrol-backend-prod-01 deploy 8995949..3d93e79 - security update 2315089 wikitext text/x-wiki se27895sx7qrzafb8csedyqj9j0nkpy 13 454383 2314546 2166200 2025-06-18T13:15:03Z DCaro (WMF) 18271 DCaro (WMF) moved page [[Help talk:Toolforge/Jobs framework/Archives/2022]] to [[Help talk:Toolforge/Jobs Service/Archives/2022]]: Matching all the other services we have 2166200 wikitext text/x-wiki avei4590es0csr9grz72671lcorjk4o 2315094 2314546 2025-06-19T08:03:35Z Taavi-WMF 41365 Taavi-WMF moved page [[Help talk:Toolforge/Jobs Service/Archives/2022]] to [[Help talk:Toolforge/Running jobs/Archives/2022]]: user-facing pages are named after functionality they provide, not the technical implementation 2166200 wikitext text/x-wiki avei4590es0csr9grz72671lcorjk4o 13 454384 2314548 2166201 2025-06-18T13:15:03Z DCaro (WMF) 18271 DCaro (WMF) moved page [[Help talk:Toolforge/Jobs framework/Archives/2023]] to [[Help talk:Toolforge/Jobs Service/Archives/2023]]: Matching all the other services we have 2166201 wikitext text/x-wiki 8lqo9v6yr9t7pg3mxiw7xgescv2qfdr 2315096 2314548 2025-06-19T08:03:35Z Taavi-WMF 41365 Taavi-WMF moved page [[Help talk:Toolforge/Jobs Service/Archives/2023]] to [[Help talk:Toolforge/Running jobs/Archives/2023]]: user-facing pages are named after functionality they provide, not the technical implementation 2166201 wikitext text/x-wiki 8lqo9v6yr9t7pg3mxiw7xgescv2qfdr 13 454385 2314550 2166202 2025-06-18T13:15:04Z DCaro (WMF) 18271 DCaro (WMF) moved page [[Help talk:Toolforge/Jobs framework/Archives/2024]] to [[Help talk:Toolforge/Jobs Service/Archives/2024]]: Matching all the other services we have 2166202 wikitext text/x-wiki c6vv6rk4ac8xgrq63zlnayvddt62a6h 2315098 2314550 2025-06-19T08:03:36Z Taavi-WMF 41365 Taavi-WMF moved page [[Help talk:Toolforge/Jobs Service/Archives/2024]] to [[Help talk:Toolforge/Running jobs/Archives/2024]]: user-facing pages are named after functionality they provide, not the technical implementation 2166202 wikitext text/x-wiki c6vv6rk4ac8xgrq63zlnayvddt62a6h 0 455989 2314631 2312284 2025-06-18T14:34:42Z Phuedx 2014 beta cluster -> Beta Cluster 2314631 wikitext text/x-wiki buj1qq6ubvte5ywcij7m135ggesi4z5 0 456150 2314587 2312566 2025-06-18T14:00:10Z XCollazo-WMF 31846 /* FAQ */ Separarte sensor guarantee to its own FAQ. 2314587 wikitext text/x-wiki ovy5tu29h3xesw4xa4l3hsefoqzec9s 0 456182 2315192 2307876 2025-06-19T09:35:41Z Mvolz (WMF) 2813 /* Reconciliation */ 2315192 wikitext text/x-wiki s4sac6xvnghpboaqisdj7ol12f91925 0 456526 2315104 2304494 2025-06-19T08:04:55Z Taavi-WMF 41365 update toolforge link 2315104 wikitext text/x-wiki ep72ml0a8869dbzejj1wum8o942k91r 13 457945 2314552 2272055 2025-06-18T13:15:04Z DCaro (WMF) 18271 DCaro (WMF) moved page [[Help talk:Toolforge/Jobs framework/Archives/2025]] to [[Help talk:Toolforge/Jobs Service/Archives/2025]]: Matching all the other services we have 2272055 wikitext text/x-wiki hznuto7tle2aduv1ext3iw68579kv11 2315100 2314552 2025-06-19T08:03:36Z Taavi-WMF 41365 Taavi-WMF moved page [[Help talk:Toolforge/Jobs Service/Archives/2025]] to [[Help talk:Toolforge/Running jobs/Archives/2025]]: user-facing pages are named after functionality they provide, not the technical implementation 2272055 wikitext text/x-wiki hznuto7tle2aduv1ext3iw68579kv11 0 458681 2314591 2313218 2025-06-18T14:04:19Z JBorun (WMF) 41299 2314591 wikitext text/x-wiki 73mboxtxr2blpqsteu78ut3kq63wi1t 2314595 2314591 2025-06-18T14:06:07Z JBorun (WMF) 41299 2314595 wikitext text/x-wiki lp1xzgb88fvv9u9h5vn8ujw1r5gexan 2314598 2314595 2025-06-18T14:06:32Z JBorun (WMF) 41299 2314598 wikitext text/x-wiki 4qh6b9610zbccmve1sg6pkfnel8234c 2314644 2314598 2025-06-18T14:51:43Z JBorun (WMF) 41299 2314644 wikitext text/x-wiki bfcznq79hyw4113asalpt9qivx3ruoy 0 458842 2314687 2313236 2025-06-18T16:05:39Z Phuedx 2014 /* Hard Dependencies */ 2314687 wikitext text/x-wiki qxupw844beb9twkj88gyefnvmp0u0gr 0 458850 2314520 2313133 2025-06-18T12:46:13Z FNegri-WMF 32595 /* System Components */ fix link to gitlab 2314520 wikitext text/x-wiki oh4kh4mglx3aurh7rd3qovht9cz4hqb 0 458853 2314868 2314092 2025-06-18T18:58:13Z MPopov (WMF) 5891 2314868 wikitext text/x-wiki d6nm8pzbge9vm9qwqbyotp5yr6fl3gs 2314872 2314868 2025-06-18T19:02:16Z MPopov (WMF) 5891 Cleaned up table of planned experiments, added [[User:KGraessle-WMF]]'s upcoming experiment 2314872 wikitext text/x-wiki pd0btzstlubnoylvcwzhuk91wamcjx4 2314874 2314872 2025-06-18T19:05:00Z MPopov (WMF) 5891 /* Planned experiments */ Documented LPL's upcoming MinT for readers experiment 2314874 wikitext text/x-wiki goxhgitfcvwfhsu8560j91hz4uppobs 2314899 2314874 2025-06-18T19:57:25Z MPopov (WMF) 5891 Formatting 2314899 wikitext text/x-wiki 87nteysoih438zpb952tv6bcmt11q85 2314900 2314899 2025-06-18T19:58:32Z MPopov (WMF) 5891 /* Active experiments */ Link to synth A/A test 2's xLab read view 2314900 wikitext text/x-wiki 1kd69blg3n5wf8l1v5y40oznyxm7ebb 2314965 2314900 2025-06-18T21:19:19Z MPopov (WMF) 5891 Link to Web's A/B test 2314965 wikitext text/x-wiki m4esvfn8ok7r6jo5obne2bvcvw0qz7d 0 458857 2314597 2313978 2025-06-18T14:06:29Z Milimetric (WMF) 39785 2314597 wikitext text/x-wiki ft2ksezkj33i92jsoar2b65dynkmkap 2 458860 2314507 2025-06-18T12:13:25Z DCaro (WMF) 18271 Created page with "{{Note|text=This service is in a beta status, expect some bugs, weird behaviors and changes in the usage flows (both cli and API). Feel free to test it though we don't recommend using it for production services yet. Feel free to report any bugs/features/feedback here <insert link to phabulous template>}} {{Toolforge nav}} Toolforge tools can use the '''Toolforge Components Service''' to configure and deploy their tools [[Help:Toolforge/Jobs|jobs]] (web services comin..." 2314507 wikitext text/x-wiki 1n1iic4da6ssm6xj2ns2ldvl2akpqfb 0 458861 2314529 2025-06-18T13:03:08Z DCaro (WMF) 18271 DCaro (WMF) moved page [[Portal:Toolforge/Admin/Jobs framework]] to [[Portal:Toolforge/Admin/Jobs Service]]: Matching all the other services we have 2314529 wikitext text/x-wiki 8yowpjgy18ukfwxlrc2rlxrys607h0n 12 458862 2314543 2025-06-18T13:15:02Z DCaro (WMF) 18271 DCaro (WMF) moved page [[Help:Toolforge/Jobs framework]] to [[Help:Toolforge/Jobs Service]]: Matching all the other services we have 2314543 wikitext text/x-wiki r5gxjecv5e4phlqd3sqvjs2a07f3v52 13 458863 2314545 2025-06-18T13:15:03Z DCaro (WMF) 18271 DCaro (WMF) moved page [[Help talk:Toolforge/Jobs framework]] to [[Help talk:Toolforge/Jobs Service]]: Matching all the other services we have 2314545 wikitext text/x-wiki k7wlyvx8riynatp96sp9y7mh9y4un1e 13 458864 2314547 2025-06-18T13:15:03Z DCaro (WMF) 18271 DCaro (WMF) moved page [[Help talk:Toolforge/Jobs framework/Archives/2022]] to [[Help talk:Toolforge/Jobs Service/Archives/2022]]: Matching all the other services we have 2314547 wikitext text/x-wiki sj10d7ig6pg1mbz19kvlezf4yv952n1 13 458865 2314549 2025-06-18T13:15:04Z DCaro (WMF) 18271 DCaro (WMF) moved page [[Help talk:Toolforge/Jobs framework/Archives/2023]] to [[Help talk:Toolforge/Jobs Service/Archives/2023]]: Matching all the other services we have 2314549 wikitext text/x-wiki 1qc1t1lpbalys8vk6uzpfbcx1gn8zvp 13 458866 2314551 2025-06-18T13:15:04Z DCaro (WMF) 18271 DCaro (WMF) moved page [[Help talk:Toolforge/Jobs framework/Archives/2024]] to [[Help talk:Toolforge/Jobs Service/Archives/2024]]: Matching all the other services we have 2314551 wikitext text/x-wiki i9stf2gmxy9yov5ip9g26cfwvqynh5i 13 458867 2314553 2025-06-18T13:15:04Z DCaro (WMF) 18271 DCaro (WMF) moved page [[Help talk:Toolforge/Jobs framework/Archives/2025]] to [[Help talk:Toolforge/Jobs Service/Archives/2025]]: Matching all the other services we have 2314553 wikitext text/x-wiki 0mai6ojbib62ffzyng4tjjxe15v3li3 0 458868 2314592 2025-06-18T14:04:47Z DCaro (WMF) 18271 Will try to add diagrams later 2314592 wikitext text/x-wiki rhmy436mg2swnt45f2u31l7qin79wq8 2314600 2314592 2025-06-18T14:07:53Z DCaro (WMF) 18271 /* See also */ 2314600 wikitext text/x-wiki l0k1sqyrspcx3y4mp6xsoj8pq8ivl1v 0 458869 2314929 2025-06-18T20:23:45Z BryanDavis 1604 Redirected page to [[Metrics Platform/Architecture#EPIC (fka MPIC) - Experiment Platform Instrument Configurator]] 2314929 wikitext text/x-wiki c6ynmsvt6m1ofu4w13mzrizopkoxej9 0 458870 2315011 2025-06-19T01:07:21Z Neil Shah-Quinn (WMF) 36601 Neil Shah-Quinn (WMF) moved page [[MariaDB/monitoring]] to [[MariaDB/Monitoring]]: Use sentence case 2315011 wikitext text/x-wiki 0e84ufawitdvvghz0kr0elkzsv0hqei 0 458871 2315013 2025-06-19T01:15:16Z Neil Shah-Quinn (WMF) 36601 Move content from "admin guides" section of [[MariaDB]] 2315013 wikitext text/x-wiki 6s2mamrkonpsphpgxj9qujz8beygso3 2315017 2315013 2025-06-19T01:22:11Z Neil Shah-Quinn (WMF) 36601 Add infrastructure navbox, SRE Data Persistence footer, and MariaDB category 2315017 wikitext text/x-wiki 2llx50ho4q3qmypp2ad5ppnb3486aqx 12 458872 2315091 2025-06-19T08:03:35Z Taavi-WMF 41365 Taavi-WMF moved page [[Help:Toolforge/Jobs Service]] to [[Help:Toolforge/Running jobs]]: user-facing pages are named after functionality they provide, not the technical implementation 2315091 wikitext text/x-wiki akgz2xdpbtp0lb31eor7znr54obpcfx 13 458873 2315093 2025-06-19T08:03:35Z Taavi-WMF 41365 Taavi-WMF moved page [[Help talk:Toolforge/Jobs Service]] to [[Help talk:Toolforge/Running jobs]]: user-facing pages are named after functionality they provide, not the technical implementation 2315093 wikitext text/x-wiki hnussb12cmr5tv0g83v6vrg7hqfcxih 13 458874 2315095 2025-06-19T08:03:35Z Taavi-WMF 41365 Taavi-WMF moved page [[Help talk:Toolforge/Jobs Service/Archives/2022]] to [[Help talk:Toolforge/Running jobs/Archives/2022]]: user-facing pages are named after functionality they provide, not the technical implementation 2315095 wikitext text/x-wiki 99klmijw84ls3znc0pf01fcc4curcxa 13 458875 2315097 2025-06-19T08:03:36Z Taavi-WMF 41365 Taavi-WMF moved page [[Help talk:Toolforge/Jobs Service/Archives/2023]] to [[Help talk:Toolforge/Running jobs/Archives/2023]]: user-facing pages are named after functionality they provide, not the technical implementation 2315097 wikitext text/x-wiki gtjqw0xprn4hx68mfdysyy9n7owgljz 13 458876 2315099 2025-06-19T08:03:36Z Taavi-WMF 41365 Taavi-WMF moved page [[Help talk:Toolforge/Jobs Service/Archives/2024]] to [[Help talk:Toolforge/Running jobs/Archives/2024]]: user-facing pages are named after functionality they provide, not the technical implementation 2315099 wikitext text/x-wiki 7wzs0cjkq92867trnjht7zqmfbrtxqt 13 458877 2315101 2025-06-19T08:03:36Z Taavi-WMF 41365 Taavi-WMF moved page [[Help talk:Toolforge/Jobs Service/Archives/2025]] to [[Help talk:Toolforge/Running jobs/Archives/2025]]: user-facing pages are named after functionality they provide, not the technical implementation 2315101 wikitext text/x-wiki 3iihbx511vgnl0kitimiq0fcnmeda45 498 458878 2315158 2025-06-19T09:04:19Z Labslogbot 55 Auto update of instance info. 2315158 wikitext text/x-wiki henzd4qx0uep5na32mbszaesniub20h 498 458879 2315159 2025-06-19T09:06:22Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project toolsbeta-logging in eqiad1 (T397339) 2315159 wikitext text/x-wiki stc9dbo0jyurveohmb343dwqdcpom69 2315160 2315159 2025-06-19T09:08:29Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.create_project for project toolsbeta-logging in eqiad1 (T397339) 2315160 wikitext text/x-wiki 0spyxe47vzozw9x4cgd0y7maomac88x 2315162 2315160 2025-06-19T09:09:31Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project toolsbeta-logging in eqiad1 (T397339) 2315162 wikitext text/x-wiki rg4qojj57f238gcer7vjkxvow7blcy1 2315191 2315162 2025-06-19T09:34:18Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.create_project for project toolsbeta-logging in eqiad1 (T397339) 2315191 wikitext text/x-wiki 5b6llzkagbheyi9uscz8eiijo7exhzb 2315193 2315191 2025-06-19T09:36:15Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project toolsbeta-logging in eqiad1 (T397339) 2315193 wikitext text/x-wiki 7xdpcnyusiwg8754s1d10cfz0v0ki37 498 458880 2315212 2025-06-19T09:46:15Z Stashbot 7414 wmbot~nokibsarkar@tools-bastion-13: Test 2315212 wikitext text/x-wiki ahopftjzolaqcnya0iksms6nplpp8rs 12 458881 2315275 2025-06-19T11:20:12Z Effie Mouzeli (WMF) 12880 Created page with "<pre> +---------+---------+-----------------------------------------------------+ | Cipher | Algo | Fingerprint | +---------+---------+-----------------------------------------------------+ | RSA | SHA-256 | SHA256:sztlinbl/mQcuTuvs+F8fcEyNUoxuilYZlCiBpyVg2s | +---------+---------+-----------------------------------------------------+ | ECDSA | SHA-256 | SHA256:Ntz7Is+g9IJlpY7FVi6shANN046TfrsP/8KTnCDAtyI | +-----..." 2315275 wikitext text/x-wiki bpegs5j5zd6qgyzch9uchxe4zq4hdw9 2315276 2315275 2025-06-19T11:22:58Z Effie Mouzeli (WMF) 12880 Protected "[[Help:SSH Fingerprints/mw-experimental.eqiad.wmnet]]" ([Edit=Allow only administrators] (indefinite) [Move=Allow only administrators] (indefinite)) 2315275 wikitext text/x-wiki bpegs5j5zd6qgyzch9uchxe4zq4hdw9 12 458882 2315278 2025-06-19T11:24:05Z Effie Mouzeli (WMF) 12880 Created page with "<pre> +---------+---------+-----------------------------------------------------+ | Cipher | Algo | Fingerprint | +---------+---------+-----------------------------------------------------+ | RSA | SHA-256 | SHA256:QzL1fnGa3qlpITMgj6mBHPW4KBtO6kWA7Q4R85rdDWM | +---------+---------+-----------------------------------------------------+ | ECDSA | SHA-256 | SHA256:ExIQ6dMwuHx3j2cYaoPa4gD76JSIl9kaduXGL1OGhEs | +-----..." 2315278 wikitext text/x-wiki 49nbss8b4fdre9njw3a3it47pozwttf 2315279 2315278 2025-06-19T11:24:20Z Effie Mouzeli (WMF) 12880 Protected "[[Help:SSH Fingerprints/mw-experimental.codfw.wmnet]]" ([Edit=Allow only administrators] (indefinite) [Move=Allow only administrators] (indefinite)) 2315278 wikitext text/x-wiki 49nbss8b4fdre9njw3a3it47pozwttf