Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.45.0-wmf.6 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 1516 2315780 2305517 2025-06-20T20:48:29Z Stashbot 7414 AntiComposite: cvn-app10: restart all bots 2315780 wikitext text/x-wiki dl9uxd92d63hrmc3oy9u1k7qldg20a3 2315781 2315780 2025-06-20T20:49:52Z Stashbot 7414 AntiComposite: cvn-app12: restart all bots 2315781 wikitext text/x-wiki ct6ihl49zk8pjwc7t7gw6lw93klz6hj 498 3086 2315797 2315481 2025-06-21T09:26:57Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console 2315797 wikitext text/x-wiki r7vq8dcahpkjkqli3tm85pfu4ueadcw 2315798 2315797 2025-06-21T09:27:02Z Stashbot 7414 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) 2315798 wikitext text/x-wiki 4oqg93pvd5vug5nsec12uuncrz3yu0i 2315799 2315798 2025-06-21T09:27:28Z Stashbot 7414 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console 2315799 wikitext text/x-wiki 8m9e6vu1ihhsoisbb8ydpelqz22hvfs 2315800 2315799 2025-06-21T10:09:34Z Stashbot 7414 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) 2315800 wikitext text/x-wiki p7riihh25p7tylk7f2v29nnohhk4rr3 0 3850 2315783 2255318 2025-06-20T20:52:20Z Novem Linguae 23682 /* Encryption */ remove GPG. is no longer in the software 2315783 wikitext text/x-wiki 6obzxz2yo706ox5ha34nceam4mka633 0 4108 2315688 2315594 2025-06-20T13:40:32Z ScheduleDeploymentBot 37566 Add [[gerrit:1161950]] to Monday, June 23 UTC afternoon backport window 2315688 wikitext text/x-wiki 0zqzospwq5c4p979wgmsc2zfjpcz5b4 2315790 2315688 2025-06-21T02:00:19Z ScheduleDeploymentBot 37566 Add [[gerrit:1162158]] to Monday, June 23 UTC late backport window 2315790 wikitext text/x-wiki f9xkirh99kau8pua6rl2l83hi0i57hv 2315791 2315790 2025-06-21T02:00:23Z DeploymentCalendarTool 20896 Remove Week of June 16 2315791 wikitext text/x-wiki 49slczqku9vakmcvaw33jliz2cb5ryi 2315796 2315791 2025-06-21T07:03:18Z A smart kitten 38172 /* {{Deployment_day|date=2025-06-23}} */ looks like the calendar tool edit-conflicted with the scheduling bot 2315796 wikitext text/x-wiki nl5680kkj15l0qz7m0aw6vca4qscuaq 0 7919 2315650 2315649 2025-06-20T11:59:06Z Stashbot 7414 jelto: import kubernetes 1.23.14-6 and 1.31.4-5 to apt host - T387548 2315650 wikitext text/x-wiki r5jro9qf2bizgbuthle8ftqvj62rb7g 2315651 2315650 2025-06-20T12:11:39Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P78538 and previous config saved to /var/cache/conftool/dbconfig/20250620-121136-marostegui.json 2315651 wikitext text/x-wiki 5gcign1le9zk7dbgxqkl2omn8jmkmpm 2315652 2315651 2025-06-20T12:18:55Z Stashbot 7414 slyngshede@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary 2315652 wikitext text/x-wiki 8uc7qv833j08e6muo57kgjml8rgasrx 2315653 2315652 2025-06-20T12:19:16Z Stashbot 7414 slyngshede@cumin1002: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary 2315653 wikitext text/x-wiki jd7kxkq61qdxhkka6lb8ill0crke2fu 2315654 2315653 2025-06-20T12:26:46Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P78539 and previous config saved to /var/cache/conftool/dbconfig/20250620-122644-marostegui.json 2315654 wikitext text/x-wiki dqf7q4bozji8jmtpumrv26i6hrjj5iy 2315655 2315654 2025-06-20T12:31:04Z Stashbot 7414 jakob@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply 2315655 wikitext text/x-wiki 941lxza4d5n8zmjvc8gou4lzx1pdh38 2315656 2315655 2025-06-20T12:31:14Z Stashbot 7414 jakob@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply 2315656 wikitext text/x-wiki dmupa41edyplsx2c9t9qfojqki5jtj4 2315657 2315656 2025-06-20T12:31:47Z Stashbot 7414 jakob@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply 2315657 wikitext text/x-wiki 0dq9cf2ee9cv6xnz11llvsr4zv9vp16 2315658 2315657 2025-06-20T12:32:04Z Stashbot 7414 jakob@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply 2315658 wikitext text/x-wiki krnlw3g10fsx33ci6rdqnwevzd4empt 2315659 2315658 2025-06-20T12:32:22Z Stashbot 7414 jakob@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply 2315659 wikitext text/x-wiki 3uhql4itbhmej7mn0rleldfbz3xk8gl 2315660 2315659 2025-06-20T12:32:36Z Stashbot 7414 jakob@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply 2315660 wikitext text/x-wiki avd6oss2ewqm3irubtet9addyofnp9d 2315661 2315660 2025-06-20T12:41:55Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2237 (T396130)', diff saved to https://phabricator.wikimedia.org/P78540 and previous config saved to /var/cache/conftool/dbconfig/20250620-124151-marostegui.json 2315661 wikitext text/x-wiki hwfepl73447qxjw597jbuh0jwv4m1e4 2315662 2315661 2025-06-20T12:42:08Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2239.codfw.wmnet with reason: Maintenance 2315662 wikitext text/x-wiki lir1ia4tdt0b22vb8v6aciidcftehuk 2315663 2315662 2025-06-20T12:54:19Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2240.codfw.wmnet with reason: Maintenance 2315663 wikitext text/x-wiki k4naqha7spwzbqad91o140hg7csv54a 2315664 2315663 2025-06-20T12:58:05Z Stashbot 7414 vgutierrez: upload liberica 0.21 to apt.wm.o (bookworm-wikimedia) 2315664 wikitext text/x-wiki nv14gqa19ihayfqrzybf1w05d5gr87p 2315665 2315664 2025-06-20T12:59:04Z Stashbot 7414 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.upgrade upgradeing P{lvs1013.eqiad.wmnet} and A:liberica 2315665 wikitext text/x-wiki 9u63swokd2ybka1kpuyn1mtl4ckxcov 2315666 2315665 2025-06-20T12:59:35Z Stashbot 7414 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgradeing P{lvs1013.eqiad.wmnet} and A:liberica 2315666 wikitext text/x-wiki 5rlgq8hksfpy7a5c6krufk0hq84nbv2 2315669 2315666 2025-06-20T13:04:27Z Stashbot 7414 fceratto@cumin1002: dbctl commit (dc=all): 'Pool in API for db1252 - see T385141', diff saved to https://phabricator.wikimedia.org/P78541 and previous config saved to /var/cache/conftool/dbconfig/20250620-130423-fceratto.json 2315669 wikitext text/x-wiki mopk7ndngeyui88s75bea8vfjeu4lyb 2315672 2315669 2025-06-20T13:07:20Z Stashbot 7414 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.upgrade upgradeing P{lvs7002.magru.wmnet} and A:liberica 2315672 wikitext text/x-wiki k3ealx6lqcxcgunlo9qt1h74y4ec8nr 2315673 2315672 2025-06-20T13:08:08Z Stashbot 7414 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgradeing P{lvs7002.magru.wmnet} and A:liberica 2315673 wikitext text/x-wiki oss10yufx7q0otwi0f0f12eb3pfg07c 2315674 2315673 2025-06-20T13:23:20Z Stashbot 7414 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.upgrade upgradeing P{lvs7001.magru.wmnet} and A:liberica 2315674 wikitext text/x-wiki ed6r9w57rp7cyapnks7j5y97kfw8nwj 2315675 2315674 2025-06-20T13:23:51Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host aux-k8s-worker1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315675 wikitext text/x-wiki 7jggyxerbdurhgttop554uo67er7vw5 2315676 2315675 2025-06-20T13:23:57Z Stashbot 7414 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgradeing P{lvs7001.magru.wmnet} and A:liberica 2315676 wikitext text/x-wiki e07ku47oq9c95j7cpf6g9xr3arbhati 2315677 2315676 2025-06-20T13:24:53Z Stashbot 7414 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.upgrade upgradeing P{lvs[4008-4009].ulsfo.wmnet} and A:liberica 2315677 wikitext text/x-wiki nzslpp6h3dav1ehaov7vnf16sbqe3xe 2315678 2315677 2025-06-20T13:26:22Z Stashbot 7414 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgradeing P{lvs[4008-4009].ulsfo.wmnet} and A:liberica 2315678 wikitext text/x-wiki 2loperjfie7sh8y4cupejrg1yocwk3f 2315679 2315678 2025-06-20T13:27:50Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host aux-k8s-worker1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315679 wikitext text/x-wiki j0wro4i0qyo0zmmw1scazatebd9ttq8 2315680 2315679 2025-06-20T13:28:00Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host aux-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315680 wikitext text/x-wiki 6u2yd1zowh83int6uvrzwsb1s7zdafq 2315681 2315680 2025-06-20T13:33:15Z Stashbot 7414 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.upgrade upgradeing P{lvs[5004-5005].eqsin.wmnet} and A:liberica 2315681 wikitext text/x-wiki 11jhywto4nz8lvrfbddk1n0un74ruxk 2315682 2315681 2025-06-20T13:35:05Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aux-k8s-worker1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315682 wikitext text/x-wiki cx8gygkvkp148fzds07apjbqj09yp4d 2315683 2315682 2025-06-20T13:36:17Z Stashbot 7414 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgradeing P{lvs[5004-5005].eqsin.wmnet} and A:liberica 2315683 wikitext text/x-wiki bweqc7xiuf705a0ttjle7z2mzgmte09 2315684 2315683 2025-06-20T13:38:03Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host aux-k8s-worker1006.eqiad.wmnet with OS bookworm 2315684 wikitext text/x-wiki 8nqbxfzi3grek8fr9koc5zoumalknc5 2315685 2315684 2025-06-20T13:39:03Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aux-k8s-worker1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315685 wikitext text/x-wiki 0qx92i9ykvk32k9crgva827yk3zjukh 2315686 2315685 2025-06-20T13:39:18Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aux-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315686 wikitext text/x-wiki 2u53qi38ewdcdtjp79q5ms0yj686de5 2315687 2315686 2025-06-20T13:39:26Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host aux-k8s-worker1007.eqiad.wmnet with OS bookworm 2315687 wikitext text/x-wiki g5sdqo8sncmjlbo2qjiy4q59vxlmw54 2315689 2315687 2025-06-20T13:40:55Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host aux-k8s-worker1009.eqiad.wmnet with OS bookworm 2315689 wikitext text/x-wiki b1betc7jd1720uv8iovrg9b3sabue5k 2315692 2315689 2025-06-20T14:16:17Z Stashbot 7414 logmsgbot: dreamyjazz Deployed security patch for T397221 2315692 wikitext text/x-wiki a77l7r523c6awf8h9s0d1dzjvzk27dy 2315693 2315692 2025-06-20T14:21:51Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host aux-k8s-worker1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315693 wikitext text/x-wiki 87nifwh0dmjt8ul8oi5yjhi0f0j531k 2315694 2315693 2025-06-20T14:21:55Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host aux-k8s-worker1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315694 wikitext text/x-wiki a24t7zthgzoke58txhob2mk2ueox8sj 2315695 2315694 2025-06-20T14:21:59Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host aux-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315695 wikitext text/x-wiki gpy25uy0v6grvqu1y5h7hmq8azjtsv4 2315696 2315695 2025-06-20T14:23:18Z Stashbot 7414 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aux-k8s-worker1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315696 wikitext text/x-wiki gykve7u4dvbr81bsiw7il7dakgce1zk 2315697 2315696 2025-06-20T14:25:23Z Stashbot 7414 inflatador: bking@cumin2002:~$ sudo cumin prometheus1007* 'run-puppet-agent' T357146 2315697 wikitext text/x-wiki ajwdm5hancs1rjrqo7wc0wzkq9dq7f6 2315698 2315697 2025-06-20T14:25:54Z Stashbot 7414 logmsgbot: dreamyjazz Deployed security patch for T397221 2315698 wikitext text/x-wiki 560r9aangkvnreblrp0pc6awa3591rc 2315699 2315698 2025-06-20T14:26:52Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host aux-k8s-worker1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315699 wikitext text/x-wiki b6vmz8jse9trvglruhblwmpseseex8i 2315701 2315699 2025-06-20T14:29:11Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aux-k8s-worker1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315701 wikitext text/x-wiki bmo0kke3ubt6ma1hnx61nwunj3pgki8 2315702 2315701 2025-06-20T14:29:18Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aux-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315702 wikitext text/x-wiki e9l5dwupbjjcuyiu5tibg2xizs8l2dh 2315706 2315702 2025-06-20T14:34:01Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aux-k8s-worker1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315706 wikitext text/x-wiki 79wdlnsplvxmw7kjo0cq43msmtuq66n 2315708 2315706 2025-06-20T14:41:35Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host aux-k8s-worker1006.eqiad.wmnet with OS bookworm 2315708 wikitext text/x-wiki s94w4h9ccrif0kh3gi4ip5o5ffjv8i8 2315709 2315708 2025-06-20T14:41:43Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host aux-k8s-worker1009.eqiad.wmnet with OS bookworm 2315709 wikitext text/x-wiki 48qsuhcfgzg69x2uuavy4xm3i1kscz3 2315710 2315709 2025-06-20T14:41:47Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host aux-k8s-worker1007.eqiad.wmnet with OS bookworm 2315710 wikitext text/x-wiki 2e6o9lcnapq7a11vh676er2p5vuujt2 2315712 2315710 2025-06-20T14:53:29Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-worker1009.eqiad.wmnet with reason: host reimage 2315712 wikitext text/x-wiki 4nhp9xqsd6s2q5q9ppwzy2hzdqv0tc5 2315713 2315712 2025-06-20T14:53:41Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-worker1006.eqiad.wmnet with reason: host reimage 2315713 wikitext text/x-wiki 1ploxsqpo55bze51yscboia715iba6l 2315714 2315713 2025-06-20T14:53:45Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-worker1007.eqiad.wmnet with reason: host reimage 2315714 wikitext text/x-wiki octozwh8l1nh6c58xbdh5pju0ihgvdt 2315716 2315714 2025-06-20T14:57:11Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-worker1009.eqiad.wmnet with reason: host reimage 2315716 wikitext text/x-wiki d8qslvhkgjraqo432p4yyncy1uijaz3 2315717 2315716 2025-06-20T15:01:33Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-worker1006.eqiad.wmnet with reason: host reimage 2315717 wikitext text/x-wiki 6kn4r0iut2bt7u9oy6flt83cttdegse 2315718 2315717 2025-06-20T15:05:09Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-worker1007.eqiad.wmnet with reason: host reimage 2315718 wikitext text/x-wiki 7bil7u2q53kdm1otgo4xctknyp4iirv 2315720 2315718 2025-06-20T15:13:13Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" 2315720 wikitext text/x-wiki klw682gw4id5aa7yn3umf3yrhon0fqh 2315721 2315720 2025-06-20T15:13:31Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" 2315721 wikitext text/x-wiki kjpyosfhepbmygfk6gng12sgwzdubo1 2315722 2315721 2025-06-20T15:13:34Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-worker1009.eqiad.wmnet with OS bookworm 2315722 wikitext text/x-wiki 3icbjc0jtkbbhtuu2cml1scpwl0bt85 2315723 2315722 2025-06-20T15:16:22Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" 2315723 wikitext text/x-wiki sjcvp78sbpst28eppj42owp8nmlpzmk 2315724 2315723 2025-06-20T15:16:39Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" 2315724 wikitext text/x-wiki 1drry4j4r6sjslvqag3juqc3vdsz4fr 2315725 2315724 2025-06-20T15:16:43Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-worker1006.eqiad.wmnet with OS bookworm 2315725 wikitext text/x-wiki bblx1e63hfj3aixp71izagjokxtjk42 2315726 2315725 2025-06-20T15:20:15Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" 2315726 wikitext text/x-wiki e8kaovp02jnbx2lar4beco1cbknqn1c 2315727 2315726 2025-06-20T15:22:41Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" 2315727 wikitext text/x-wiki o82kx6jrcuj0ae94opan4vuk585fq33 2315728 2315727 2025-06-20T15:22:45Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-worker1007.eqiad.wmnet with OS bookworm 2315728 wikitext text/x-wiki lid0cr30h6r3yxda1826fdiit5qj657 2315729 2315728 2025-06-20T15:23:02Z Stashbot 7414 fceratto@cumin1002: START - Cookbook sre.hosts.remove-downtime for es2045.codfw.wmnet 2315729 wikitext text/x-wiki riqhcoanpmmejqco9lqupnmicnm4i2b 2315730 2315729 2025-06-20T15:23:05Z Stashbot 7414 fceratto@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for es2045.codfw.wmnet 2315730 wikitext text/x-wiki at735e1sbmt19yvrgn3psf82bj776t4 2315731 2315730 2025-06-20T15:31:09Z Stashbot 7414 fceratto@cumin1002: START - Cookbook sre.mysql.pool es2045* slowly with 10 steps - Pooling in slowly 2315731 wikitext text/x-wiki gvvmdgvy9armpnb98zakln7mozivjsm 2315732 2315731 2025-06-20T15:43:30Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host aux-k8s-worker1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315732 wikitext text/x-wiki 9nn1lbogf9eqm6pfe22nnuemcmyul66 2315733 2315732 2025-06-20T15:47:46Z Stashbot 7414 bking@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply 2315733 wikitext text/x-wiki 5dcuvftc3x6t3rp5wb4k5o48c8wdtyo 2315734 2315733 2025-06-20T15:48:21Z Stashbot 7414 bking@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply 2315734 wikitext text/x-wiki caaltkyx6ketuc0kjquuy48yakd89te 2315735 2315734 2025-06-20T15:48:56Z Stashbot 7414 bking@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply 2315735 wikitext text/x-wiki nivu3tv4xvwb6drlc6g21lriunjscal 2315736 2315735 2025-06-20T15:49:04Z Stashbot 7414 bking@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply 2315736 wikitext text/x-wiki 2vuememw0x9wsj44wlo9kjcprsgto5d 2315737 2315736 2025-06-20T15:49:58Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.181.0" for 2 host(s) 2315737 wikitext text/x-wiki s7q9xwf055kzsnp2chs46fa4yse1d2l 2315738 2315737 2025-06-20T15:51:53Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.181.0" completed for 2 hosts 2315738 wikitext text/x-wiki 01uc02wdlitr0pdj2bqp7nvatxdll0e 2315739 2315738 2025-06-20T15:54:55Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aux-k8s-worker1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315739 wikitext text/x-wiki rp5efj2oelaulvz9mhugmxhgnhq9oyv 2315740 2315739 2025-06-20T15:55:15Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host aux-k8s-worker1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315740 wikitext text/x-wiki qe2wywfr08gby4mg4ls33ule3gy04ot 2315741 2315740 2025-06-20T16:01:47Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aux-k8s-worker1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315741 wikitext text/x-wiki 26r47kukt362p5s34chi6fdbgvuvlel 2315742 2315741 2025-06-20T16:02:12Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host aux-k8s-worker1008.eqiad.wmnet with OS bookworm 2315742 wikitext text/x-wiki 8myiinud0b4fehvimtmyjaj2tszflza 2315745 2315742 2025-06-20T17:01:02Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host aux-k8s-worker1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315745 wikitext text/x-wiki 0ch5o6go5rgs1deu4r6c422zqpgk2lz 2315746 2315745 2025-06-20T17:06:12Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aux-k8s-worker1008.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2315746 wikitext text/x-wiki 748pdwn0o628y0040cbg7umk9n30j04 2315748 2315746 2025-06-20T17:21:13Z Stashbot 7414 aokoth@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply 2315748 wikitext text/x-wiki prfbfiq5yq5a3wruxdglcbv1ta2affo 2315749 2315748 2025-06-20T17:21:25Z Stashbot 7414 aokoth@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply 2315749 wikitext text/x-wiki 18s1e21kywbl8yv7vyaf89obk41y393 2315750 2315749 2025-06-20T17:25:39Z Stashbot 7414 aokoth@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply 2315750 wikitext text/x-wiki hlfr1wo9enu2j84q9trvrqhbofls0nt 2315751 2315750 2025-06-20T17:35:49Z Stashbot 7414 aokoth@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply 2315751 wikitext text/x-wiki tu4cvisc59vo3znasaz14vua9lhxubi 2315752 2315751 2025-06-20T17:47:09Z Stashbot 7414 fceratto@cumin1002: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) es2045* slowly with 10 steps - Pooling in slowly 2315752 wikitext text/x-wiki 2upg7sw1hu0g3recygn9xiaisekt05e 2315753 2315752 2025-06-20T17:47:38Z Stashbot 7414 aokoth@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply 2315753 wikitext text/x-wiki 949mnspmrqi2gk9c8ynzbw7u6kj0ss9 2315754 2315753 2025-06-20T17:49:43Z Stashbot 7414 gmodena@deploy1003: helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply 2315754 wikitext text/x-wiki 7k2hlhd9ah6btu9fwm9dwdc2749ff60 2315755 2315754 2025-06-20T17:49:47Z Stashbot 7414 gmodena@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply 2315755 wikitext text/x-wiki hm351oqm1pmtdps0p3kl5276v4mr8ha 2315756 2315755 2025-06-20T17:53:32Z Stashbot 7414 gmodena@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply 2315756 wikitext text/x-wiki e03awlpfjqlf97xac8tm92zyqcca942 2315757 2315756 2025-06-20T17:53:37Z Stashbot 7414 gmodena@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply 2315757 wikitext text/x-wiki 1g5za57sqjfnzfq1duqr3hblrjv0vvp 2315758 2315757 2025-06-20T17:55:51Z Stashbot 7414 gmodena@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply 2315758 wikitext text/x-wiki qvktg3hnn0orqp5i557789t8rkk24om 2315759 2315758 2025-06-20T17:55:55Z Stashbot 7414 gmodena@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply 2315759 wikitext text/x-wiki orv28392ksun1a3rp4by0k858zb40op 2315760 2315759 2025-06-20T18:00:25Z Stashbot 7414 aokoth@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply 2315760 wikitext text/x-wiki dt80pw23vzby7zl84xhuo20z5ctqbp1 2315761 2315760 2025-06-20T18:05:11Z Stashbot 7414 aokoth@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply 2315761 wikitext text/x-wiki c8cqx8bfs72kdpkay4d0n1gvzc9dmlc 2315762 2315761 2025-06-20T18:05:47Z Stashbot 7414 aokoth@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply 2315762 wikitext text/x-wiki mqs4m4e9ph3iydhqx4rmzgwcihirlol 2315763 2315762 2025-06-20T18:08:23Z Stashbot 7414 aokoth@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply 2315763 wikitext text/x-wiki qhpipk40emx770ze5qoub0yh08uflh5 2315765 2315763 2025-06-20T18:19:55Z Stashbot 7414 aokoth@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply 2315765 wikitext text/x-wiki 0ihyhosnapw13y0rb2jjw9nqw0gtaqi 2315766 2315765 2025-06-20T18:30:58Z Stashbot 7414 aokoth@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply 2315766 wikitext text/x-wiki 1iunismgzgi42rszlwgg88ma81ipzef 2315767 2315766 2025-06-20T18:32:25Z Stashbot 7414 aokoth@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply 2315767 wikitext text/x-wiki 95b5pdgxjiandxxhxoxzgheyz5z0kz3 2315768 2315767 2025-06-20T18:33:50Z Stashbot 7414 aokoth@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply 2315768 wikitext text/x-wiki raajunq9u020i2qwydizrgj64rjh3od 2315769 2315768 2025-06-20T18:34:23Z Stashbot 7414 aokoth@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply 2315769 wikitext text/x-wiki 6tsv5xmro2muu9yrge2slyirs507o7m 2315770 2315769 2025-06-20T18:36:15Z Stashbot 7414 bking@cumin2002: conftool action : set/weight=10:pooled=no; selector: name=cirrussearch2113\.codfw\.wmnet 2315770 wikitext text/x-wiki 6s3pi8ette9wkem2nffvk155qo6emu5 2315773 2315770 2025-06-20T19:10:06Z Stashbot 7414 sukhe: sudo cumin 'A:cp' "disable-puppet 'merging CR 1160381'": T390924 2315773 wikitext text/x-wiki 9qeklm4cm1hixq0yz3unyr119utqp95 2315774 2315773 2025-06-20T19:16:11Z Stashbot 7414 sukhe: enabling puppet on cp4037 to merge CR 1160381: add `ismobile=1' for mobile requests: T390924 2315774 wikitext text/x-wiki pry5lr26vyrs1yy65scnmoos9mqdc2y 2315775 2315774 2025-06-20T19:19:08Z Stashbot 7414 sukhe: sudo cumin -b11 'A:cp' "run-puppet-agent 'merging CR 1160381'": T390924 2315775 wikitext text/x-wiki nvotpxgl5y6jo5r4m4hc3dl05i8zkur 2315776 2315775 2025-06-20T19:19:24Z Stashbot 7414 sukhe: sudo cumin -b11 'A:cp' "run-puppet-agent --enable 'merging CR 1160381'": T390924 2315776 wikitext text/x-wiki g5cnh880ld5z9sep6epf2x8eky61ivj 2315785 2315776 2025-06-20T21:14:17Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply 2315785 wikitext text/x-wiki 8wh9vx0owvgxkuri09r8he1slsv1uct 2315786 2315785 2025-06-20T21:14:44Z Stashbot 7414 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply 2315786 wikitext text/x-wiki f4xqqhdf6f29o7f78yoaqk81cme9bil 0 17290 2315764 2315196 2025-06-20T18:18:20Z Stashbot 7414 dduvall: seeing numerous image pull errors in gitlab-cloud-runner cluster 2315764 wikitext text/x-wiki 63ergx9o2844lzzh0vwdqonprzfhibq 2315771 2315764 2025-06-20T18:41:07Z Stashbot 7414 dduvall: deleted docker-hub-mirror statefulset and admission controller deployment. reapplying production cluster configuration 2315771 wikitext text/x-wiki gjwcqkqzc5tz1d4ff5flrys3r0joytg 2315772 2315771 2025-06-20T18:57:04Z Stashbot 7414 dduvall: ran `helm --namespace gitlab-runner uninstall docker-hub-mirror` to fix helm state. reapplying production cluster configuration 2315772 wikitext text/x-wiki mow7tarnk4pkvn7qq7byy25ioai2hah 2315793 2315772 2025-06-21T02:54:10Z Stashbot 7414 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1162106 2315793 wikitext text/x-wiki 94k2j6p63w05cdz78kjos5udokrf723 498 30942 2315747 2315022 2025-06-20T17:20:03Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph}' 2315747 wikitext text/x-wiki cilh6zuoabopfh46m0ygy3utkund7t5 2315782 2315747 2025-06-20T20:51:28Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova,cinder,neutron 2315782 wikitext text/x-wiki 66hcyq384c6xcq7lvkmpfe8x9t27u0j 2315784 2315782 2025-06-20T20:58:50Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,nova,cinder,neutron 2315784 wikitext text/x-wiki 179hsl3nagmjb1qazq7y0tepuj34l2f 2315794 2315784 2025-06-21T03:08:37Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services 2315794 wikitext text/x-wiki iby1tv0vvaomi9sxv9mgod4pkanvft4 2315795 2315794 2025-06-21T03:17:02Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services 2315795 wikitext text/x-wiki ovqzbve2tsjdskyl5b91803yb8bt74z 498 73236 2315779 2303058 2025-06-20T20:34:11Z Stashbot 7414 wmbot~anticomposite@tools-bastion-13: stewardbots/StewardBot/manage.sh restart # Pinged out 2315779 wikitext text/x-wiki 67t4xxwbm9sasrlsj782qqpedss7i1p 0 441451 2315700 2249499 2025-06-20T14:28:52Z Andrewbogott 33 /* Initial designate/pdns node setup */ 2315700 wikitext text/x-wiki ei7q16cwntrgfrnt3odq2vvj58ajgi0 2315703 2315700 2025-06-20T14:30:03Z Andrewbogott 33 /* Initial designate/pdns node setup */ 2315703 wikitext text/x-wiki 2fg3f70ae8tixwi0vzkbx422s85p0c1 2315704 2315703 2025-06-20T14:32:30Z Andrewbogott 33 /* Maintenance operations */ 2315704 wikitext text/x-wiki 0wfafww4x5sylfb88klbd3edg7cvji3 2315715 2315704 2025-06-20T14:55:46Z Andrewbogott 33 /* Initial designate/pdns node setup */ 2315715 wikitext text/x-wiki fc99vr1j3w00hngrxzpykjr96pq6fg6 2315719 2315715 2025-06-20T15:07:03Z Andrewbogott 33 /* Initial designate/pdns node setup */ 2315719 wikitext text/x-wiki mik9tj38i1f9t5jtz3xycxuz8dyitku 0 447921 2315691 2174450 2025-06-20T13:56:42Z FCeratto-WMF 39104 Fix typo (I asked cdanis) 2315691 wikitext text/x-wiki pb0qjo794rrk3w55xfgapsa0eavkkvk 0 449160 2315671 2315586 2025-06-20T13:06:18Z Dexbot 30554 Bot: Updating the report 2315671 wikitext text/x-wiki t3lx64ras8azuc8ixhdfm3ori5jt4tq 2315788 2315671 2025-06-21T00:02:09Z Dexbot 30554 Bot: Updating the report 2315788 wikitext text/x-wiki pp4i60prgxmr9xbiog0km7x6f58msct 116 453906 2315789 2314654 2025-06-21T00:48:23Z Gitlabaccountapprovalbot 37332 @vriaa was approved. 2315789 wikitext text/x-wiki p6wwqb52vt3ihbqcd5q4wl7svp1ckeb 2 454463 2315667 2298631 2025-06-20T13:03:37Z FFurnari-WMF 36523 2315667 wikitext text/x-wiki pfloxe3u5iwye9gb22j57k68x7oqkwh 2315668 2315667 2025-06-20T13:03:53Z FFurnari-WMF 36523 2315668 wikitext text/x-wiki ct76clzr4hmz1esuyopzhmj6blmbz9v 0 458775 2315777 2308744 2025-06-20T19:57:56Z MPopov (WMF) 5891 /* Register experiment */ 2315777 wikitext text/x-wiki a8jlsqcoxg5zlkr9hqvmzsvpecmcsx0 2315778 2315777 2025-06-20T19:58:26Z MPopov (WMF) 5891 /* Define metric */ 2315778 wikitext text/x-wiki pgs9wi5j2svkpmbbzcph6508hvomgbj 0 458830 2315792 2312759 2025-06-21T02:00:33Z DeploymentCalendarTool 20896 Add last week 2315792 wikitext text/x-wiki ratsoqa02xuzjfn3o202h4dermzzuuj 0 458856 2315670 2315596 2025-06-20T13:04:53Z HNowlan (WMF) 15760 /* Using the service */ correct frequency of timer 2315670 wikitext text/x-wiki r7d3nt5tampkgenfuqx282h7hrn9wr6 2315743 2315670 2025-06-20T16:06:04Z ADancy (WMF) 17694 /* Design */ add missing word 2315743 wikitext text/x-wiki hrnul8y942an819btuokrgnovj7ijfp 0 458857 2315744 2314597 2025-06-20T16:52:29Z MPopov (WMF) 5891 Clarifications and formatting improvements 2315744 wikitext text/x-wiki bs1g5irmns7vgovfdjv3iembpzux4f3 0 458889 2315690 2025-06-20T13:56:12Z FFurnari-WMF 36523 Created page with "=== Step 1 === * In trixie docker container, dependencies: <code>libpcre2-dev libjemalloc-dev python3-sphinx zlib1g-dev build-essential devscripts libssl-dev liblua5.3-dev libsystemd-dev libopentracing-c-wrapper-dev python3-mako cmake systemd-dev</code> <code>golang-go</code> * Compile aws-lc: ** <code>git clone <nowiki>https://github.com/aws/aws-lc.git</nowiki></code> ** <code>cmake -S . -B ./build -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_STANDARD=11 -DCMAKE_INSTALL_PREFI..." 2315690 wikitext text/x-wiki ihb5gj85y7jcpv99no6y59ydrtvxp9m 2315705 2315690 2025-06-20T14:33:48Z FFurnari-WMF 36523 /* Step 2 */ 2315705 wikitext text/x-wiki 5ltjbowocb0bp6pfd0pddtvfct3a0h7 2315707 2315705 2025-06-20T14:34:27Z FFurnari-WMF 36523 /* Step 2 */ 2315707 wikitext text/x-wiki qx9r59velevdyxo4eon8csd57ke1ede 2315711 2315707 2025-06-20T14:43:53Z FFurnari-WMF 36523 /* Step 2 */ 2315711 wikitext text/x-wiki t2o41s31jvuvctnal7sohggfc8jvqmc 3 458890 2315787 2025-06-20T21:33:45Z StrikerBot 8475 Welcome to Toolforge! 2315787 wikitext text/x-wiki a1yk8h5cmf7dvdk2kwy8dbd3mcll16w