Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.2 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 890 2413953 2237795 2026-05-14T16:03:30Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase by 6 cores, 12288 ram 2413953 wikitext text/x-wiki 29ykewdnax5i2lrlul9z8k70acjrgtj 2413955 2413953 2026-05-14T16:03:35Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 6 cores, 12288 ram 2413955 wikitext text/x-wiki t5d1k0gso57juoafsnuxxb5zxcxz8rd 498 3086 2413763 2413339 2026-05-14T13:27:37Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway 2413763 wikitext text/x-wiki 1l1451akjmqkcth9zpfdgvrnl74bx9d 2413766 2413763 2026-05-14T13:30:16Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway 2413766 wikitext text/x-wiki nlu7n4dn0f9wkumua0135xjf98325mv 2413769 2413766 2026-05-14T13:31:27Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway 2413769 wikitext text/x-wiki 95qxwtno6ni95vfogn25e74p5g495da 2413786 2413769 2026-05-14T13:49:35Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway 2413786 wikitext text/x-wiki 2nby6m270a3n4bvjq5vomjt8jb6oisb 2413895 2413786 2026-05-14T14:57:04Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder 2413895 wikitext text/x-wiki a7ggbtp17ekfe8vf88o0wnxno0x917v 2413900 2413895 2026-05-14T15:02:19Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder 2413900 wikitext text/x-wiki 4pn3xvmvcev48k6mya5z1tvr25mtktd 2413911 2413900 2026-05-14T15:16:08Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2413911 wikitext text/x-wiki 9bcqyg9t1d5p8kk8l3e4e4qkvtkgh46 2413916 2413911 2026-05-14T15:28:20Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2413916 wikitext text/x-wiki 3qa9scmw4lj3tdrkrshk3bq6g18rzwh 2413926 2413916 2026-05-14T15:39:06Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission 2413926 wikitext text/x-wiki 52gf4kfsgmevxy0u8hqcf6yn9w3bklo 2413935 2413926 2026-05-14T15:48:21Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission 2413935 wikitext text/x-wiki mamd24d81411pevyhizdrbg9odrn1ms 2413971 2413935 2026-05-14T16:18:53Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers 2413971 wikitext text/x-wiki tsspiq0jpiplklm7ykjreaod9upj21b 2413972 2413971 2026-05-14T16:19:04Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers 2413972 wikitext text/x-wiki dat93tz2acf4hqnstm83ncizhb9j1fa 0 3917 2413825 2390033 2026-05-14T14:09:51Z Krinkle 44 2413825 wikitext text/x-wiki 081m2uoemcj0r5ccuu6bs182x4dn4wr 2413850 2413825 2026-05-14T14:23:42Z Krinkle 44 /* throttle.php */ 2413850 wikitext text/x-wiki 03puiyzel5meo7m7neyjjgbhtdbesjg 2413854 2413850 2026-05-14T14:26:17Z Krinkle 44 /* Reset */ 2413854 wikitext text/x-wiki 4fcxfpuavh1um5xbrax60a0xd1gx6qz 0 4108 2413813 2413651 2026-05-14T14:01:03Z ScheduleDeploymentBot 37566 Add [[gerrit:1287427]] to Thursday, May 14 UTC late backport window 2413813 wikitext text/x-wiki em74nmgjvi2ktdu7htk70uqoyrr7t3f 2413881 2413813 2026-05-14T14:51:51Z ScheduleDeploymentBot 37566 Add [[gerrit:1287433]] to Thursday, May 14 UTC late backport window 2413881 wikitext text/x-wiki dfb3vy1ef53e0e0zxr5udskc81wbkj2 2414028 2413881 2026-05-14T19:24:49Z ScheduleDeploymentBot 37566 Add [[gerrit:1287002]] to Thursday, May 14 UTC late backport window 2414028 wikitext text/x-wiki 6ldzxlx5pgahft9n6x7y10fkufaeqo4 2414081 2414028 2026-05-14T21:19:39Z ScheduleDeploymentBot 37566 Add [[gerrit:1287485]] to Monday, May 18 UTC morning backport window 2414081 wikitext text/x-wiki raajxe15ja00hrhw6v1t65xm8nwak3t 2414082 2414081 2026-05-14T21:21:22Z JDrewniak (WMF) 38238 /* Monday, May 18 */ 2414082 wikitext text/x-wiki 6ldzxlx5pgahft9n6x7y10fkufaeqo4 2414083 2414082 2026-05-14T21:22:02Z JDrewniak (WMF) 38238 /* Thursday, May 14 */ 2414083 wikitext text/x-wiki d1wouv7xwol0hl38r8zk2xqi7agurmn 0 4218 2414098 2412720 2026-05-14T22:14:07Z Novem Linguae 23682 {{Notice|Since the creation of [[Corto]], this page is being used much less frequently. A chronological list of Corto incidents can be found using [https://phabricator.wikimedia.org/dashboard/view/778/ this Phabricator dashboard].}} there's some discussion of this in [[phab:T426137]] 2414098 wikitext text/x-wiki cqu2ee9nfxh4io24wt8belq6juppxq7 2414166 2414098 2026-05-15T08:54:06Z A smart kitten 38172 imo the mailing list link was also useful to have from this notice (at least for right now), @[[user:novem Linguae|Novem]] i hope you don't mind me adding it back :) feel free to revert. [also, changed the link to point to the whole thread (for additional context), rather than just one message in the thread] 2414166 wikitext text/x-wiki quumvfr2khy20q9sc30cef0ng4x25al 0 6479 2413697 2412075 2026-05-14T12:38:58Z Effie Mouzeli (WMF) 12880 2413697 wikitext text/x-wiki n9khb0mb32bi99izoods9y0j8h23k6w 2413699 2413697 2026-05-14T12:40:48Z Effie Mouzeli (WMF) 12880 /* Pair 1 (rdb1013 and rdb2007) */ 2413699 wikitext text/x-wiki 2k5ypt057yxasiu1cbokvducsawzqpf 498 6726 2413883 2413684 2026-05-14T14:52:34Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder 2413883 wikitext text/x-wiki rc6661go2zm2ahharj8fxzu76coa4ig 2413894 2413883 2026-05-14T14:56:42Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder 2413894 wikitext text/x-wiki aexzudwb5hn0007emixzq06sb34k9y4 2413901 2413894 2026-05-14T15:02:59Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2413901 wikitext text/x-wiki pl9jr03iowhvalacc4xf6c1g62e3cl4 2413909 2413901 2026-05-14T15:14:14Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2413909 wikitext text/x-wiki 0pbltbagoanwem3mvliqr5xph0tbj02 2413918 2413909 2026-05-14T15:30:06Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission 2413918 wikitext text/x-wiki i2spukqfncy5jua8h48v6b98yx4cpzw 2413925 2413918 2026-05-14T15:38:16Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission 2413925 wikitext text/x-wiki rea9y9h7um5kncyd4u7ynoe5wvinlbo 0 7919 2413689 2413688 2026-05-14T12:18:42Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc2023 to pc3 T418973', diff saved to https://phabricator.wikimedia.org/P92524 and previous config saved to /var/cache/conftool/dbconfig/20260514-121839-marostegui.json 2413689 wikitext text/x-wiki mdo2vbdty52pg801lljvur34xlkmilq 2413690 2413689 2026-05-14T12:20:01Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Add pc2023 to pc3 codfw master T418973', diff saved to https://phabricator.wikimedia.org/P92525 and previous config saved to /var/cache/conftool/dbconfig/20260514-121958-marostegui.json 2413690 wikitext text/x-wiki elhw890lnnpfq7xltccsqjugmncnil2 2413692 2413690 2026-05-14T12:21:16Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply 2413692 wikitext text/x-wiki kljdlj2tpnj9kn8s9u24e9t29nqelcf 2413693 2413692 2026-05-14T12:21:33Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply 2413693 wikitext text/x-wiki qp7vjjcztocnsjr1g0zyws0nojdw375 2413694 2413693 2026-05-14T12:27:10Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Repool pc3 with pc2023 as codfw master T418973', diff saved to https://phabricator.wikimedia.org/P92526 and previous config saved to /var/cache/conftool/dbconfig/20260514-122707-marostegui.json 2413694 wikitext text/x-wiki qthh5yfocwb0vhgzu32wwjx96uwv8vl 2413695 2413694 2026-05-14T12:27:37Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 28458 2413695 wikitext text/x-wiki 44b8aaqunfx2q6pqaowgouvos37eeb8 2413696 2413695 2026-05-14T12:31:14Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 28458 2413696 wikitext text/x-wiki 5tbtzsnz1vg73pay5kk1v3jwi46dov7 2413698 2413696 2026-05-14T12:40:31Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin1003.eqiad.wmnet with reason: update bgp groups for dse-k8s-wdqs - cmooney@cumin1003 2413698 wikitext text/x-wiki 29eevrhqj2qhzahpq0fa23i6hj5spva 2413700 2413698 2026-05-14T12:42:11Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin1003.eqiad.wmnet with reason: update bgp groups for dse-k8s-wdqs - cmooney@cumin1003 2413700 wikitext text/x-wiki i9wt01zmxn3ov2b1zuqaj2kxgaxvyqt 2413701 2413700 2026-05-14T12:42:56Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2413701 wikitext text/x-wiki 7mjgmabe7eh6f43pohktya5phsu5v2c 2413702 2413701 2026-05-14T12:46:41Z Stashbot 7414 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply 2413702 wikitext text/x-wiki 6xt74i41hj3rrotm4jzr98v2bdzddpj 2413703 2413702 2026-05-14T12:47:04Z Stashbot 7414 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply 2413703 wikitext text/x-wiki mnuvewsvxr59nbc313xzqolc2vdc56y 2413704 2413703 2026-05-14T12:47:24Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1279] - vriley@cumin1003" 2413704 wikitext text/x-wiki 7dqtuq5xwb51td2x8n1d2o84o9ghd7p 2413705 2413704 2026-05-14T12:47:30Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1279] - vriley@cumin1003" 2413705 wikitext text/x-wiki rx83cd8zr08cvk49p55xiki1aeao7dp 2413706 2413705 2026-05-14T12:47:31Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2413706 wikitext text/x-wiki 5oag80y39o8j4zrmjg5dqnupqvk1egs 2413707 2413706 2026-05-14T12:47:56Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1279 2413707 wikitext text/x-wiki nbok6rzoos9lfoofuw1vghg58vkgezd 2413708 2413707 2026-05-14T12:49:29Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2413708 wikitext text/x-wiki mrgwyqawz1u82z1cgxz79qa9hn19esg 2413709 2413708 2026-05-14T12:49:45Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s8 T426291 2413709 wikitext text/x-wiki emz2k5rne8zib353c1relgrfttavvxg 2413710 2413709 2026-05-14T12:49:47Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1279 2413710 wikitext text/x-wiki acp8p4prd9cgyg54gdjc20ptrq45386 2413711 2413710 2026-05-14T12:50:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2161 with weight 0 T426291', diff saved to https://phabricator.wikimedia.org/P92527 and previous config saved to /var/cache/conftool/dbconfig/20260514-125014-fceratto.json 2413711 wikitext text/x-wiki ahhuby147o2n40bfnvvnq31zodzf87o 2413713 2413711 2026-05-14T12:50:35Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1279.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413713 wikitext text/x-wiki svqowkbuh1pzo4z1j49fdz0nly6mgzw 2413714 2413713 2026-05-14T12:53:16Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1280] - vriley@cumin1003" 2413714 wikitext text/x-wiki lud6mkdg6l11z7lpg1egdfsdw54d3ik 2413715 2413714 2026-05-14T12:53:22Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1280] - vriley@cumin1003" 2413715 wikitext text/x-wiki dh08vqjt4rqkvvrwqibnl24jthwz7ds 2413716 2413715 2026-05-14T12:53:23Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2413716 wikitext text/x-wiki snazrp4tie0g27ppi4vzqejl3u4rgko 2413717 2413716 2026-05-14T12:53:40Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1280 2413717 wikitext text/x-wiki h5izx70dibakdgbz48932hjf10iinq9 2413718 2413717 2026-05-14T12:54:55Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1280 2413718 wikitext text/x-wiki r14l0j2avhjpmog8b8jvi52gmye37ai 2413719 2413718 2026-05-14T12:55:17Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2413719 wikitext text/x-wiki eu0qwu5rd60v68s62ros4jwfhg33ykb 2413720 2413719 2026-05-14T12:55:49Z Stashbot 7414 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply 2413720 wikitext text/x-wiki k0jktu6dpsu4iliig9mlv90rwob79by 2413721 2413720 2026-05-14T12:56:15Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1280.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413721 wikitext text/x-wiki rfxfq1r3lhcmwyi18bnxkv45euzmvvt 2413722 2413721 2026-05-14T12:56:24Z Stashbot 7414 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply 2413722 wikitext text/x-wiki j9je8mf03y5yiormew967jl3967xlmv 2413723 2413722 2026-05-14T12:57:41Z Stashbot 7414 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply 2413723 wikitext text/x-wiki bwepoxslgzxdi2z1deefwz4dgceonl5 2413724 2413723 2026-05-14T12:58:15Z Stashbot 7414 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply 2413724 wikitext text/x-wiki pt152x9q52orwg6f6zqcv2jv6vnjzqe 2413725 2413724 2026-05-14T12:59:06Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1281] - vriley@cumin1003" 2413725 wikitext text/x-wiki 8d2bb95406wh57wa2yow26h9argdpl0 2413726 2413725 2026-05-14T12:59:12Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1281] - vriley@cumin1003" 2413726 wikitext text/x-wiki 8j75wqmx8hkyhfvclfwwm88pshjgbjq 2413727 2413726 2026-05-14T12:59:14Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2413727 wikitext text/x-wiki gt5cohisaj9hwrzm00iuttnc9agkcce 2413728 2413727 2026-05-14T12:59:33Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1281 2413728 wikitext text/x-wiki mqw7c0olbevlrg7ao3oxnt6p9vv29tv 2413730 2413728 2026-05-14T13:00:42Z Stashbot 7414 kart_: Updated cxserver to 2026-05-14-123010-production (T426174, T404298) 2413730 wikitext text/x-wiki 97ejamr7pumyuy237qbpzev3237z9k7 2413731 2413730 2026-05-14T13:00:45Z Stashbot 7414 federico3: Starting s8 codfw failover from db2165 to db2161 - T426291 2413731 wikitext text/x-wiki oudymstq8vbrcnjed1fbiecyh2ub6uj 2413732 2413731 2026-05-14T13:01:34Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1281 2413732 wikitext text/x-wiki nyicinv9iw0z6nkee7p0eu2sz5m02dx 2413733 2413732 2026-05-14T13:01:45Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2413733 wikitext text/x-wiki emhcjpffbvwgfnmcsxy92w4wabthvyf 2413734 2413733 2026-05-14T13:02:16Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2161 to s8 primary T426291', diff saved to https://phabricator.wikimedia.org/P92528 and previous config saved to /var/cache/conftool/dbconfig/20260514-130213-fceratto.json 2413734 wikitext text/x-wiki 98uudeqmcmahv39s8r0kbwtaj6f4ehj 2413735 2413734 2026-05-14T13:02:23Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1281.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413735 wikitext text/x-wiki ecmwqh3u0p99wv8jsajnztc2avfhdi4 2413736 2413735 2026-05-14T13:05:20Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1281.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413736 wikitext text/x-wiki 87jfkcnskey2pibb28ss1fdzjmebxh9 2413737 2413736 2026-05-14T13:06:15Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1282] - vriley@cumin1003" 2413737 wikitext text/x-wiki lwcc9el0n3x42kt1mynsbomhtbidjm0 2413738 2413737 2026-05-14T13:06:20Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1282] - vriley@cumin1003" 2413738 wikitext text/x-wiki 5mocyq54zge7q63zetpt3ozzu550vm3 2413739 2413738 2026-05-14T13:06:22Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2413739 wikitext text/x-wiki 0pd4rme66m96qz252tzr6fd1vrhfnhm 2413740 2413739 2026-05-14T13:07:19Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1282 2413740 wikitext text/x-wiki 58ru6uugklm8pveaphenc24fbzt1724 2413741 2413740 2026-05-14T13:07:46Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Set correct weight T426291', diff saved to https://phabricator.wikimedia.org/P92529 and previous config saved to /var/cache/conftool/dbconfig/20260514-130743-fceratto.json 2413741 wikitext text/x-wiki 1gixslwryo0u7f9ieopcfzj1idixwxk 2413742 2413741 2026-05-14T13:08:18Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2165: Repooling after switchover 2413742 wikitext text/x-wiki eltqwm06o7ioi5cfbhwrwj2hamaeh68 2413743 2413742 2026-05-14T13:08:25Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1279.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413743 wikitext text/x-wiki 55jei43c3y7eqvnlmgojljcozxup9ta 2413744 2413743 2026-05-14T13:08:44Z Stashbot 7414 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1287043|Enable the Article Guidance experiment on simplewiki (T426278)]] 2413744 wikitext text/x-wiki 2a94eearefuw5s3geeaxahr24eirhkp 2413745 2413744 2026-05-14T13:08:45Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1282 2413745 wikitext text/x-wiki 5x5hpr3eyq3lux8ntwte7w8ur1gi1yq 2413746 2413745 2026-05-14T13:09:03Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1281.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413746 wikitext text/x-wiki k763ypygbl44ag94s23w6x9zs0frb05 2413747 2413746 2026-05-14T13:10:21Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2165: Repooling after switchover 2413747 wikitext text/x-wiki 59famebob1q1cd0gc6621dv3vvihz81 2413748 2413747 2026-05-14T13:10:29Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2413748 wikitext text/x-wiki f28rt9cxo6c4hav7xgxq30p0yo5kvbg 2413749 2413748 2026-05-14T13:10:35Z Stashbot 7414 sbisson@deploy1003: sbisson: Backport for [[gerrit:1287043|Enable the Article Guidance experiment on simplewiki (T426278)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2413749 wikitext text/x-wiki dv9lsaq87mb9gacdrdn7tffz2gl3zb1 2413750 2413749 2026-05-14T13:12:05Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1282.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413750 wikitext text/x-wiki 7ipu07hbaea0a5o6pp4r0rami4ytv86 2413751 2413750 2026-05-14T13:12:09Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1281.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413751 wikitext text/x-wiki k7l2rxy6fpg6al0n20tyeeee2s82msr 2413752 2413751 2026-05-14T13:12:42Z Stashbot 7414 sbisson@deploy1003: sbisson: Continuing with deployment 2413752 wikitext text/x-wiki k3r225m7qwgmfbp3tjqkzqxoejppy5l 2413753 2413752 2026-05-14T13:14:23Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1280.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413753 wikitext text/x-wiki skxzbxq16zfgj2t1z7w6v0bdokj34tb 2413754 2413753 2026-05-14T13:15:19Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1283] - vriley@cumin1003" 2413754 wikitext text/x-wiki i4yxflyybj3rg0v4l9u1xfx5uuk2h0h 2413755 2413754 2026-05-14T13:15:24Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1283] - vriley@cumin1003" 2413755 wikitext text/x-wiki evmplzih36wukqn6cnxlq99wtpxmmli 2413756 2413755 2026-05-14T13:15:26Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2413756 wikitext text/x-wiki 5fuoexkmf2g8fmh70pzcoy7lcjdtt1r 2413757 2413756 2026-05-14T13:16:54Z Stashbot 7414 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1287043|Enable the Article Guidance experiment on simplewiki (T426278)]] (duration: 08m 10s) 2413757 wikitext text/x-wiki kf9t0473ftbeuzmj8qdnfccibu97qoe 2413758 2413757 2026-05-14T13:18:06Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1283 2413758 wikitext text/x-wiki g7k7ihgw3bnqnkm8txeduw4694s69ng 2413759 2413758 2026-05-14T13:19:08Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1281.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413759 wikitext text/x-wiki las4by5pmmouldg8aza81f8u9gg5hmi 2413760 2413759 2026-05-14T13:19:29Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1283 2413760 wikitext text/x-wiki tdonry9a5ta2yotni7474mqpe664218 2413761 2413760 2026-05-14T13:20:38Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1283.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413761 wikitext text/x-wiki be87tk8cy13uc7zhjf7m852fhjw7nx5 2413762 2413761 2026-05-14T13:22:47Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1281.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413762 wikitext text/x-wiki cb8a7n197ud62gxop6pb2ms9k4drc7p 2413764 2413762 2026-05-14T13:29:28Z Stashbot 7414 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1285913|Add ReadingLists Account Creation CTA campaign (T422169)]], [[gerrit:1286327|WelcomeSurvey: Respect returnTo for campaigns skipping the survey (T422169)]] 2413764 wikitext text/x-wiki qa7qw4i8gvbp7wuamt3vfpainxi97vc 2413765 2413764 2026-05-14T13:29:51Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1279.eqiad.wmnet with OS bookworm 2413765 wikitext text/x-wiki rbkphogijwm4il66cae63mj23vak3uv 2413767 2413765 2026-05-14T13:30:35Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1282.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413767 wikitext text/x-wiki gw1ksszturzsvsd4vq53cwfytr4ibmd 2413768 2413767 2026-05-14T13:31:15Z Stashbot 7414 krinkle@deploy1003: krinkle, annet: Backport for [[gerrit:1285913|Add ReadingLists Account Creation CTA campaign (T422169)]], [[gerrit:1286327|WelcomeSurvey: Respect returnTo for campaigns skipping the survey (T422169)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2413768 wikitext text/x-wiki pl3p7d8ksojm2vrtaxo3qaavqk64j2l 2413770 2413768 2026-05-14T13:31:51Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2150: Host will be decommissioned 2413770 wikitext text/x-wiki 749ml3i8y7ga0tgee24soionpnwa6gs 2413771 2413770 2026-05-14T13:32:11Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2150: Host will be decommissioned 2413771 wikitext text/x-wiki ok2sh8s3wf0ryxaz29xmgu3p8gyz8bj 2413772 2413771 2026-05-14T13:33:07Z Stashbot 7414 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2151: Host will be decommissioned 2413772 wikitext text/x-wiki 70gk0x3hqkopbboii2f19xiukeo8cta 2413773 2413772 2026-05-14T13:33:27Z Stashbot 7414 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2151: Host will be decommissioned 2413773 wikitext text/x-wiki 40bpc76h27tnum9513tzwzz5nqhf714 2413775 2413773 2026-05-14T13:37:48Z Stashbot 7414 krinkle@deploy1003: krinkle, annet: Continuing with deployment 2413775 wikitext text/x-wiki hg49537bjf9xlhi1dhlwf0a5p7ih2bs 2413776 2413775 2026-05-14T13:38:13Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1283.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413776 wikitext text/x-wiki ibhljo268pn59f45c4rkbs7okm0fyzx 2413777 2413776 2026-05-14T13:40:29Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host sretest2010.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2413777 wikitext text/x-wiki g37a1s5wq9pgwopir2ncm3q442wftin 2413778 2413777 2026-05-14T13:42:02Z Stashbot 7414 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1285913|Add ReadingLists Account Creation CTA campaign (T422169)]], [[gerrit:1286327|WelcomeSurvey: Respect returnTo for campaigns skipping the survey (T422169)]] (duration: 12m 33s) 2413778 wikitext text/x-wiki kuhjk7u88hil3rysialpvf6rfesbser 2413779 2413778 2026-05-14T13:42:56Z Stashbot 7414 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1269442|Enable wgTrackMediaRequestProvenance on remaining Wikipedias (T414338)]] 2413779 wikitext text/x-wiki 3cpj5gdrbrvk5nm63j2q6gxuqtc0na5 2413780 2413779 2026-05-14T13:42:58Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1280.eqiad.wmnet with OS bookworm 2413780 wikitext text/x-wiki o8chkazaax1smmc569ikf1jfxym85eb 2413781 2413780 2026-05-14T13:44:43Z Stashbot 7414 krinkle@deploy1003: krinkle: Backport for [[gerrit:1269442|Enable wgTrackMediaRequestProvenance on remaining Wikipedias (T414338)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2413781 wikitext text/x-wiki rn1x4iiqs84qusm87ydk0u7o67szvqz 2413782 2413781 2026-05-14T13:45:12Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2010.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2413782 wikitext text/x-wiki rn4hua9b41ndua6v6rymdejwm4azjxn 2413783 2413782 2026-05-14T13:45:24Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1279.eqiad.wmnet with reason: host reimage 2413783 wikitext text/x-wiki 77u6i9e3juuq9ichbynbbhk82d34tpi 2413784 2413783 2026-05-14T13:45:50Z Stashbot 7414 krinkle@deploy1003: krinkle: Continuing with deployment 2413784 wikitext text/x-wiki 8t2l3e24r9zpfhkwhkmbd3rkjao6d2d 2413785 2413784 2026-05-14T13:48:32Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host sretest2010.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2413785 wikitext text/x-wiki odd00j5b1834gm0512vohwte4uwwzot 2413787 2413785 2026-05-14T13:49:54Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1279.eqiad.wmnet with reason: host reimage 2413787 wikitext text/x-wiki 4gf7ze2npe3kdhsm564miht4w399tft 2413788 2413787 2026-05-14T13:49:59Z Stashbot 7414 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1269442|Enable wgTrackMediaRequestProvenance on remaining Wikipedias (T414338)]] (duration: 07m 03s) 2413788 wikitext text/x-wiki bdp7vgxtvh5whyq8ygsvoclafkq2thz 2413789 2413788 2026-05-14T13:52:43Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db2150.codfw.wmnet with reason: Depooled host, will be decommissioned 2413789 wikitext text/x-wiki r6w81ltyhy23kyibsf4vl3t1zrb919l 2413790 2413789 2026-05-14T13:53:05Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db2151.codfw.wmnet with reason: Depooled host, will be decommissioned 2413790 wikitext text/x-wiki spytj45uha7w3f2oclqbdkvp6snlnce 2413791 2413790 2026-05-14T13:53:08Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2165.codfw.wmnet with reason: Maintenance 2413791 wikitext text/x-wiki frgrczdvirihxh11ar3w6icdopish4o 2413792 2413791 2026-05-14T13:53:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2165 (T419635)', diff saved to https://phabricator.wikimedia.org/P92533 and previous config saved to /var/cache/conftool/dbconfig/20260514-135315-fceratto.json 2413792 wikitext text/x-wiki f03hg52uhawa2z27hcrpgcua37w92yo 2413793 2413792 2026-05-14T13:53:27Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2010.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART 2413793 wikitext text/x-wiki 5m3j3dn4ihu28hywdarvd2w62xzo98x 2413794 2413793 2026-05-14T13:53:29Z Stashbot 7414 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db2152.codfw.wmnet with reason: Depooled host, will be decommissioned 2413794 wikitext text/x-wiki 076e8ltw5o4f8rl0uari0tgzvl7phce 2413796 2413794 2026-05-14T13:53:45Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2413796 wikitext text/x-wiki 9nee18lkalhsi9uscgj7hpw6ohhjqug 2413797 2413796 2026-05-14T13:53:49Z Stashbot 7414 mfossati@deploy1003: Started scap sync-world: Backport for [[gerrit:1287363|Scale share-highlight card to fit small viewports (T426247)]] 2413797 wikitext text/x-wiki lygpf4l1fgmxbrx3gpn2tja24bh6202 2413798 2413797 2026-05-14T13:54:26Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2413798 wikitext text/x-wiki 2pvb2s2pez7d7qph37u221torrncvoh 2413799 2413798 2026-05-14T13:54:38Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2413799 wikitext text/x-wiki guf4o0krjak09xvelzkkuhdmxtyy5xp 2413801 2413799 2026-05-14T13:55:37Z Stashbot 7414 mfossati@deploy1003: mfossati: Backport for [[gerrit:1287363|Scale share-highlight card to fit small viewports (T426247)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2413801 wikitext text/x-wiki 4rtf55a69246myftue4fg478ghgv6co 2413802 2413801 2026-05-14T13:56:02Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2413802 wikitext text/x-wiki 8x6d63ex1ttv1jfcilk49s01nv74yxe 2413803 2413802 2026-05-14T13:56:05Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2413803 wikitext text/x-wiki 1rlrm2hrjl7649k46jefn3oeixlk21q 2413804 2413803 2026-05-14T13:56:23Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2413804 wikitext text/x-wiki spfzvfa1l3bcwhv3ei2z2lhr4l0iq03 2413805 2413804 2026-05-14T13:56:29Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 (T419635)', diff saved to https://phabricator.wikimedia.org/P92534 and previous config saved to /var/cache/conftool/dbconfig/20260514-135626-fceratto.json 2413805 wikitext text/x-wiki t8b13ql544i9dgx99qh1esduew3ik6q 2413806 2413805 2026-05-14T13:56:43Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2413806 wikitext text/x-wiki lenzemqrmx6y4uo2qivf6zgw0vfvpsh 2413807 2413806 2026-05-14T13:56:50Z Stashbot 7414 mfossati@deploy1003: mfossati: Continuing with deployment 2413807 wikitext text/x-wiki i2633o2xf815znizy0ifw80aoridwyg 2413808 2413807 2026-05-14T13:56:52Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2413808 wikitext text/x-wiki rmza9ao95116412pzo5r7f4nisr1gdv 2413809 2413808 2026-05-14T13:57:05Z Stashbot 7414 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2413809 wikitext text/x-wiki iwhp77qjm5smrp6qzpit14hk7s3j1c3 2413810 2413809 2026-05-14T13:58:24Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1280.eqiad.wmnet with reason: host reimage 2413810 wikitext text/x-wiki gadicf2gzxqo5lmfs8jb26vwwnylnpf 2413811 2413810 2026-05-14T13:59:47Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1282.eqiad.wmnet with OS bookworm 2413811 wikitext text/x-wiki isf0ftaza0cimca7zd42qgohoiri8o8 2413812 2413811 2026-05-14T14:00:58Z Stashbot 7414 mfossati@deploy1003: Finished scap sync-world: Backport for [[gerrit:1287363|Scale share-highlight card to fit small viewports (T426247)]] (duration: 07m 09s) 2413812 wikitext text/x-wiki ssclla3p1ny4v702yrnuisuzw1wwlvz 2413814 2413812 2026-05-14T14:01:13Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Remove db2151 from dbctl T424343', diff saved to https://phabricator.wikimedia.org/P92535 and previous config saved to /var/cache/conftool/dbconfig/20260514-140110-cwilliams.json 2413814 wikitext text/x-wiki e2uynvzpznt43tfqj0nc21arppxwomi 2413815 2413814 2026-05-14T14:04:36Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1280.eqiad.wmnet with reason: host reimage 2413815 wikitext text/x-wiki rf2hsuk3hke4afggjxbqtgz50un5ulu 2413816 2413815 2026-05-14T14:05:16Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie 2413816 wikitext text/x-wiki 7lopkv6oaevpyvoza1ccb3y8vdqvisg 2413817 2413816 2026-05-14T14:06:08Z Stashbot 7414 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1287367|throttle rule for ESEAP Conference 2026 15-18 May 2026 (T426295)]] 2413817 wikitext text/x-wiki gm6f424n6iivqa240jby4kef4yo942b 2413818 2413817 2026-05-14T14:06:37Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P92536 and previous config saved to /var/cache/conftool/dbconfig/20260514-140635-fceratto.json 2413818 wikitext text/x-wiki c6yeuvhp1rbmnljfytwkf9s98u25nvf 2413819 2413818 2026-05-14T14:06:58Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2413819 wikitext text/x-wiki sxorlu91rrt76e9q5in5mv4xizilezt 2413820 2413819 2026-05-14T14:07:33Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2413820 wikitext text/x-wiki kstwz3gnq1p8rcwgo9q7m1sb0x7e11e 2413821 2413820 2026-05-14T14:07:34Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1279.eqiad.wmnet with OS bookworm 2413821 wikitext text/x-wiki 66frai1nr6h186otahr0fx0rgxcwiek 2413822 2413821 2026-05-14T14:08:01Z Stashbot 7414 krinkle@deploy1003: krinkle, robertsky: Backport for [[gerrit:1287367|throttle rule for ESEAP Conference 2026 15-18 May 2026 (T426295)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2413822 wikitext text/x-wiki 3puum57fzx2o8bi4j62ru9fhxi7tcmc 2413823 2413822 2026-05-14T14:09:42Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2413823 wikitext text/x-wiki 5po3g6tuql1fl483iiatrmhfoudespj 2413824 2413823 2026-05-14T14:09:44Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2413824 wikitext text/x-wiki nq6jfrbraalkhh2mv0bp8qefbgkb0wm 2413826 2413824 2026-05-14T14:09:58Z Stashbot 7414 krinkle@deploy1003: krinkle, robertsky: Continuing with deployment 2413826 wikitext text/x-wiki o38633ksljy01zzbwj16jrwyid8bkpf 2413827 2413826 2026-05-14T14:13:05Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2413827 wikitext text/x-wiki g9j79wwroicgmeyaivpovchozuputn5 2413829 2413827 2026-05-14T14:14:09Z Stashbot 7414 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1287367|throttle rule for ESEAP Conference 2026 15-18 May 2026 (T426295)]] (duration: 08m 00s) 2413829 wikitext text/x-wiki l534zxw0wg7of29apqw37uivo976zg9 2413831 2413829 2026-05-14T14:15:26Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1282.eqiad.wmnet with reason: host reimage 2413831 wikitext text/x-wiki jkmqszu0arc1fy3j8wgqz9569q4ue8s 2413832 2413831 2026-05-14T14:16:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P92537 and previous config saved to /var/cache/conftool/dbconfig/20260514-141644-fceratto.json 2413832 wikitext text/x-wiki 9tag75qf4rn7cx9ormchmmx93k08yj8 2413833 2413832 2026-05-14T14:17:05Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1284] - vriley@cumin1003" 2413833 wikitext text/x-wiki 3qmt58j7935dj3ebvddb3p4duwoe9q6 2413834 2413833 2026-05-14T14:17:11Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1284] - vriley@cumin1003" 2413834 wikitext text/x-wiki d6a97gmm1ffs15phgprssrugju8x54e 2413835 2413834 2026-05-14T14:17:12Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2413835 wikitext text/x-wiki ckenq7ugcx9bbdldq441khi1a7rh6cy 2413836 2413835 2026-05-14T14:17:26Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1284 2413836 wikitext text/x-wiki 986juswtumhn6uk1zvhvpm9ks6c2t2p 2413837 2413836 2026-05-14T14:18:16Z Stashbot 7414 cwilliams@cumin1003: dbctl commit (dc=all): 'Remove db2150 from dbctl T424342', diff saved to https://phabricator.wikimedia.org/P92538 and previous config saved to /var/cache/conftool/dbconfig/20260514-141812-cwilliams.json 2413837 wikitext text/x-wiki 2i87qqtqkhhtw3ietme15jrjorzovac 2413838 2413837 2026-05-14T14:18:19Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1283.eqiad.wmnet with OS bookworm 2413838 wikitext text/x-wiki auc9w2kyvfdgjlmscnlwzu43m9fmjjq 2413839 2413838 2026-05-14T14:18:55Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1282.eqiad.wmnet with reason: host reimage 2413839 wikitext text/x-wiki n7d5qbsb6g2jys77t96l7gyhcx1m3e1 2413840 2413839 2026-05-14T14:19:16Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance 2413840 wikitext text/x-wiki 7er88jqd4mf3v0ysbd714rh16hkh811 2413841 2413840 2026-05-14T14:19:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2165 (T419961)', diff saved to https://phabricator.wikimedia.org/P92539 and previous config saved to /var/cache/conftool/dbconfig/20260514-141922-fceratto.json 2413841 wikitext text/x-wiki dnpcyjuxvriacv7g35ccll7a0ees35d 2413842 2413841 2026-05-14T14:19:37Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1284 2413842 wikitext text/x-wiki nn1xn9mgkdxztwi1r26yyz4ath08pmv 2413843 2413842 2026-05-14T14:20:13Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1284.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413843 wikitext text/x-wiki 29rxpqbm2nbgr7ghkw4wjjbey3cqdku 2413845 2413843 2026-05-14T14:21:34Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2413845 wikitext text/x-wiki 3bdqjdo1njlf5bat29f9g2bu6x1absu 2413846 2413845 2026-05-14T14:22:46Z Stashbot 7414 phuedx@deploy1003: Started scap sync-world: Backport for [[gerrit:1287368|ext.wikimediaEvents: Add synth-aa-ncs-1 experiment (T419514)]] 2413846 wikitext text/x-wiki f9uepqgwidg5w1pzv7vz6mcf9cbghs5 2413847 2413846 2026-05-14T14:23:03Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2413847 wikitext text/x-wiki 7zk4yuw11gumwgj9jj47tukdbk3eghg 2413848 2413847 2026-05-14T14:23:05Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1280.eqiad.wmnet with OS bookworm 2413848 wikitext text/x-wiki 0wr89e38jbncfc0vqqm6rk0lz5cou6b 2413852 2413848 2026-05-14T14:24:35Z Stashbot 7414 phuedx@deploy1003: phuedx: Backport for [[gerrit:1287368|ext.wikimediaEvents: Add synth-aa-ncs-1 experiment (T419514)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2413852 wikitext text/x-wiki 0d62r864ofvt5oc2im8cc80732nene7 2413853 2413852 2026-05-14T14:26:11Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2413853 wikitext text/x-wiki b50j35qhssk8ctbfmz52zsurgps7kqr 2413855 2413853 2026-05-14T14:26:52Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 (T419961)', diff saved to https://phabricator.wikimedia.org/P92540 and previous config saved to /var/cache/conftool/dbconfig/20260514-142650-fceratto.json 2413855 wikitext text/x-wiki s4n3ubcbo6dog8lejfgi1pfoagdphsr 2413856 2413855 2026-05-14T14:27:30Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2413856 wikitext text/x-wiki ig7wb9cwpkoblfrgn2bugrz9zfoh3zm 2413857 2413856 2026-05-14T14:29:47Z Stashbot 7414 phuedx@deploy1003: phuedx: Continuing with deployment 2413857 wikitext text/x-wiki l08nx4veki2tfokqegm48nniw0565xi 2413858 2413857 2026-05-14T14:31:47Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1285] - vriley@cumin1003" 2413858 wikitext text/x-wiki gcdlxtauvvb8a1ozqtwa3yces7ab9yz 2413859 2413858 2026-05-14T14:31:53Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1285] - vriley@cumin1003" 2413859 wikitext text/x-wiki qexa5enuxqm32jltst9pf4cqg3own3x 2413860 2413859 2026-05-14T14:31:54Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2413860 wikitext text/x-wiki qs84lv1mcqbzgkgvyh916fwq7xb8i1v 2413861 2413860 2026-05-14T14:32:17Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1285 2413861 wikitext text/x-wiki sm47f4034rhrd3xvngf9ncnedbr9kgx 2413862 2413861 2026-05-14T14:33:48Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1285 2413862 wikitext text/x-wiki cudhqdmvzf4tekass3d1xfe0v9jsbdm 2413863 2413862 2026-05-14T14:33:50Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1283.eqiad.wmnet with reason: host reimage 2413863 wikitext text/x-wiki oq6hmuxycoxqc9zj5pgokp6j031rwe4 2413864 2413863 2026-05-14T14:33:57Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2413864 wikitext text/x-wiki hiow4wr13lm8xoc2dpn2gdhs6k5z5bi 2413865 2413864 2026-05-14T14:34:01Z Stashbot 7414 phuedx@deploy1003: Finished scap sync-world: Backport for [[gerrit:1287368|ext.wikimediaEvents: Add synth-aa-ncs-1 experiment (T419514)]] (duration: 11m 14s) 2413865 wikitext text/x-wiki n46420c40nlif0mn0yg5nkwygfs4bwn 2413866 2413865 2026-05-14T14:35:21Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2413866 wikitext text/x-wiki 48k8nd2y3izd8uz8105c067v1dhn516 2413867 2413866 2026-05-14T14:35:38Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2413867 wikitext text/x-wiki bzfqvpgl87dfx2m67i62r40w53rttky 2413868 2413867 2026-05-14T14:35:40Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1282.eqiad.wmnet with OS bookworm 2413868 wikitext text/x-wiki 4cbyn09lg30jwsbnugdj9odhj2az8h7 2413869 2413868 2026-05-14T14:37:01Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P92541 and previous config saved to /var/cache/conftool/dbconfig/20260514-143659-fceratto.json 2413869 wikitext text/x-wiki k06yqvd2y6yg97p3qvh4tqgrr1aoz0e 2413870 2413869 2026-05-14T14:37:03Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1289 2413870 wikitext text/x-wiki 6upzcv48dtuvbmor5a3vt83kyw40aik 2413871 2413870 2026-05-14T14:37:05Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host db1289 2413871 wikitext text/x-wiki ro3df9zl6yn53jzkvndvnmxv5p92vk4 2413872 2413871 2026-05-14T14:38:43Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1287] - vriley@cumin1003" 2413872 wikitext text/x-wiki 5n4zhzqgnnxn7fav5sexjreoshid7jp 2413873 2413872 2026-05-14T14:38:49Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1287] - vriley@cumin1003" 2413873 wikitext text/x-wiki jbpiyi76v6lymrlt15329clxlcvvids 2413874 2413873 2026-05-14T14:38:50Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2413874 wikitext text/x-wiki bkhvljb5euqud45fvbgv6ywurad4awx 2413875 2413874 2026-05-14T14:38:55Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1284.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413875 wikitext text/x-wiki h5ed09zfwdwryq8fa7q3ajs9264fqb6 2413876 2413875 2026-05-14T14:39:03Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1283.eqiad.wmnet with reason: host reimage 2413876 wikitext text/x-wiki mt17tnkoak7k7lrjcie2bwwzpzc3hcv 2413877 2413876 2026-05-14T14:44:16Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1285.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413877 wikitext text/x-wiki mwfbd740p6vmth03d2amxq8fpqd6kot 2413878 2413877 2026-05-14T14:46:52Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1287.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413878 wikitext text/x-wiki kbgzcymx5mhc8vakwtta53ofb2ex5ev 2413879 2413878 2026-05-14T14:47:08Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P92542 and previous config saved to /var/cache/conftool/dbconfig/20260514-144707-fceratto.json 2413879 wikitext text/x-wiki cqw4hodvnkpoftf0tqeugdbawlzsn55 2413880 2413879 2026-05-14T14:49:46Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2413880 wikitext text/x-wiki e3e6hwusmz0n3gw6hc1b1xyzeqbn1fh 2413882 2413880 2026-05-14T14:52:24Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1284.eqiad.wmnet with OS bookworm 2413882 wikitext text/x-wiki 9cj9szt31da7syf7gezzux3s3i7ozk9 2413884 2413882 2026-05-14T14:53:45Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1288] - vriley@cumin1003" 2413884 wikitext text/x-wiki o64ds4y8ubducgo71i6o5vldcjntyf8 2413885 2413884 2026-05-14T14:53:51Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1288] - vriley@cumin1003" 2413885 wikitext text/x-wiki dvsshbt2pqgkk1pv0l74gtw8uyfwpd2 2413886 2413885 2026-05-14T14:53:53Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2413886 wikitext text/x-wiki fxblfczt90bhne2eaigz5m0uvfrh8xx 2413887 2413886 2026-05-14T14:53:54Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2413887 wikitext text/x-wiki tqx6ykcnacp7k7gvzvi579hcjuny5sh 2413888 2413887 2026-05-14T14:54:07Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1288 2413888 wikitext text/x-wiki hu287wpef39cu3mm7uirddjys82vazv 2413889 2413888 2026-05-14T14:54:09Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage 2413889 wikitext text/x-wiki h6kr2gtmwatm43i4hrvls2cvinmz4b9 2413890 2413889 2026-05-14T14:54:30Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2413890 wikitext text/x-wiki p7bweve5j2gvnl6lbiwrdgxokffprs8 2413891 2413890 2026-05-14T14:54:32Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1283.eqiad.wmnet with OS bookworm 2413891 wikitext text/x-wiki m4r7m2vqo9cjmjuaks8k3m4z6ti1hnq 2413892 2413891 2026-05-14T14:55:07Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1285.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413892 wikitext text/x-wiki h3p92exbh7sc2564eh4t06uluqrkluv 2413893 2413892 2026-05-14T14:55:19Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1288 2413893 wikitext text/x-wiki 26b5z0tuva3lkf1jc9p8z57rk2883lx 2413896 2413893 2026-05-14T14:57:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 (T419961)', diff saved to https://phabricator.wikimedia.org/P92544 and previous config saved to /var/cache/conftool/dbconfig/20260514-145715-fceratto.json 2413896 wikitext text/x-wiki 5dzpcequ7tctgpi9xkszjpf5xygovrb 2413897 2413896 2026-05-14T14:57:20Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1288.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413897 wikitext text/x-wiki gj0ekblunsempip0jwik59f9hdahu1i 2413898 2413897 2026-05-14T14:58:50Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1287.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413898 wikitext text/x-wiki dxrue6soai928p41vfgtnrv7s1jurzf 2413899 2413898 2026-05-14T14:59:51Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage 2413899 wikitext text/x-wiki 7igpqh9m6d1a6u9nzca61bc5brnq08v 2413902 2413899 2026-05-14T15:07:29Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1285.eqiad.wmnet with OS bookworm 2413902 wikitext text/x-wiki dzjgqh85zzrzrz8mjl812q1zny9eysx 2413903 2413902 2026-05-14T15:08:06Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1287.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413903 wikitext text/x-wiki 4ofsdffj9loq0jr6b2rmj705bt5ns7w 2413904 2413903 2026-05-14T15:08:14Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1284.eqiad.wmnet with reason: host reimage 2413904 wikitext text/x-wiki lyj6u6umcgzjdz7pevzchhkbqrsy72k 2413905 2413904 2026-05-14T15:10:16Z Stashbot 7414 bearloga@deploy1003: Started scap sync-world: Backport for [[gerrit:1287422|EventStreamConfig: fix product_metrics.web_base (T426209)]] 2413905 wikitext text/x-wiki 721qrbdcr5x4i6mibhh6xsj4zo5buw6 2413906 2413905 2026-05-14T15:12:03Z Stashbot 7414 bearloga@deploy1003: bearloga: Backport for [[gerrit:1287422|EventStreamConfig: fix product_metrics.web_base (T426209)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2413906 wikitext text/x-wiki 8xoax4idamdgv6e6729wo936akvb0xx 2413907 2413906 2026-05-14T15:12:08Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1287.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413907 wikitext text/x-wiki 3y1ksnu4lbh0gg8voshlwm99tn1by7r 2413908 2413907 2026-05-14T15:12:22Z Stashbot 7414 bearloga@deploy1003: bearloga: Continuing with deployment 2413908 wikitext text/x-wiki 50x768tlq5am5if8n8uz1w3fblchwyn 2413910 2413908 2026-05-14T15:14:44Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1284.eqiad.wmnet with reason: host reimage 2413910 wikitext text/x-wiki kaj3e0t04t220agpy6decaid12b2iap 2413912 2413910 2026-05-14T15:16:14Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1288.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413912 wikitext text/x-wiki diwjolbqszhcq9ehzzjw03msr1p6m6d 2413913 2413912 2026-05-14T15:16:36Z Stashbot 7414 bearloga@deploy1003: Finished scap sync-world: Backport for [[gerrit:1287422|EventStreamConfig: fix product_metrics.web_base (T426209)]] (duration: 06m 20s) 2413913 wikitext text/x-wiki ku0q7veogfblyon64sr5gpb7qv1mbgr 2413914 2413913 2026-05-14T15:23:15Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1285.eqiad.wmnet with reason: host reimage 2413914 wikitext text/x-wiki pq2up4qkrd4q6mmrbdsetf8yvd1at57 2413915 2413914 2026-05-14T15:25:18Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie 2413915 wikitext text/x-wiki kobub6cjb89l1e9wqctzt9rxlzlqfcb 2413917 2413915 2026-05-14T15:29:27Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1285.eqiad.wmnet with reason: host reimage 2413917 wikitext text/x-wiki g0eggqrel6xxuhdudeg3kc0729nkttv 2413919 2413917 2026-05-14T15:31:57Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2413919 wikitext text/x-wiki h8iapp8f3ak97o62kb6hdjdg0ds14rt 2413920 2413919 2026-05-14T15:32:39Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2413920 wikitext text/x-wiki 069nt8dqmmcdo9h2jh26ymd7ammc2xb 2413921 2413920 2026-05-14T15:32:40Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1284.eqiad.wmnet with OS bookworm 2413921 wikitext text/x-wiki 9tp66t4l5075psj3aii3jh7e2nz4k0o 2413922 2413921 2026-05-14T15:33:37Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1288.eqiad.wmnet with OS bookworm 2413922 wikitext text/x-wiki 4mgzkrdlu5424hz3v44pdeeaqmu9hkc 2413924 2413922 2026-05-14T15:35:25Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2413924 wikitext text/x-wiki 3z1o84keu2xxx4d3k59cjyocvv111bq 2413927 2413924 2026-05-14T15:41:12Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1289] - vriley@cumin1003" 2413927 wikitext text/x-wiki l25fnvu4nsz0d86yjt9pz824cs92xze 2413928 2413927 2026-05-14T15:41:18Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1289] - vriley@cumin1003" 2413928 wikitext text/x-wiki 1hjmsmhjn93biybizcu1wqrljl45wd8 2413929 2413928 2026-05-14T15:41:19Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2413929 wikitext text/x-wiki 6l0wtya6kbu7d6r45o6nzhiw598lw7g 2413930 2413929 2026-05-14T15:41:31Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1289 2413930 wikitext text/x-wiki nhcl8nd13xujabspoo7ww1v089ecoc3 2413931 2413930 2026-05-14T15:42:52Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1289 2413931 wikitext text/x-wiki mmihufzmyvmvems6zn2w2i29i7jioqp 2413932 2413931 2026-05-14T15:45:52Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2413932 wikitext text/x-wiki 7dq5w0azbkvftme388ehbk9inbuh2m4 2413933 2413932 2026-05-14T15:45:57Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1289.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413933 wikitext text/x-wiki 12xes8ntzqubgtoezghjlaqgmoltkvj 2413934 2413933 2026-05-14T15:46:48Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging1006.eqiad.wmnet with OS trixie 2413934 wikitext text/x-wiki hk9w296pgfl59a5eh9b85tfje7ix0z5 2413936 2413934 2026-05-14T15:48:53Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2413936 wikitext text/x-wiki fpje9nwhwc3z3bv7klxy6ajrzxkfe2e 2413937 2413936 2026-05-14T15:48:55Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1285.eqiad.wmnet with OS bookworm 2413937 wikitext text/x-wiki 7e4lnjumcqolk0077veby04y1zzq832 2413938 2413937 2026-05-14T15:49:28Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin1003.eqiad.wmnet with reason: Release v0.11.2 - cmooney@cumin1003 2413938 wikitext text/x-wiki pm407dch76xk29b16ebe6bvuw79vzrx 2413939 2413938 2026-05-14T15:49:38Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1288.eqiad.wmnet with reason: host reimage 2413939 wikitext text/x-wiki bkfrzi51x8ekeuq32r8h3rfuadtk5z2 2413940 2413939 2026-05-14T15:50:06Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2413940 wikitext text/x-wiki 3mb4aii9dk3akvrdbx310qivufr3ugp 2413941 2413940 2026-05-14T15:51:08Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin1003.eqiad.wmnet with reason: Release v0.11.2 - cmooney@cumin1003 2413941 wikitext text/x-wiki aadbwqwh2k2n2a5s9o0qriofqnxfpry 2413943 2413941 2026-05-14T15:54:50Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1288.eqiad.wmnet with reason: host reimage 2413943 wikitext text/x-wiki 59vcr4agv4qfeirb8eoeb5gstpwwfrx 2413944 2413943 2026-05-14T15:54:58Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1290] - vriley@cumin1003" 2413944 wikitext text/x-wiki 3rjlusbv72x6qcwyvhwfu7z5wztrv9k 2413945 2413944 2026-05-14T15:55:04Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1290] - vriley@cumin1003" 2413945 wikitext text/x-wiki 6as3pbqqpny07ag52ioqns547mx405p 2413946 2413945 2026-05-14T15:55:06Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2413946 wikitext text/x-wiki 7td7lrbyvyzd7xlppvvtv23bw0uui0y 2413947 2413946 2026-05-14T15:55:23Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1290 2413947 wikitext text/x-wiki oz6qimnodcp4bp6c4ud6i3iwcad8abn 2413948 2413947 2026-05-14T15:56:43Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1290 2413948 wikitext text/x-wiki g04b213tgu6hfuuk44wfanhixl9h3yv 2413949 2413948 2026-05-14T15:57:48Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1290.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413949 wikitext text/x-wiki 2nctpu8p4umzkpj9sjw97iqicjg1hn8 2413950 2413949 2026-05-14T15:59:13Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply 2413950 wikitext text/x-wiki t196rh4k676n7wvefren4v31xpkutj2 2413951 2413950 2026-05-14T15:59:17Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1289.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413951 wikitext text/x-wiki aydvt21vsotpnmw6pnnuwomyju4gsna 2413952 2413951 2026-05-14T15:59:19Z Stashbot 7414 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply 2413952 wikitext text/x-wiki eezhj1ojw92lwki470lbbalxmzs6gd0 2413956 2413952 2026-05-14T16:04:06Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2413956 wikitext text/x-wiki 8qjnn03n4ruytffprtupv25tgq27pbw 2413959 2413956 2026-05-14T16:06:58Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply 2413959 wikitext text/x-wiki 2xzfcw4yvztcsiubtw4jsry9w7oufl7 2413960 2413959 2026-05-14T16:07:17Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove records for deleted IPs esams,drmrs and magru - cmooney@cumin1003" 2413960 wikitext text/x-wiki 98go1z179j2iobrmwq09g1vhrzz9g66 2413961 2413960 2026-05-14T16:07:39Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove records for deleted IPs esams,drmrs and magru - cmooney@cumin1003" 2413961 wikitext text/x-wiki 5jkxd4mkldqzefs96oh5bzu7jmoqtv4 2413962 2413961 2026-05-14T16:07:40Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2413962 wikitext text/x-wiki asrv4mu0mn60kh4r1dkg56wue8ayph7 2413963 2413962 2026-05-14T16:11:34Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply 2413963 wikitext text/x-wiki cb8wdj49y0zt1in5niwk2nbzdyj41qb 2413964 2413963 2026-05-14T16:12:49Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2413964 wikitext text/x-wiki 4xt0p9ves3kw3pfs57wko4q3fdodmuu 2413965 2413964 2026-05-14T16:13:55Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2413965 wikitext text/x-wiki c8byhmv3ko0h12v8qfcpy7qh9ubq3lq 2413966 2413965 2026-05-14T16:13:56Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1288.eqiad.wmnet with OS bookworm 2413966 wikitext text/x-wiki adc16md1yuhzx0pz91u9l2h6xbf0aix 2413967 2413966 2026-05-14T16:14:04Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1289.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413967 wikitext text/x-wiki r4krtulxdki3wmd2uhtgxyqh8lh2gxe 2413968 2413967 2026-05-14T16:15:39Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1290.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413968 wikitext text/x-wiki gksk49j79tsya4aymvducjmn0d2fqmw 2413969 2413968 2026-05-14T16:16:07Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1289.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2413969 wikitext text/x-wiki 5cm48k9nm83aqe7hz31hvspfsct3k3i 2413970 2413969 2026-05-14T16:17:52Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply 2413970 wikitext text/x-wiki 84xl0j1wnoy46lt10nfq2l1akd6v1jg 2413973 2413970 2026-05-14T16:19:58Z Stashbot 7414 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply 2413973 wikitext text/x-wiki 3qgmfy5znnvmxmn4c9n1fanjdojozm8 2413974 2413973 2026-05-14T16:20:33Z Stashbot 7414 rzl@deploy1003: helmfile [codfw] START helmfile.d/services/mw-cron: apply 2413974 wikitext text/x-wiki 8ll28m24m9l71hfagb00pnnvxds216m 2413975 2413974 2026-05-14T16:20:57Z Stashbot 7414 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-cron: apply 2413975 wikitext text/x-wiki oocomscry7j1kjbonu5kefugggb0irs 2413976 2413975 2026-05-14T16:21:16Z Stashbot 7414 topranks: disable core router direct link at magru now that traffic is flowing via switches T424611 2413976 wikitext text/x-wiki 79exvpmxra68c6jdgoo478y0oeem8hb 2413977 2413976 2026-05-14T16:25:15Z Stashbot 7414 topranks: disable core router direct link at drmrs now that traffic is flowing via switches T424611 2413977 wikitext text/x-wiki 2rkuqt033n2pq76obdi04cwi6xuynpl 2413979 2413977 2026-05-14T16:31:48Z Stashbot 7414 topranks: disable core router direct link at esams now that traffic is flowing via switches T424611 2413979 wikitext text/x-wiki llazb1enhoo00z2sfb6iatebw153rj2 2413980 2413979 2026-05-14T16:35:51Z Stashbot 7414 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-logging1006.eqiad.wmnet with OS trixie 2413980 wikitext text/x-wiki hfnmkfy1no73fgeh8hcwwqvlyowdgx3 2413981 2413980 2026-05-14T16:36:09Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2413981 wikitext text/x-wiki 39lht7d4wrdggj5q3u8759sqcislc17 2413982 2413981 2026-05-14T16:36:12Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2413982 wikitext text/x-wiki h91we2mf7vqu9iy1rxhvynzmykcaa88 2413984 2413982 2026-05-14T16:49:12Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2413984 wikitext text/x-wiki dst9t8h5v7fj0ebezgc0kgi1k5i7jxi 2413985 2413984 2026-05-14T16:49:17Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2413985 wikitext text/x-wiki 3g7wk2ziv42b5gbmijxl877qp4zyvy3 2413986 2413985 2026-05-14T16:58:07Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T426298 2413986 wikitext text/x-wiki r574150q7vqoaxwr9gi6gw7qnchwa5t 2413988 2413986 2026-05-14T17:06:59Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T426298 2413988 wikitext text/x-wiki 5yhjbs6xpufek3gx4v6qdflkyv2hwfo 2413989 2413988 2026-05-14T17:09:12Z Stashbot 7414 cmooney@dns2005: START - running authdns-update 2413989 wikitext text/x-wiki qagshkk44vswdm2jyl5vzy0mu7593tg 2413990 2413989 2026-05-14T17:10:28Z Stashbot 7414 cmooney@dns2005: END - running authdns-update 2413990 wikitext text/x-wiki 9lmrwarxdfvfh56h8fv4hnoqb5tc2hx 2413991 2413990 2026-05-14T17:14:02Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release - T426298 2413991 wikitext text/x-wiki 6l1glr6h718i21e7mf8cswd6yg91t0z 2413992 2413991 2026-05-14T17:15:59Z Stashbot 7414 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply 2413992 wikitext text/x-wiki 4b68ft3qa3ucsy15i4585gx9odfkeu3 2413993 2413992 2026-05-14T17:16:14Z Stashbot 7414 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply 2413993 wikitext text/x-wiki 0m8oed5xygj8ht6a9iub0zjghgaf2rz 2413994 2413993 2026-05-14T17:16:38Z Stashbot 7414 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply 2413994 wikitext text/x-wiki raywu8s1adju8v2lv2nag7xslzk7a5c 2413995 2413994 2026-05-14T17:16:54Z Stashbot 7414 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply 2413995 wikitext text/x-wiki ik48puj1cpvd89jxk0dpwmumq8gj9z4 2413996 2413995 2026-05-14T17:17:27Z Stashbot 7414 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply 2413996 wikitext text/x-wiki 2eirdllfxu03p1zzuqn9l7prpdh1eo6 2413997 2413996 2026-05-14T17:17:48Z Stashbot 7414 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply 2413997 wikitext text/x-wiki nczko95mn7st2gggkfxdqij8rhonh2e 2413999 2413997 2026-05-14T17:23:04Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Security Release - T426298 2413999 wikitext text/x-wiki kyfvzgrv5fxga071fw41s7rn391nd18 2414000 2413999 2026-05-14T17:31:15Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2414000 wikitext text/x-wiki pbhamkdd31ck26rzz8sib99r2jynrr3 2414001 2414000 2026-05-14T17:32:07Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2414001 wikitext text/x-wiki gcd34t1yg54wxsx0oog3xxpykphoa1a 2414005 2414001 2026-05-14T18:09:05Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1274.eqiad.wmnet with OS bookworm 2414005 wikitext text/x-wiki gx6g97c6atauxo3xndas93ki2qj1udg 2414006 2414005 2026-05-14T18:14:09Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2414006 wikitext text/x-wiki 5urwe92kw1g2zk4l55pwz2cpgixtv8f 2414007 2414006 2026-05-14T18:16:49Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2414007 wikitext text/x-wiki lfj3mhrdewezpqho4ak1dhmxf0d9m62 2414008 2414007 2026-05-14T18:17:23Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1281.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414008 wikitext text/x-wiki idutk594yiafclnc2ogxcoc5qcs5bm5 2414009 2414008 2026-05-14T18:25:03Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1274.eqiad.wmnet with reason: host reimage 2414009 wikitext text/x-wiki 9j48rko1jxsf4ic293zhcjbxo65phb1 2414011 2414009 2026-05-14T18:29:38Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1274.eqiad.wmnet with reason: host reimage 2414011 wikitext text/x-wiki ecc3tvmsjbsbjh3bcubs8ntopkpbb98 2414012 2414011 2026-05-14T18:40:49Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1281.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414012 wikitext text/x-wiki d03ka5i020ogv6bgyw3um8gqzkfothj 2414013 2414012 2026-05-14T18:47:06Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2414013 wikitext text/x-wiki ndlgt5fttjsmh6bjbuol6dulsv6yurd 2414016 2414013 2026-05-14T18:57:58Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2414016 wikitext text/x-wiki fr7bu02b86vt8eedadflanwe1t9juwt 2414017 2414016 2026-05-14T18:58:01Z Stashbot 7414 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2414017 wikitext text/x-wiki 5ubjrzd7zkiwesix0vvqmlo8keza6q1 2414019 2414017 2026-05-14T19:06:51Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1281.eqiad.wmnet with OS bookworm 2414019 wikitext text/x-wiki bsy0u25pzzvd4d4g8ph8rw2slcjtlk6 2414023 2414019 2026-05-14T19:14:59Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2414023 wikitext text/x-wiki g23cdkosj1mc1ec8cwt7m90dvecqcxb 2414024 2414023 2026-05-14T19:15:01Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1274.eqiad.wmnet with OS bookworm 2414024 wikitext text/x-wiki 81hvtwism3pei4wapq0lekfh8jrj2us 2414026 2414024 2026-05-14T19:22:33Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2414026 wikitext text/x-wiki 77vqq4umxf1ltfvubke761kiynhh6qr 2414027 2414026 2026-05-14T19:23:08Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1281.eqiad.wmnet with reason: host reimage 2414027 wikitext text/x-wiki kgcg6hwvi35qjrs75ebficsnhk71489 2414029 2414027 2026-05-14T19:26:26Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1286] - vriley@cumin1003" 2414029 wikitext text/x-wiki 2kupcoiibt8qzqbcp6bakgytay77i0v 2414030 2414029 2026-05-14T19:26:31Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [db1286] - vriley@cumin1003" 2414030 wikitext text/x-wiki 8gcttx6p2ttd8fhdf5q8rwtl5k88int 2414031 2414030 2026-05-14T19:26:33Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2414031 wikitext text/x-wiki 510kwjh6qdykowbd0bbugolvdgs62sx 2414032 2414031 2026-05-14T19:26:47Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1286 2414032 wikitext text/x-wiki foqfap9qiqxjhi6h1cwa06ocit76kiq 2414033 2414032 2026-05-14T19:28:04Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1286 2414033 wikitext text/x-wiki byyyytj0h2ukcfgld45j52ai9w9sti9 2414034 2414033 2026-05-14T19:28:35Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1286.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414034 wikitext text/x-wiki h63mcj7xdce0yehfsn0qgcbd9rdgjfo 2414035 2414034 2026-05-14T19:29:08Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1281.eqiad.wmnet with reason: host reimage 2414035 wikitext text/x-wiki d4orce63vbr4vfumr694ouf74deis5d 2414038 2414035 2026-05-14T19:38:14Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1287.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414038 wikitext text/x-wiki 45bl4u2i0s11f827az39vp3vbxo9z3q 2414039 2414038 2026-05-14T19:45:48Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2414039 wikitext text/x-wiki i1wyzjyil1pj0s95tyhdxy5b4sx7bqs 2414040 2414039 2026-05-14T19:46:23Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1286.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414040 wikitext text/x-wiki o3r9azy8tqx9f2vclimx0s0zk5ovkja 2414041 2414040 2026-05-14T19:49:14Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2414041 wikitext text/x-wiki rk58gkbjyfryunq2ryvxmz2o5bawhc6 2414042 2414041 2026-05-14T19:49:15Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1281.eqiad.wmnet with OS bookworm 2414042 wikitext text/x-wiki n8don5p62w7rzxdggqjws8qksyyu47e 2414043 2414042 2026-05-14T19:56:49Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1287.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414043 wikitext text/x-wiki e5ouccptkhjl8xu9f4ibwajd5e36l0y 2414044 2414043 2026-05-14T20:02:25Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1286.eqiad.wmnet with OS bookworm 2414044 wikitext text/x-wiki px04t5xvpt0yvwqoz2ajaabcgye5ul6 2414045 2414044 2026-05-14T20:03:20Z Stashbot 7414 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2414045 wikitext text/x-wiki 210j4fno2fm8qqeoqy3sm35t7m1v2pq 2414046 2414045 2026-05-14T20:11:14Z Stashbot 7414 jsn@deploy1003: Started scap sync-world: Backport for [[gerrit:1192921|Enable AutoModerator on Italian Wikipedia (T405152)]], [[gerrit:1286974|Enable AutoModerator on Albanian Wikipedia (T420450)]], [[gerrit:1286975|Enable AutoModerator on Dutch Wikipedia (T425509)]] 2414046 wikitext text/x-wiki 64rbke37a33im9md01dy9ofdsoum2cm 2414047 2414046 2026-05-14T20:13:08Z Stashbot 7414 jsn@deploy1003: kgraessle, jsn: Backport for [[gerrit:1192921|Enable AutoModerator on Italian Wikipedia (T405152)]], [[gerrit:1286974|Enable AutoModerator on Albanian Wikipedia (T420450)]], [[gerrit:1286975|Enable AutoModerator on Dutch Wikipedia (T425509)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2414047 wikitext text/x-wiki ghzoh7m9d17rjhhqlfg2i1p4gnfl7t8 2414048 2414047 2026-05-14T20:14:40Z Stashbot 7414 jsn@deploy1003: kgraessle, jsn: Continuing with deployment 2414048 wikitext text/x-wiki hj46qhk8i006jw9khk7nckpmqwe7em4 2414049 2414048 2026-05-14T20:18:18Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1286.eqiad.wmnet with reason: host reimage 2414049 wikitext text/x-wiki kzralnqcal45bfkwhi1n5kzwtx75jz1 2414050 2414049 2026-05-14T20:19:02Z Stashbot 7414 jsn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1192921|Enable AutoModerator on Italian Wikipedia (T405152)]], [[gerrit:1286974|Enable AutoModerator on Albanian Wikipedia (T420450)]], [[gerrit:1286975|Enable AutoModerator on Dutch Wikipedia (T425509)]] (duration: 07m 48s) 2414050 wikitext text/x-wiki kvv46ljpkwrticg9x94z07fibp9cxmo 2414051 2414050 2026-05-14T20:19:46Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1287.eqiad.wmnet with OS bookworm 2414051 wikitext text/x-wiki ow8z4lpo4vdtxnxn3vp9r82t524hyl2 2414052 2414051 2026-05-14T20:23:28Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1286.eqiad.wmnet with reason: host reimage 2414052 wikitext text/x-wiki jbf5hd0auk8gvfwaoljoey88302c689 2414053 2414052 2026-05-14T20:25:17Z Stashbot 7414 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1287002|Disable wgNewUserMessageOnAutoCreate on all WMF wikis (T426206)]] 2414053 wikitext text/x-wiki sunwo6a8p35w5obkuad6u5i0871xkl1 2414054 2414053 2026-05-14T20:27:07Z Stashbot 7414 cjming@deploy1003: cjming, neriah: Backport for [[gerrit:1287002|Disable wgNewUserMessageOnAutoCreate on all WMF wikis (T426206)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2414054 wikitext text/x-wiki rv6oqripuhifjh2wnqxtlbbzz2oq0ll 2414055 2414054 2026-05-14T20:28:53Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1289.eqiad.wmnet with OS bookworm 2414055 wikitext text/x-wiki cjv1076sdl2kwh8a0d8oul2zj8tc0jq 2414056 2414055 2026-05-14T20:29:11Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1289.eqiad.wmnet with OS bookworm 2414056 wikitext text/x-wiki i29dtpjlcdtgsyqjx6x4irnb4kinuze 2414057 2414056 2026-05-14T20:29:47Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1289.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414057 wikitext text/x-wiki k8iswzd1empec9t42hwsl6v7sqdp2ch 2414058 2414057 2026-05-14T20:31:27Z Stashbot 7414 cjming@deploy1003: cjming, neriah: Continuing with deployment 2414058 wikitext text/x-wiki oz653ywka9j91ndswzef823lr3p1n06 2414059 2414058 2026-05-14T20:31:34Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1289.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414059 wikitext text/x-wiki 1j4prf1fleyeb96s31dc63j7mp05x3f 2414060 2414059 2026-05-14T20:35:36Z Stashbot 7414 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1287002|Disable wgNewUserMessageOnAutoCreate on all WMF wikis (T426206)]] (duration: 10m 18s) 2414060 wikitext text/x-wiki en6m90mjun2uaju0hacr1z2ixnuvkux 2414061 2414060 2026-05-14T20:35:40Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1287.eqiad.wmnet with reason: host reimage 2414061 wikitext text/x-wiki sfdylb51jtfxqx761vkxd1kvufw1ca1 2414062 2414061 2026-05-14T20:38:47Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1287.eqiad.wmnet with reason: host reimage 2414062 wikitext text/x-wiki pnlrmlov4pck757t8mgg6ebqwk4ae9i 2414063 2414062 2026-05-14T20:40:35Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2414063 wikitext text/x-wiki jhzqbpjhn895kd2sv0i0yi3zfc1t360 2414064 2414063 2026-05-14T20:43:16Z Stashbot 7414 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2414064 wikitext text/x-wiki d3euxw81lhnucgk1hz2rbmqlxk9dg6s 2414065 2414064 2026-05-14T20:43:43Z Stashbot 7414 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1287427|Simplewiki: include article wizard in AG experiment (T426278)]] 2414065 wikitext text/x-wiki a2ejvef8lat2wvfxn98xa093qcxp0qz 2414066 2414065 2026-05-14T20:45:32Z Stashbot 7414 sbisson@deploy1003: sbisson: Backport for [[gerrit:1287427|Simplewiki: include article wizard in AG experiment (T426278)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2414066 wikitext text/x-wiki jo09wm6elhh594o06tizmuf3m9gzc6s 2414067 2414066 2026-05-14T20:46:36Z Stashbot 7414 sbisson@deploy1003: sbisson: Continuing with deployment 2414067 wikitext text/x-wiki dw4alwt3ljmqf5p0vm7yyx1pius5ecx 2414068 2414067 2026-05-14T20:50:46Z Stashbot 7414 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1287427|Simplewiki: include article wizard in AG experiment (T426278)]] (duration: 07m 03s) 2414068 wikitext text/x-wiki fsg6wa7j4dsqenplll3pf65q7p3skdl 2414069 2414068 2026-05-14T20:55:57Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2414069 wikitext text/x-wiki ite7949pqzvfe8q46wf1ehxbjo21ge2 2414070 2414069 2026-05-14T20:56:25Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2414070 wikitext text/x-wiki hjjmjjz15faodby02ew0ioa3ja2ovih 2414071 2414070 2026-05-14T20:56:26Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1287.eqiad.wmnet with OS bookworm 2414071 wikitext text/x-wiki jfsd3a0t2o3clfra4ov2sue01kt64sy 2414074 2414071 2026-05-14T21:10:03Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1287479|Enable hCaptcha for account creation API on group 0 wiki's]], [[gerrit:1287484|Remove DynamicPageList from legalteamwiki as unused]] 2414074 wikitext text/x-wiki sj2rys168i4eavx4ygo7tzohfvw51xo 2414076 2414074 2026-05-14T21:11:50Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz, seddon: Backport for [[gerrit:1287479|Enable hCaptcha for account creation API on group 0 wiki's]], [[gerrit:1287484|Remove DynamicPageList from legalteamwiki as unused]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2414076 wikitext text/x-wiki dbqr09sx3lo8mv5vyhmin2gdskmepta 2414077 2414076 2026-05-14T21:12:27Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz, seddon: Continuing with deployment 2414077 wikitext text/x-wiki 5cxn8l4nf0gbc8xv1rm6r5z4puqyl5c 2414078 2414077 2026-05-14T21:15:09Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2414078 wikitext text/x-wiki 3k26gpzkkrjbmu3zyeeouy3bh77yau8 2414079 2414078 2026-05-14T21:15:11Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1286.eqiad.wmnet with OS bookworm 2414079 wikitext text/x-wiki 0inhiwfckrzz5ttvytxb7curhho9ceq 2414080 2414079 2026-05-14T21:16:36Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1287479|Enable hCaptcha for account creation API on group 0 wiki's]], [[gerrit:1287484|Remove DynamicPageList from legalteamwiki as unused]] (duration: 06m 33s) 2414080 wikitext text/x-wiki qt4hl0q4xsoyy73ln3xxlyn3sj5e2qt 2414084 2414080 2026-05-14T21:24:35Z Stashbot 7414 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1287485|Disable Reading Lists survey for Wikipedias (T421776)]] 2414084 wikitext text/x-wiki dovg1od0p9t9k15ahe4b197cgvi5rls 2414085 2414084 2026-05-14T21:26:24Z Stashbot 7414 jdrewniak@deploy1003: jdrewniak: Backport for [[gerrit:1287485|Disable Reading Lists survey for Wikipedias (T421776)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2414085 wikitext text/x-wiki baqfoda8l8s3j596iepsx8op16bl4ln 2414086 2414085 2026-05-14T21:29:38Z Stashbot 7414 jdrewniak@deploy1003: jdrewniak: Continuing with deployment 2414086 wikitext text/x-wiki hxwqdvmsj5cs7s5vfwcond44inu9qab 2414087 2414086 2026-05-14T21:33:51Z Stashbot 7414 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1287485|Disable Reading Lists survey for Wikipedias (T421776)]] (duration: 09m 15s) 2414087 wikitext text/x-wiki 1xxxqdo3mhova63gdw6jux6q1b9s7tb 2414088 2414087 2026-05-14T21:40:07Z Stashbot 7414 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1287488|Share Highlight: overdraw photo on share card canvas (T426344)]] 2414088 wikitext text/x-wiki rm2x7n9716ctyqqiktivmwlun7fu1rx 2414089 2414088 2026-05-14T21:41:53Z Stashbot 7414 egardner@deploy1003: egardner: Backport for [[gerrit:1287488|Share Highlight: overdraw photo on share card canvas (T426344)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2414089 wikitext text/x-wiki qb25u5ibkixmf4w9zf1o3kiymyjv2cz 2414090 2414089 2026-05-14T21:43:12Z Stashbot 7414 egardner@deploy1003: egardner: Continuing with deployment 2414090 wikitext text/x-wiki ghczohihmqbnb2ct4hu0er1ojauyq9e 2414092 2414090 2026-05-14T21:47:21Z Stashbot 7414 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1287488|Share Highlight: overdraw photo on share card canvas (T426344)]] (duration: 07m 14s) 2414092 wikitext text/x-wiki amkctxd9ihi87qvsgojcz44hakiwvza 2414102 2414092 2026-05-14T23:07:18Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2414102 wikitext text/x-wiki h5kr446xdj0wb83kpch1g5v75f21hvm 2414103 2414102 2026-05-14T23:10:06Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2414103 wikitext text/x-wiki czfcjd8f1377skenjvl6b193iyrqzn5 2414104 2414103 2026-05-14T23:11:16Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1289 2414104 wikitext text/x-wiki 9gi8jhpn0l6r26hsajfm9lx43jbn5w1 2414105 2414104 2026-05-14T23:12:55Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1289 2414105 wikitext text/x-wiki az7oevvolebgye6be69dxtks4dugrl5 2414106 2414105 2026-05-14T23:13:35Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1289.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414106 wikitext text/x-wiki t6fesnnlv4hynulpexcpdfnsyfaewoh 2414107 2414106 2026-05-14T23:14:46Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1289.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414107 wikitext text/x-wiki 1ahf4dcjd70pduzuicwblske33k2bn5 2414108 2414107 2026-05-14T23:24:20Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1289.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414108 wikitext text/x-wiki 54dyw43hyee3e32am4v8qx8hogzorzp 2414109 2414108 2026-05-14T23:26:19Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1289.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414109 wikitext text/x-wiki gcwlq9g1k3rsblo67mtu7uh7riaphi5 2414110 2414109 2026-05-14T23:27:29Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1289.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414110 wikitext text/x-wiki 88c29fqv3r58564wdsepcmcld1lpgix 2414111 2414110 2026-05-14T23:30:02Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1289.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414111 wikitext text/x-wiki 31hir9p1nldf7wsimx709ibpvuxhzhf 2414112 2414111 2026-05-14T23:34:51Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1290.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414112 wikitext text/x-wiki palc7o4ur2058064vgau4qvqkidmwhb 2414113 2414112 2026-05-14T23:39:08Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1290.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414113 wikitext text/x-wiki 9x0dxzcj4pchiyoyfmte2uiac56s8tz 2414114 2414113 2026-05-14T23:49:50Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1290.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414114 wikitext text/x-wiki 6o1zhsba5fsy23nd7sjv9z9cgkzwzjz 2414115 2414114 2026-05-14T23:54:05Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1290.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414115 wikitext text/x-wiki nqk6vl61ghttaevjafkqitczly3aef3 2414116 2414115 2026-05-14T23:55:49Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host db1290 2414116 wikitext text/x-wiki eonszhpb3k0x5fsx9x9ugamnmu3fgsx 2414117 2414116 2026-05-14T23:57:01Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1290 2414117 wikitext text/x-wiki ekicx4xg11z4z4uhwj865whuum44wfo 2414118 2414117 2026-05-14T23:57:26Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1290.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414118 wikitext text/x-wiki 7859e8wg39mqa5a1g370essiq02l2dt 2414119 2414118 2026-05-14T23:58:56Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1290.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414119 wikitext text/x-wiki 1ve8vczn7dq8v4zhvchjoji9l2v35tv 2414120 2414119 2026-05-14T23:59:22Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1290.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414120 wikitext text/x-wiki 9rx7ip3lrzs2m2uf0iavkvvybylni84 2414122 2414120 2026-05-15T00:01:37Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1290.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414122 wikitext text/x-wiki i2ai0f0wlx2jcq6gnw2ycbhdx1jp7wt 2414124 2414122 2026-05-15T00:02:56Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1289.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414124 wikitext text/x-wiki qre1q0app52eo5607bztbv7e51swyif 2414125 2414124 2026-05-15T00:14:02Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1290.eqiad.wmnet with OS bookworm 2414125 wikitext text/x-wiki amorlktcluoom5dizc5aqnztj1r92fo 2414126 2414125 2026-05-15T00:14:43Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1289.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414126 wikitext text/x-wiki qveewywi57ljgyl13w7cy4307xqtiea 2414127 2414126 2026-05-15T00:39:11Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1290.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414127 wikitext text/x-wiki 98dnncqzue6r1os538d6vzmkz9wq1pr 2414128 2414127 2026-05-15T00:42:43Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1290.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414128 wikitext text/x-wiki 4nxs7du6c9ibppyrih8y6atytu87730 2414129 2414128 2026-05-15T00:43:28Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1290.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414129 wikitext text/x-wiki ij7z3h2iyaux64jh2a4d4whfigpkslo 2414130 2414129 2026-05-15T00:46:05Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1289.eqiad.wmnet with reason: host reimage 2414130 wikitext text/x-wiki qratd9y9rh46wwfbrg3rs290kz40cec 2414131 2414130 2026-05-15T00:49:47Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1289.eqiad.wmnet with reason: host reimage 2414131 wikitext text/x-wiki b44dno5up0ge0brpl9dpb94fpdysro9 2414132 2414131 2026-05-15T00:50:38Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1290.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414132 wikitext text/x-wiki 5ct2jc61qm5dklh6hk2monpk840gx8f 2414133 2414132 2026-05-15T01:06:40Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2414133 wikitext text/x-wiki 8awap638tmq4jb5rv65ut3grpzrorga 2414134 2414133 2026-05-15T01:10:15Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" 2414134 wikitext text/x-wiki f7uqw5jn4u0wmuvtix75olj82pvrf3t 2414135 2414134 2026-05-15T01:10:17Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1289.eqiad.wmnet with OS bookworm 2414135 wikitext text/x-wiki olcs0471gqlt0w154rytx6z3vw6q6n7 2414136 2414135 2026-05-15T01:14:17Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host db1290.eqiad.wmnet with OS bookworm 2414136 wikitext text/x-wiki p3w1kg2ypfb25crdoq1ql3vqhmsg5jp 2414137 2414136 2026-05-15T02:00:38Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2414137 wikitext text/x-wiki 08i0vn06eykzjisrmvxpgydb9y2kg3k 2414138 2414137 2026-05-15T02:06:49Z Stashbot 7414 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1290.eqiad.wmnet with OS bookworm 2414138 wikitext text/x-wiki mnommisxj2ysbegsy48cn4tf7pkc9il 2414139 2414138 2026-05-15T02:07:29Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 50s) 2414139 wikitext text/x-wiki 3r4rv9yq0i1dstcyim5x94eqjsfa7qa 2414140 2414139 2026-05-15T02:31:40Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.dns.netbox 2414140 wikitext text/x-wiki iv4gqhus8mqmfu9ikiwngjrxuccazic 2414141 2414140 2026-05-15T02:34:21Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2414141 wikitext text/x-wiki gootuv4nw4rs3fz4whb8mo4ajiudyye 2414142 2414141 2026-05-15T02:34:58Z Stashbot 7414 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1290.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414142 wikitext text/x-wiki be0zm9s3kz8g6nchlx31dm69tf4xiby 2414143 2414142 2026-05-15T02:37:16Z Stashbot 7414 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1290.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2414143 wikitext text/x-wiki l4ozkz8sedof48wktdyo1l0bivd2q4n 2414147 2414143 2026-05-15T07:10:59Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2414147 wikitext text/x-wiki 9t9mg945ya3o3n9e799fpwt392whi3n 2414148 2414147 2026-05-15T07:39:50Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.powercycle for host sretest2010 2414148 wikitext text/x-wiki fly0pzjndp1h4foe6aw761dfgubo8n3 2414149 2414148 2026-05-15T07:41:38Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.powercycle (exit_code=0) for host sretest2010 2414149 wikitext text/x-wiki 0cbrui4wta1ry4xurthbjdqn3gws1ut 2414150 2414149 2026-05-15T07:42:13Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie 2414150 wikitext text/x-wiki al6ptth8m2zlm1oc6kjzpr55gcss82c 2414151 2414150 2026-05-15T07:54:54Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2414151 wikitext text/x-wiki gbbfo6hvw8ej52ai7cdaxoahgoh0vdw 2414152 2414151 2026-05-15T07:54:57Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2414152 wikitext text/x-wiki oh069zgked9ig8o3462x8jn1q17qsyd 2414153 2414152 2026-05-15T07:55:45Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be2064 2414153 wikitext text/x-wiki kg3rl93hk7m0336j032h7hg7nucf6yg 2414154 2414153 2026-05-15T07:55:56Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2414154 wikitext text/x-wiki hb4mvh4bfku23agsxxgzdrqos43i4bg 2414155 2414154 2026-05-15T07:56:00Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2414155 wikitext text/x-wiki lkudlgw0xzao6e1b5qvjtekz08awd2u 2414156 2414155 2026-05-15T07:58:57Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2414156 wikitext text/x-wiki ra6gchz1k9mb7g7req20igr6ytpzbtz 2414157 2414156 2026-05-15T07:59:00Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2414157 wikitext text/x-wiki 75ij4iqhd7ooba8e6kke2t705bqp0c5 2414158 2414157 2026-05-15T08:03:10Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2414158 wikitext text/x-wiki kc2r0nfpn25sh6xsd4zezmqesb55d8x 2414159 2414158 2026-05-15T08:03:14Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2414159 wikitext text/x-wiki 6e7j41wcnzwgw7xphoidh6ujgemhcgz 2414160 2414159 2026-05-15T08:05:46Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2414160 wikitext text/x-wiki trv5nox77azix0pfebbnemfm5klzhsr 2414161 2414160 2026-05-15T08:16:20Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART 2414161 wikitext text/x-wiki qfuxzj5hzfmw7heeekvu9tpzf0ourne 2414162 2414161 2026-05-15T08:17:57Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-logging1006.eqiad.wmnet with OS trixie 2414162 wikitext text/x-wiki t7j7es4mrfudgsb6ayz01ioazgpe4vf 2414163 2414162 2026-05-15T08:28:02Z Stashbot 7414 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kafka-logging1006.eqiad.wmnet with OS trixie 2414163 wikitext text/x-wiki a9m3di27dkg3467gha3996gtkia549e 2414164 2414163 2026-05-15T08:41:28Z Stashbot 7414 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be2064 2414164 wikitext text/x-wiki qsb5i9oub3s8x70yhhwvms19gsq0kj4 2414165 2414164 2026-05-15T08:41:45Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.swift.convert-disks for host ms-be2065 2414165 wikitext text/x-wiki hl19kj5r40itj5zg7bx1k6pghye57rz 2414167 2414165 2026-05-15T08:54:23Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Set db2220 with weight 0 T426380', diff saved to https://phabricator.wikimedia.org/P92551 and previous config saved to /var/cache/conftool/dbconfig/20260515-085420-marostegui.json 2414167 wikitext text/x-wiki miuj745fy5awbmga525lc10zpy6vt9y 2414168 2414167 2026-05-15T08:54:28Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s7 T426380 2414168 wikitext text/x-wiki n1htqy1dvouoeb2id1ohdahshdy1h74 2414170 2414168 2026-05-15T08:56:44Z Stashbot 7414 marostegui: Starting s7 codfw failover from db2218 to db2220 - T426380 2414170 wikitext text/x-wiki 3l152e702xbj62mc6kx4sldp137y1d0 2414171 2414170 2026-05-15T08:58:39Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db2220 to s7 primary T426380', diff saved to https://phabricator.wikimedia.org/P92552 and previous config saved to /var/cache/conftool/dbconfig/20260515-085836-marostegui.json 2414171 wikitext text/x-wiki 32dzy8bxlhofmzp45kgs7h04zz4yort 2414172 2414171 2026-05-15T09:00:02Z Stashbot 7414 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2218 T426380', diff saved to https://phabricator.wikimedia.org/P92553 and previous config saved to /var/cache/conftool/dbconfig/20260515-090000-marostegui.json 2414172 wikitext text/x-wiki jq09rf1psork5qfw73cncem7chxl2ad 2414173 2414172 2026-05-15T09:01:45Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2064.codfw.wmnet with OS bullseye 2414173 wikitext text/x-wiki ifbjvgn1xtz31nvzoqmq41103kl1xzp 2414174 2414173 2026-05-15T09:02:05Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2064 2414174 wikitext text/x-wiki 4k84ay2o0ac6iaf5ftehuwjv4to89sg 2414175 2414174 2026-05-15T09:02:15Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.netbox 2414175 wikitext text/x-wiki tjs5zc6r7v6jbic6zqzbfjlgwz73iiy 2414176 2414175 2026-05-15T09:03:10Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2414176 wikitext text/x-wiki jk24czp22934wltbn7f5sraesmosk3g 2414177 2414176 2026-05-15T09:06:15Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2064 - mvernon@cumin2002" 2414177 wikitext text/x-wiki gg70vu5fb4a3i4d484xlcc9yy9q0b83 2414178 2414177 2026-05-15T09:06:21Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2064 - mvernon@cumin2002" 2414178 wikitext text/x-wiki ihb4hixauw5y7ubi96qncnn1c8jkuyr 2414179 2414178 2026-05-15T09:06:22Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2414179 wikitext text/x-wiki rk7kh4ybmp3k7tparxva748xx4rdsb3 2414180 2414179 2026-05-15T09:06:24Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2064.codfw.wmnet 56.32.192.10.in-addr.arpa 6.5.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2414180 wikitext text/x-wiki f05809lcjeprgo0o5umhb3ad3ub5v71 2414181 2414180 2026-05-15T09:06:25Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2064.codfw.wmnet 56.32.192.10.in-addr.arpa 6.5.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2414181 wikitext text/x-wiki cpdxh9pj52lwqcr6tcsrs0mj5jod93d 2414182 2414181 2026-05-15T09:06:27Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2064 2414182 wikitext text/x-wiki eagzplwr09826zl6x88bry6s4ufktne 2414183 2414182 2026-05-15T09:08:15Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2064 2414183 wikitext text/x-wiki slf1wdled6lewxyibsf4mv0ezgjc8gc 2414184 2414183 2026-05-15T09:08:17Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2064 2414184 wikitext text/x-wiki 7ytptqd3tfglewjfx0o8aa6j5a6tul8 2414185 2414184 2026-05-15T09:08:58Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on db2218.codfw.wmnet with reason: Host crashed T426383 2414185 wikitext text/x-wiki t6r6czj2iecobl2jgwgbe48sk02d30l 2414186 2414185 2026-05-15T09:10:53Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie 2414186 wikitext text/x-wiki 7453gbiiex4n7xuabl63tv7s0uyfp4q 2414187 2414186 2026-05-15T09:30:02Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply 2414187 wikitext text/x-wiki dtowfzsceur4abtmm0yg909daubnmnd 2414188 2414187 2026-05-15T09:30:13Z Stashbot 7414 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be2065 2414188 wikitext text/x-wiki 9ycrm9yu61cqi7vzo1hxm8k21oya64i 2414189 2414188 2026-05-15T09:30:17Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply 2414189 wikitext text/x-wiki rxctr6m1uqkjakwmlvu0uk8mqzkcwsg 2414190 2414189 2026-05-15T09:30:42Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply 2414190 wikitext text/x-wiki gl2d230cci4xvzdidg35h43kvx5b9tu 2414191 2414190 2026-05-15T09:32:18Z Stashbot 7414 topranks: Migrate cr4-ulsfo link to asw1-23-ulsfo to tagged interface T424611 2414191 wikitext text/x-wiki nmicnsij3ryurx16g09s8nurq9qrlb4 2414192 2414191 2026-05-15T09:32:26Z Stashbot 7414 mvernon@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ms-be2064.codfw.wmnet with OS bullseye 2414192 wikitext text/x-wiki qxie1x4u9sdsejvhggo2b40gs6cnqka 2414193 2414192 2026-05-15T09:33:13Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2414193 wikitext text/x-wiki jdssr8qgkaq2hqg5syefowt6gatv2p4 2414194 2414193 2026-05-15T09:48:04Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie 2414194 wikitext text/x-wiki 9vt9f2ifhn00dqliruvybagfd1k54sf 2414195 2414194 2026-05-15T09:48:21Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2414195 wikitext text/x-wiki 2d89ze3bamf6j0t9tzwg0qyq7adw7cb 2414196 2414195 2026-05-15T09:49:40Z Stashbot 7414 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply 2414196 wikitext text/x-wiki fqahhnmthb689orn5wwwj32q64pk4jr 2414197 2414196 2026-05-15T09:56:07Z Stashbot 7414 topranks: Migrate cr3-ulsfo link to asw1-22-ulsfo to tagged interface T424611 2414197 wikitext text/x-wiki j8l7u0o8gayih32u44mlslkq6hrp2j7 2414198 2414197 2026-05-15T10:00:58Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie 2414198 wikitext text/x-wiki n6ic5qtnhavlbgu0dhq7wxqaym4oaxy 2414199 2414198 2026-05-15T10:01:01Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T426298 2414199 wikitext text/x-wiki ntas6410ujas5lvjenig8bv2oexrdvb 2414200 2414199 2026-05-15T10:01:02Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2414200 wikitext text/x-wiki hep08zx6gw3nz3145g6g441pjjrki04 2414201 2414200 2026-05-15T10:04:15Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2064.codfw.wmnet with OS bullseye 2414201 wikitext text/x-wiki lg9mwv1ovxqgs45hzwhw7owyfdpve6u 2414202 2414201 2026-05-15T10:04:41Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2414202 wikitext text/x-wiki 3u9809zc2usedeghgm8yqahhfilu6wb 2414203 2414202 2026-05-15T10:10:21Z Stashbot 7414 topranks: Migrate ulsfo cr<->cr traffic to use path via switches not direct link T424611 2414203 wikitext text/x-wiki mgtdzjbl5l0sc9wvohowzy6nkhs9gqe 2414204 2414203 2026-05-15T10:12:24Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: modify entries for ulsfo router interfaces - cmooney@cumin1003" 2414204 wikitext text/x-wiki n87vfmjfs5r3n33gcmoh2rj7gvt6rwj 2414205 2414204 2026-05-15T10:12:29Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: modify entries for ulsfo router interfaces - cmooney@cumin1003" 2414205 wikitext text/x-wiki c12etzxkzpvv2huyjtvkixkfrduyjlz 2414206 2414205 2026-05-15T10:12:31Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2414206 wikitext text/x-wiki 394gwtxir046d8wzmgzwez8tcjunv6g 2414207 2414206 2026-05-15T10:20:50Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2064.codfw.wmnet with reason: host reimage 2414207 wikitext text/x-wiki e2orcqjs3hacbooadr71b2zfma5347j 2414208 2414207 2026-05-15T10:22:43Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2414208 wikitext text/x-wiki c8m1rbg1yb6pf02r3omrakgmviut3ql 2414209 2414208 2026-05-15T10:22:47Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2414209 wikitext text/x-wiki pvya7op8w0itz7pwi0tne17iznej6zh 2414210 2414209 2026-05-15T10:23:56Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2414210 wikitext text/x-wiki bzjhcnadyu1tfd4h8vts0nyr2efuf57 2414211 2414210 2026-05-15T10:24:00Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2414211 wikitext text/x-wiki ekg6s59fr3j2oh5qmv633w2mole0cq8 2414212 2414211 2026-05-15T10:24:57Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2064.codfw.wmnet with reason: host reimage 2414212 wikitext text/x-wiki ip6tdzlvfzrvvt9k8ehuum42tq5ds02 2414213 2414212 2026-05-15T10:28:35Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage 2414213 wikitext text/x-wiki 0gwxy4a1x9eamf0gca6gqirm142mt4j 2414215 2414213 2026-05-15T10:31:01Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2414215 wikitext text/x-wiki 806cm44b5kw8i67rlc5dd2gedd41edz 2414217 2414215 2026-05-15T10:31:04Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2414217 wikitext text/x-wiki rigvjfpty47fw4cyrm2eq854yqsoa56 2414219 2414217 2026-05-15T10:34:00Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage 2414219 wikitext text/x-wiki dsrhpqyr64ybeh3kzxjnohxbsfffv70 2414220 2414219 2026-05-15T10:35:58Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2065.codfw.wmnet with OS bullseye 2414220 wikitext text/x-wiki i7e6og97kf0ambzf769vkz9n7ddqfif 2414221 2414220 2026-05-15T10:36:17Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2065 2414221 wikitext text/x-wiki thxjsgf987eo6umbwtz1i3p0fmre0t8 2414222 2414221 2026-05-15T10:36:39Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.netbox 2414222 wikitext text/x-wiki kih22sbsme64txxkxyxbvojkdmj8ena 2414223 2414222 2026-05-15T10:40:38Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2065 - mvernon@cumin2002" 2414223 wikitext text/x-wiki 7hfe82vg7ks444ck2fmflk5jmbepk1k 2414224 2414223 2026-05-15T10:40:44Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2065 - mvernon@cumin2002" 2414224 wikitext text/x-wiki p9nof5ap4cbo4eyr9u1bfcii76s2348 2414225 2414224 2026-05-15T10:40:45Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2414225 wikitext text/x-wiki et9yfmz69clwdyeaxk4pwjbhymhyj9i 2414226 2414225 2026-05-15T10:40:46Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2065.codfw.wmnet 167.48.192.10.in-addr.arpa 7.6.1.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2414226 wikitext text/x-wiki 13hyyhf609twv05dkox9mcbbtiydazg 2414227 2414226 2026-05-15T10:40:48Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2065.codfw.wmnet 167.48.192.10.in-addr.arpa 7.6.1.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors 2414227 wikitext text/x-wiki 1bot46goxerx24lmq7v1waxtt6pbmwp 2414228 2414227 2026-05-15T10:40:50Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2065 2414228 wikitext text/x-wiki dkd3qg7i2heg0v933tmqilmy1sbi140 2414229 2414228 2026-05-15T10:41:44Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2065 2414229 wikitext text/x-wiki 5fpf35hgntbafob8dn0i4qc63sr7pku 2414230 2414229 2026-05-15T10:41:46Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2065 2414230 wikitext text/x-wiki rznzm89ic18w10pneqoxeeadrzlogot 2414231 2414230 2026-05-15T10:42:57Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie 2414231 wikitext text/x-wiki tojdf22efkgx9nxeh0wad1kgmatlhoj 2414232 2414231 2026-05-15T10:43:25Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie 2414232 wikitext text/x-wiki 216ubj148a07qw8tpl9stjken81fyek 2414233 2414232 2026-05-15T10:46:10Z Stashbot 7414 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie 2414233 wikitext text/x-wiki dffc4j4ckio7jl7mrwdcvcma7tqreei 2414234 2414233 2026-05-15T10:52:20Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2414234 wikitext text/x-wiki 9mi6tuo6rduugrvus71h6jem5lzzuql 2414235 2414234 2026-05-15T10:52:24Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2414235 wikitext text/x-wiki nchtbrs24mdaxhhedpv7xpdk1doddn4 2414236 2414235 2026-05-15T10:55:57Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2064.codfw.wmnet with OS bullseye 2414236 wikitext text/x-wiki dqhdoiq4cgmfy6teo2zw8y0y1d3b3mx 2414237 2414236 2026-05-15T10:59:32Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2065.codfw.wmnet with reason: host reimage 2414237 wikitext text/x-wiki bqdn6pyhi0dtwmywb0p266ysfn3xejy 2414238 2414237 2026-05-15T11:04:17Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2065.codfw.wmnet with reason: host reimage 2414238 wikitext text/x-wiki tc69i1pjkirpjvpgsc46anv0ghgmexl 2414240 2414238 2026-05-15T11:10:39Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T426298 2414240 wikitext text/x-wiki 9dpnej9ga9smgvz9bdbd8b2fy4jxmt0 2414241 2414240 2026-05-15T11:14:51Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2414241 wikitext text/x-wiki 77vwqzv8zseogsn41hpclrx7sajiwbi 2414242 2414241 2026-05-15T11:24:12Z Stashbot 7414 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2065.codfw.wmnet with OS bullseye 2414242 wikitext text/x-wiki hzf927o6ha8omqzkhxrgpk7nf16lj1b 2414244 2414242 2026-05-15T11:34:30Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2414244 wikitext text/x-wiki p948j27xqhunvjl5eoufxddgrtxeonz 2414245 2414244 2026-05-15T11:34:33Z Stashbot 7414 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply 2414245 wikitext text/x-wiki m33aq74kn2j7w5vlmccpcdmymrv2afj 2414249 2414245 2026-05-15T11:58:37Z Stashbot 7414 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-fe2009.codfw.wmnet 2414249 wikitext text/x-wiki lxe3q6csib878ggvoig3p816y3zxwy2 2 443136 2414146 1809409 2026-05-15T05:30:35Z ZI Jony 13427 /* */ + 2414146 wikitext text/x-wiki kgticvnxoh53969igejzphp9qg6olqy 498 443628 2414014 2413101 2026-05-14T18:48:49Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25878752565 (https://github.com/cluebotng/component-configs/commits/21e928fa1870ddaf5fae15afc6f92aa3cb3fb970) 2414014 wikitext text/x-wiki ruaec0dfhwl1fuy7slth6fg93vjl9m0 498 443947 2413851 2404117 2026-05-14T14:24:04Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 5e96e1a7f9 (l10n updates: ko, vi) 2413851 wikitext text/x-wiki cz23gnpqsekqmuzwhq65dwi4c8v03ep 0 446462 2414100 2380172 2026-05-14T23:00:24Z Novem Linguae 23682 /* Set global batphone */ fix double bold 2414100 wikitext text/x-wiki gyc7j6tjbr4j9s3hktaca4r6xv3r7ml 2414101 2414100 2026-05-14T23:00:44Z Novem Linguae 23682 /* WMCS Usage */ fix double line break 2414101 wikitext text/x-wiki fkshzedyq6desccy5mw40ko9bz0ejtz 0 446795 2413729 2413678 2026-05-14T12:59:46Z CWilliams-WMF 55316 /* Remove all other puppet entries */ updated example patch and notes 2413729 wikitext text/x-wiki lxyccxx7sjqyn8lqf9cqgnf8ljb1ufp 498 447034 2413849 2388318 2026-05-14T14:23:08Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 9c42483353 (l10n updates: ko) 2413849 wikitext text/x-wiki ewo0s4tywr7w73kb9stsmxz9amc6b9l 0 449160 2413691 2413595 2026-05-14T12:20:18Z Dexbot 30554 Bot: Updating the report 2413691 wikitext text/x-wiki flbwllbnnlgqv8dc5dygete3fbx65kn 2413712 2413691 2026-05-14T12:50:22Z Dexbot 30554 Bot: Updating the report 2413712 wikitext text/x-wiki ommq5oceaq45ja7hofxtl3c3g6yozm2 2413795 2413712 2026-05-14T13:53:31Z Dexbot 30554 Bot: Updating the report 2413795 wikitext text/x-wiki ml5zpo4kdgfd1tgyfoab3wlnw431sd0 2413844 2413795 2026-05-14T14:20:36Z Dexbot 30554 Bot: Updating the report 2413844 wikitext text/x-wiki fwuokzcxd1zhwn3qjyhp1ubfxsc36f6 2414121 2413844 2026-05-15T00:00:49Z Dexbot 30554 Bot: Updating the report 2414121 wikitext text/x-wiki ne6iva8omyrk8x03d0algbmb7rcj16e 2414169 2414121 2026-05-15T08:55:55Z Dexbot 30554 Bot: Updating the report 2414169 wikitext text/x-wiki 7a0td0bvyfpfya0xfw20vvhqto2t2zr 12 451671 2413923 2353270 2026-05-14T15:33:45Z Taavi-WMF 41365 primarly link to tech blog instead of phame 2413923 wikitext text/x-wiki hdwxlpxat1lthlt46hx5np0mqp1vwwm 0 452153 2414094 2339398 2026-05-14T22:04:34Z BLiviero-WMF 53603 Removed Superset, added lima-kilo and Toolforge API Gateway (more details forthcoming) 2414094 wikitext text/x-wiki rg4w8qkh8ty73riwbdz77dd9iarmpth 498 452890 2414015 2413102 2026-05-14T18:49:28Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25878752531 (https://github.com/cluebotng/component-configs/commits/21e928fa1870ddaf5fae15afc6f92aa3cb3fb970) 2414015 wikitext text/x-wiki aiwbc72tvsnosrs3bnic32wsh643zfp 116 453906 2414072 2413304 2026-05-14T21:00:49Z Gitlabaccountapprovalbot 37332 @nehemienathan was approved. 2414072 wikitext text/x-wiki 8psf6l6xsa85t8v24usn9dguhzv6qvk 0 458728 2414002 2298491 2026-05-14T17:35:11Z AStein-WMF 39485 update hostnames 2414002 wikitext text/x-wiki 8i3yhotrq2yryqf3wmioqj0qs8qnnra 2414003 2414002 2026-05-14T17:37:05Z AStein-WMF 39485 2414003 wikitext text/x-wiki 59l3a1w9xoaar4fg5higd9us7phuk7s 2414004 2414003 2026-05-14T17:37:20Z AStein-WMF 39485 2414004 wikitext text/x-wiki 3y79bacjlfp8glfhcvidsx9gzhkh6pf 0 458868 2413942 2384089 2026-05-14T15:54:33Z FNegri-WMF 32595 /* Components */ typo 2413942 wikitext text/x-wiki n5dycrha1towuwlpowvqwkbthzvofvf 498 459243 2414010 2413098 2026-05-14T18:29:37Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25877799217 (https://github.com/cluebotng/component-configs/commits/df1444522160198ee3ab31965936679052be618b) 2414010 wikitext text/x-wiki c9qs6qd0pieiwleki6509wwfns34w0c 2414073 2414010 2026-05-14T21:01:41Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885319112 (https://github.com/cluebotng/component-configs/commits/8369d11ffe6281d9c21f2e63e23bd03ac30ca98a) 2414073 wikitext text/x-wiki 2hrau3wu3w5ihoaqj7dtetl7v72x767 2414075 2414073 2026-05-14T21:10:37Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885710134 (https://github.com/cluebotng/component-configs/commits/c2d1834d313a784edca516e9f9d4e29fd90a3e5f) 2414075 wikitext text/x-wiki o2kmtpw9v10hjyl5s6rg3rdrmfzib1w 2414091 2414075 2026-05-14T21:47:04Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887423820 (https://github.com/cluebotng/component-configs/commits/55401ec37eaad5c0bffbfbdac85eda0a67eed5c0) 2414091 wikitext text/x-wiki ru6t8qx1s786k1lanx7vfhtxzprjrns 2414093 2414091 2026-05-14T21:52:04Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887641793 (https://github.com/cluebotng/component-configs/commits/a533fb138a23d96ee62535463d21714a7e24fe18) 2414093 wikitext text/x-wiki sgt8bfhzhwhs41981qlyb64zqg5smik 498 459244 2413987 2413086 2026-05-14T17:02:19Z Stashbot 7414 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25873443162 (https://github.com/cluebotng/component-configs/commits/815c3361fd913b83b92ecd2a0d61ae298b921dda) 2413987 wikitext text/x-wiki rq8osqxqqqkntan10a7lp5vbihoczg7 0 459283 2413828 2402873 2026-05-14T14:13:06Z FCeratto-WMF 39104 Add logging tips 2413828 wikitext text/x-wiki l8uc7z6r3vdrl2q44v8cto80dlbh8v8 2413830 2413828 2026-05-14T14:15:16Z FCeratto-WMF 39104 2413830 wikitext text/x-wiki h74xjx31i3b8rccslh18ih89cdxoaqb 0 459290 2413998 2413527 2026-05-14T17:22:20Z AKhatun (WMF) 50461 2413998 wikitext text/x-wiki 5qni2j57ah520c782t6coe3zhsl2grg 0 459699 2413983 2412882 2026-05-14T16:48:18Z Stashbot 7414 ejegg: standalone (IPN listener) SmashPig upgraded from bb833986 to a2a3a015 2413983 wikitext text/x-wiki 61q35i7pax9p7sbrpsdt6bxx8xiuwwe 2414018 2413983 2026-05-14T19:04:51Z Stashbot 7414 ejegg: disabled fundraising scheduled jobs for CiviCRM deployment 2414018 wikitext text/x-wiki byvli3nbegy4obm01vnmox4hkxhmw4b 2414020 2414018 2026-05-14T19:06:56Z Stashbot 7414 ejegg: fundraising civicrm upgraded from 950908ec to e25fa223 2414020 wikitext text/x-wiki qryavh2ajtij6zxe74lpii6c07px2zx 2414036 2414020 2026-05-14T19:29:51Z Stashbot 7414 ejegg: re-enabled fundraising scheduled jobs 2414036 wikitext text/x-wiki hrkh3rj8jw95ubb5t0v6r0ffn46uosw 2414095 2414036 2026-05-14T22:09:23Z Stashbot 7414 ejegg: fundraising scheduled jobs disabled for Civi update 2414095 wikitext text/x-wiki 3z1rdszljzga3v9fcrom1oxutj2y55p 2414096 2414095 2026-05-14T22:11:00Z Stashbot 7414 ejegg: fundraising civicrm upgraded from e25fa223 to 6d8ce7a3 2414096 wikitext text/x-wiki opn8u1xvapz2azsxmso9pn5gj0ymrga 2414097 2414096 2026-05-14T22:12:28Z Stashbot 7414 eileen: cv upgraded from f19e0961 to b8a8dd6a 2414097 wikitext text/x-wiki 8htbr1q3xpow32t7n8vzno76c3pgkqk 2414099 2414097 2026-05-14T22:22:31Z Stashbot 7414 ejegg: fundraising scheduled jobs re-enabled 2414099 wikitext text/x-wiki kkggm0dg1qjodnm1ctw04q08e3zksil 2414123 2414099 2026-05-15T00:02:05Z Stashbot 7414 eileen: civicrm upgraded from 6d8ce7a3 to 6a2258ff 2414123 wikitext text/x-wiki 0ahvey29cwwvoe7ki34ayx8pkmua7ze 0 460052 2413774 2404853 2026-05-14T13:37:28Z MPopov (WMF) 5891 /* Requesting Access */ Details about CustomElevatedAccess role's permissions 2413774 wikitext text/x-wiki rnrtghx3g4l4eqe4ybs81ll6dyeurj3 0 460059 2414025 2406536 2026-05-14T19:17:56Z AStein-WMF 39485 add troubleshooting info to dbt page 2414025 wikitext text/x-wiki 4ft3qobzfjburuzl21n22tr8evw8x74 2 460097 2414243 2406701 2026-05-15T11:27:47Z Effie Mouzeli (WMF) 12880 2414243 wikitext text/x-wiki sug9t6pw4ugpop48wht6sbtfskjd7rr 2414248 2414243 2026-05-15T11:58:12Z Effie Mouzeli (WMF) 12880 2414248 wikitext text/x-wiki 8j8a7q7gc12vmj1pl8c3xcnfbwqcsl7 498 460098 2413957 2410998 2026-05-14T16:06:40Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase by 2 instances 2413957 wikitext text/x-wiki 4hajp3t6hyzdo53n31wnkkorneznjmo 2413958 2413957 2026-05-14T16:06:48Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 2 instances 2413958 wikitext text/x-wiki c5wexubmgfa32kxrhs8fdqk0b1rwmch 2 460100 2414216 2406966 2026-05-15T10:31:01Z Effie Mouzeli (WMF) 12880 2414216 wikitext text/x-wiki 0eargasu2tr4unhmg5obrogj3ijk4fa 2414218 2414216 2026-05-15T10:32:35Z Effie Mouzeli (WMF) 12880 /* SCROLL: service name */ 2414218 wikitext text/x-wiki 9bqy5bzm3134vfb0nyfde2x5y6wlss8 2414246 2414218 2026-05-15T11:56:16Z Effie Mouzeli (WMF) 12880 2414246 wikitext text/x-wiki 1rhdjk1nn6ihco3pbu76hebhxjskqvk 2 460101 2414214 2406819 2026-05-15T10:29:59Z Effie Mouzeli (WMF) 12880 2414214 sanitized-css text/css aqnqwujb9t2q12sjwau9qyyoczykryn 0 460169 2413800 2413586 2026-05-14T13:55:07Z BTracy-WMF 46548 2413800 wikitext text/x-wiki qwuq76sblddnx8765t1byow02ndgcxa 2414239 2413800 2026-05-15T11:08:12Z GModena (WMF) 20194 2414239 wikitext text/x-wiki p6bi9xvf4qhztk9j1knpiywahql87u8 0 460171 2413954 2026-05-14T16:03:30Z RLazarus (WMF) 15215 Redirected page to [[Mw-cron jobs]] 2413954 wikitext text/x-wiki k4k4ozg52ncq6wei8exzqbubfg1dkul 3 460172 2413978 2026-05-14T16:30:39Z StrikerBot 8475 Welcome to Toolforge! 2413978 wikitext text/x-wiki 0br7yil1lxgueptad6mhymqna5ecaks 498 460173 2414021 2026-05-14T19:07:24Z ZI Jony 13427 + 2414021 wikitext text/x-wiki 76i36b06v0ell2e9mmohj4s27cfz3h4 2414037 2414021 2026-05-14T19:32:30Z JJMC89 7474 JJMC89 moved page [[WikiLounge]] to [[Nova Resource:Thelounge/Documentation]] without leaving a redirect 2414021 wikitext text/x-wiki 76i36b06v0ell2e9mmohj4s27cfz3h4 0 460174 2414022 2026-05-14T19:08:09Z ZI Jony 13427 + 2414022 wikitext text/x-wiki bjsaw8adp5j1d108425pectu53rizvl 2414144 2414022 2026-05-15T05:19:48Z ZI Jony 13427 /* */ Fix 2414144 wikitext text/x-wiki emdgokg2m39xw057r2x7jvqmtke658v 0 460175 2414145 2026-05-15T05:20:02Z ZI Jony 13427 +Redirect 2414145 wikitext text/x-wiki emdgokg2m39xw057r2x7jvqmtke658v 2 460176 2414247 2026-05-15T11:57:54Z Effie Mouzeli (WMF) 12880 Created page with "{{Draft}} <templatestyles src="User:Effie_Mouzeli_(WMF)/SCROLL/styles.css"/> = SCROLL guide = ''Scrolling to production: a starter guide for humans.'' This page explains how a service moves from "we want to ship this" to "running in production with users", using the SCROLL (''Service Checklist for Readiness, Operations, Launch and Lifecycle'') process. If you are about to start your first SCROLL, read this page end-to-end. If you have done one before, skip to #The c..." 2414247 wikitext text/x-wiki 4o0otgtono6oa54dbbkta9xlr99akkb