Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.45.0-wmf.9 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2322161 2321952 2025-07-09T14:11:25Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli 2322161 wikitext text/x-wiki fth84o8fas37wh2vulje7maxkwcqubz 2322174 2322161 2025-07-09T14:22:23Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli 2322174 wikitext text/x-wiki carcwr7rjp26ui6dzulbqo3poee3umi 2322253 2322174 2025-07-09T17:05:21Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2322253 wikitext text/x-wiki mgpl899ka6d54hcvhaesx1hfs0n252s 2322254 2322253 2025-07-09T17:10:17Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2322254 wikitext text/x-wiki qh7b9qp46k3h9vn5kd6syydjynayotd 0 3170 2322259 2321554 2025-07-09T17:38:47Z SSingh (WMF) 13928 /* DnsboxServiceMismatch */ 2322259 wikitext text/x-wiki nkgobvjchzzjd5o28pksjx4a5p8v979 0 3205 2322268 2297821 2025-07-09T18:05:45Z BCornwall-WMF 31521 /* IPIP encapsulation experiments */ grammar 2322268 wikitext text/x-wiki sug1y4zwsnik0ah7ddk7wttu0wdl95r 0 4039 2322352 2317510 2025-07-09T22:00:46Z MStyles (WMF) 15405 Updating security patch development instructions 2322352 wikitext text/x-wiki j4frt8zexkj7rl0vd6y9buxtddoet97 2322357 2322352 2025-07-09T22:39:09Z MStyles (WMF) 15405 /* Deployment: Manual */ updating patch deployment instructions 2322357 wikitext text/x-wiki rg04oep834p4kiho5l6kqdl4m4cj0jr 0 4108 2322028 2321994 2025-07-09T12:00:58Z ScheduleDeploymentBot 37566 Add [[gerrit:1167592]] to Wednesday, July 09 UTC afternoon backport window 2322028 wikitext text/x-wiki s8le2wbire75php1qh7bbouf9uydp0z 2322106 2322028 2025-07-09T13:42:04Z ScheduleDeploymentBot 37566 Add [[gerrit:1167622]] to Wednesday, July 09 UTC late backport window 2322106 wikitext text/x-wiki rpx9cz8im7u424jfufayrk68jdev5wa 2322145 2322106 2025-07-09T13:57:35Z ScheduleDeploymentBot 37566 Add [[gerrit:1153557]] to Wednesday, July 09 UTC late backport window 2322145 wikitext text/x-wiki qba1402r2fbgr9jcpyszqkyb4ukmhc0 2322279 2322145 2025-07-09T18:50:41Z ScheduleDeploymentBot 37566 Add [[gerrit:1167688]] to Wednesday, July 09 UTC late backport window 2322279 wikitext text/x-wiki 8bw5xhu3yvpswe88rjpwwa9it5e1a27 2322280 2322279 2025-07-09T18:50:43Z ScheduleDeploymentBot 37566 Add [[gerrit:1167688]] to Wednesday, July 09 UTC late backport window 2322280 wikitext text/x-wiki rqirkqrya60fp9ixxg768rxvmzzpmgg 2322281 2322280 2025-07-09T18:51:31Z Aca 11922 /* {{Deployment_day|date=2025-07-09}} */ whoops 2322281 wikitext text/x-wiki jabqlov6mm4407bivr81w7dtxhl1o8p 2322316 2322281 2025-07-09T20:14:33Z Jdforrester (WMF) 468 /* {{Deployment_day|date=2025-07-09}} */ Two done. 2322316 wikitext text/x-wiki qatairflhyi8t1jzhdcd9s0j0qx1pux 2322322 2322316 2025-07-09T20:28:59Z Jdforrester (WMF) 468 /* {{Deployment_day|date=2025-07-09}} */ One done. 2322322 wikitext text/x-wiki rp6o42shp2h3wo942v50pcv3d0sntfa 2322332 2322322 2025-07-09T20:50:33Z ScheduleDeploymentBot 37566 Add [[gerrit:558052]] to Wednesday, July 09 UTC late backport window 2322332 wikitext text/x-wiki okad56kmfrbmy81ybun0dj9pv93lmq4 2322334 2322332 2025-07-09T20:56:39Z Winston Sung 21096 /* {{Deployment_day|date=2025-07-09}} */ 2322334 wikitext text/x-wiki rp6o42shp2h3wo942v50pcv3d0sntfa 2322335 2322334 2025-07-09T21:03:33Z Winston Sung 21096 2322335 wikitext text/x-wiki okad56kmfrbmy81ybun0dj9pv93lmq4 2322340 2322335 2025-07-09T21:15:30Z Winston Sung 21096 /* {{Deployment_day|date=2025-07-09}} */ 2322340 wikitext text/x-wiki e43er8qm14bknqk8dwc9yt3bpqppkya 0 4116 2322341 2229282 2025-07-09T21:15:34Z Mutante 20 /* Deleting pads via site admin */ add a section about pad names in non-latin charsets 2322341 wikitext text/x-wiki 8bk7qd72ym1rs0qa0keoelp7vzlvvdc 498 6726 2322153 2321946 2025-07-09T14:00:17Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli 2322153 wikitext text/x-wiki 406g9ts40kmgucu2082bjs7mii7acg1 2322158 2322153 2025-07-09T14:10:08Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli 2322158 wikitext text/x-wiki dzwrsg7obcady664ufr62yjolmql4gt 2322249 2322158 2025-07-09T16:52:20Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2322249 wikitext text/x-wiki 6t0esjgx6eog41xu7qbe2dlj1gad78k 2322250 2322249 2025-07-09T16:52:23Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api 2322250 wikitext text/x-wiki gl7bwhvgiyy28qh3gcojbqa1xtjnvg5 2322251 2322250 2025-07-09T16:58:44Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2322251 wikitext text/x-wiki 2k9do5rde4qtyeth8kb1aemn5ugaus4 2322252 2322251 2025-07-09T17:03:28Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2322252 wikitext text/x-wiki ky0wcalbzq5a3puye4kiekdyd499loj 2322476 2322252 2025-07-10T09:26:58Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2322476 wikitext text/x-wiki px8s4w9gez0id2to331ynjixmgudahb 2322477 2322476 2025-07-10T09:27:05Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli 2322477 wikitext text/x-wiki 3r0ehoybr94hm4ul0tocgftppt72ocu 2322478 2322477 2025-07-10T09:30:51Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2322478 wikitext text/x-wiki on7pib5az91ckbsrt51s7jbmrisdiea 2322479 2322478 2025-07-10T09:31:15Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli 2322479 wikitext text/x-wiki dg79fgesq64yut6gl3fgk3rjov126y8 2322498 2322479 2025-07-10T10:45:03Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli 2322498 wikitext text/x-wiki ng8bilc6esbi0jq02nuxh1y9865zs2m 2322499 2322498 2025-07-10T10:45:27Z Stashbot 7414 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli 2322499 wikitext text/x-wiki klz4djgihvaazb27w1jzvxj3dwyc13u 0 7919 2322029 2322027 2025-07-09T12:01:54Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/changeprop: apply 2322029 wikitext text/x-wiki a02jwtm5tz5bpeqqk4spem3u12ox7ry 2322030 2322029 2025-07-09T12:02:18Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-jumbo-eqiad 2322030 wikitext text/x-wiki e4ajevl38ukw06wzojz9uj1z36ulpqw 2322031 2322030 2025-07-09T12:02:27Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/changeprop: apply 2322031 wikitext text/x-wiki qd1fac2s1pdcuk4gg1ef5rsqltgmwnp 2322032 2322031 2025-07-09T12:02:54Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply 2322032 wikitext text/x-wiki 5zalfl462qharzkq3j7kbspngx5o3kj 2322033 2322032 2025-07-09T12:04:00Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply 2322033 wikitext text/x-wiki o6m47gqmi1vs9qbgxuzpsvuxg868cjo 2322034 2322033 2025-07-09T12:07:24Z Stashbot 7414 brouberol@cumin1003: END (FAIL) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=1) rolling restart_daemons on A:kafka-jumbo-eqiad 2322034 wikitext text/x-wiki jjv2a6w6gv7sw6l5kchbf1e5g3xiu34 2322035 2322034 2025-07-09T12:07:25Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-jumbo-eqiad 2322035 wikitext text/x-wiki 72r2xskwf02bpfpec6ru8qvidhztboi 2322036 2322035 2025-07-09T12:08:13Z Stashbot 7414 moritzm: installing nginx security updates 2322036 wikitext text/x-wiki f2bhqu8dz48c8rtfwp6v3xzoz82d87r 2322037 2322036 2025-07-09T12:08:21Z Stashbot 7414 brouberol@cumin1003: END (ERROR) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=97) rolling restart_daemons on A:kafka-jumbo-eqiad 2322037 wikitext text/x-wiki r8vdt2p4l2w03ungem4rz34ekge7e2r 2322038 2322037 2025-07-09T12:09:20Z Stashbot 7414 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade Replica to GitLab 18.0 2322038 wikitext text/x-wiki 7z9l5crwcvdxvhte4khu2plo3mubpv1 2322039 2322038 2025-07-09T12:11:42Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1050.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322039 wikitext text/x-wiki ti6rd888ryhcdgv2ntxr59evvu6xt2h 2322040 2322039 2025-07-09T12:11:46Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322040 wikitext text/x-wiki pmhhjdo18x15nx7yirvf0w6e5951jq1 2322041 2322040 2025-07-09T12:14:21Z Stashbot 7414 moritzm: installing openjdk-17 security updates 2322041 wikitext text/x-wiki cxf2s06grv7oftprfv75xilczuvho4o 2322042 2322041 2025-07-09T12:14:41Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1048.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322042 wikitext text/x-wiki k87kxuhfyw3p3l3goy3d9t6oitclw8b 2322043 2322042 2025-07-09T12:15:09Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1049.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322043 wikitext text/x-wiki f9scwccsjqjxw9d7o6a2a1m5hpfka85 2322045 2322043 2025-07-09T12:16:55Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1048.eqiad.wmnet with OS bullseye 2322045 wikitext text/x-wiki swc7bl5ommnn527nl5f4b02giwpzmuq 2322046 2322045 2025-07-09T12:17:09Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1049.eqiad.wmnet with OS bullseye 2322046 wikitext text/x-wiki c4i6q1142y7zew0eeqdmudty21zqt27 2322047 2322046 2025-07-09T12:25:54Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp[4052].ulsfo.wmnet} and A:cp - 2.8.15 upgrade (T398720) 2322047 wikitext text/x-wiki scdrlhxaqe64wumfk28u6z1ejjnkh1q 2322048 2322047 2025-07-09T12:30:21Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp[4052].ulsfo.wmnet} and A:cp - 2.8.15 upgrade (T398720) 2322048 wikitext text/x-wiki hsx5hvh2wy71vkwk94uzrxp0dbmg2gw 2322049 2322048 2025-07-09T12:34:19Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1048.eqiad.wmnet with reason: host reimage 2322049 wikitext text/x-wiki hojs0yfyg1w6pf78k1r1yy14xkcxtke 2322050 2322049 2025-07-09T12:34:29Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1049.eqiad.wmnet with reason: host reimage 2322050 wikitext text/x-wiki nysweqq555izhwofjrq7dikyxriz4hv 2322051 2322050 2025-07-09T12:36:24Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1050.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322051 wikitext text/x-wiki 8kb6xozen9jih9ycipv49ejorhtg369 2322052 2322051 2025-07-09T12:36:27Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322052 wikitext text/x-wiki nra3qlq0p722vfo5anm5ezicth87bar 2322053 2322052 2025-07-09T12:37:29Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 2.8.15 upgrade (T398720) 2322053 wikitext text/x-wiki ib3ykf4xd0mlp0e8viiyeb4zvy26a9v 2322054 2322053 2025-07-09T12:38:13Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1048.eqiad.wmnet with reason: host reimage 2322054 wikitext text/x-wiki j1kqfgl66whrzgt8vv9gzf96cgsdgfx 2322055 2322054 2025-07-09T12:39:17Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 2.8.15 upgrade (T398720) 2322055 wikitext text/x-wiki thd6g1hmv286piygorzd62j014v4ddw 2322056 2322055 2025-07-09T12:39:23Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1050 2322056 wikitext text/x-wiki aacgbor9ootm8vwy409l0idpe5scrzf 2322057 2322056 2025-07-09T12:39:30Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1050 2322057 wikitext text/x-wiki 4y1zf7y86z80fy8te87zuf40cadbqfz 2322058 2322057 2025-07-09T12:39:33Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1051 2322058 wikitext text/x-wiki bvpllyvsnu240ndbpafq2o0qa631l9h 2322059 2322058 2025-07-09T12:39:40Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1051 2322059 wikitext text/x-wiki 7d779el6vcil2fy1t2es7cehfw1kp49 2322060 2322059 2025-07-09T12:41:40Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1049.eqiad.wmnet with reason: host reimage 2322060 wikitext text/x-wiki 8wh3eobaqxtfhza07rdnyymhz8cbi8l 2322061 2322060 2025-07-09T12:44:20Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1051 2322061 wikitext text/x-wiki 1ridni2c78wm6g3h2hcuspto11z6wjn 2322062 2322061 2025-07-09T12:44:27Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1051 2322062 wikitext text/x-wiki dflqncf1jsx07yw4klzw98uroj6z7yz 2322063 2322062 2025-07-09T12:48:37Z Stashbot 7414 hashar@deploy1003: Started deploy [gerrit/gerrit@9666238]: Add readonly plugin - T387833 2322063 wikitext text/x-wiki pu9yn1pk8en602mhwapkujvzn2cyxmy 2322064 2322063 2025-07-09T12:48:46Z Stashbot 7414 hashar@deploy1003: Finished deploy [gerrit/gerrit@9666238]: Add readonly plugin - T387833 (duration: 00m 11s) 2322064 wikitext text/x-wiki jmrt5tr5mgce8jt69hla34u71yvfyzr 2322065 2322064 2025-07-09T12:50:37Z Stashbot 7414 hashar@deploy1003: Started deploy [gerrit/gerrit@9666238]: Add readonly plugin - T387833 2322065 wikitext text/x-wiki 56sqpkux2jaaod1lbighf3zhm4hik8z 2322066 2322065 2025-07-09T12:50:48Z Stashbot 7414 hashar@deploy1003: Finished deploy [gerrit/gerrit@9666238]: Add readonly plugin - T387833 (duration: 00m 10s) 2322066 wikitext text/x-wiki t2pwfkstunjcvnkvetjjntr8t7cv3e6 2322067 2322066 2025-07-09T12:54:12Z Stashbot 7414 moritzm: installing jetty9 security updates 2322067 wikitext text/x-wiki 3714xba2lmud5ffyfwn6j0wweya5min 2322068 2322067 2025-07-09T12:54:40Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host flink-zk1001.eqiad.wmnet 2322068 wikitext text/x-wiki 2w2u1chzhhpyua84gewlw7nxf9ul3qm 2322069 2322068 2025-07-09T12:54:43Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" 2322069 wikitext text/x-wiki dq1jr9nej77junjyw4mrtkj5plkcecr 2322070 2322069 2025-07-09T12:55:14Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" 2322070 wikitext text/x-wiki qzbhqso8wmbvjgkv27jjvpxyw51clxv 2322071 2322070 2025-07-09T12:55:15Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1048.eqiad.wmnet with OS bullseye 2322071 wikitext text/x-wiki krir7dr2wqdie7byon4j39pdta5vl31 2322072 2322071 2025-07-09T12:58:00Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" 2322072 wikitext text/x-wiki 12i8pu3290ycquexs0ttm2ed4xu6dyy 2322073 2322072 2025-07-09T12:58:18Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" 2322073 wikitext text/x-wiki gh8v13bq0dtkmqwzgut04eufjhdc4mb 2322074 2322073 2025-07-09T12:58:20Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1049.eqiad.wmnet with OS bullseye 2322074 wikitext text/x-wiki htinnlu85iy7kj38ytytmub1tsro7pw 2322075 2322074 2025-07-09T12:58:36Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1001.eqiad.wmnet 2322075 wikitext text/x-wiki tch91q63uqlbk49nmqjk0wypass0dme 2322076 2322075 2025-07-09T12:59:36Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1051.eqiad.wmnet with OS bullseye 2322076 wikitext text/x-wiki 4tvnk99ryhe1p27406726phv38tuhyi 2322077 2322076 2025-07-09T13:01:19Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1050.eqiad.wmnet with OS bullseye 2322077 wikitext text/x-wiki ed86kprkjebhp3x1ykwq9qj58mdjoj7 2322078 2322077 2025-07-09T13:09:37Z Stashbot 7414 brouberol@cumin1003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-test-eqiad 2322078 wikitext text/x-wiki n9nv1i4y2jw7b562297b6wt5t3u1v9n 2322079 2322078 2025-07-09T13:10:31Z Stashbot 7414 sgimeno@deploy1003: Started scap sync-world: Backport for [[gerrit:1167592|Add new script to update old freetext country data new schema (T397270)]], [[gerrit:1167587|Growth: Enable limiting Add Link for dewiki (T396382)]] 2322079 wikitext text/x-wiki rdxtypqbhzj60k75l77fxmra591rykg 2322080 2322079 2025-07-09T13:11:48Z Stashbot 7414 brouberol@cumin1003: END (ERROR) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=97) rolling restart_daemons on A:kafka-test-eqiad 2322080 wikitext text/x-wiki b7gidvzh9mm1a290yioi053bm2lrlc7 2322081 2322080 2025-07-09T13:12:35Z Stashbot 7414 sgimeno@deploy1003: mhorsey, sgimeno, migr: Backport for [[gerrit:1167592|Add new script to update old freetext country data new schema (T397270)]], [[gerrit:1167587|Growth: Enable limiting Add Link for dewiki (T396382)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2322081 wikitext text/x-wiki b7hqfsj34y1c2rl1293l1w0x6y7yqzd 2322083 2322081 2025-07-09T13:15:13Z Stashbot 7414 sgimeno@deploy1003: mhorsey, sgimeno, migr: Continuing with sync 2322083 wikitext text/x-wiki p5u7irxjl0s0sa8of3q6dwn5c447kd6 2322084 2322083 2025-07-09T13:18:14Z Stashbot 7414 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . 2322084 wikitext text/x-wiki omt8mmxxnuy0jr84znqp75guf442wfw 2322085 2322084 2025-07-09T13:20:38Z Stashbot 7414 sgimeno@deploy1003: Finished scap sync-world: Backport for [[gerrit:1167592|Add new script to update old freetext country data new schema (T397270)]], [[gerrit:1167587|Growth: Enable limiting Add Link for dewiki (T396382)]] (duration: 10m 07s) 2322085 wikitext text/x-wiki fkjaefi6r3tjcbv6x7429ry9ax2xjfl 2322086 2322085 2025-07-09T13:21:12Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1041', diff saved to https://phabricator.wikimedia.org/P78828 and previous config saved to /var/cache/conftool/dbconfig/20250709-132111-marostegui.json 2322086 wikitext text/x-wiki tq2t3yuqhvmg47qyvwasj8mog5z6tvw 2322087 2322086 2025-07-09T13:21:58Z Stashbot 7414 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on es1041.eqiad.wmnet with reason: Maintenance 2322087 wikitext text/x-wiki 1yqf0ft10ysjrb7t2ae6nhoqbmowla8 2322088 2322087 2025-07-09T13:24:59Z Stashbot 7414 marostegui@cumin1002: START - Cookbook sre.hosts.reboot-single for host es1041.eqiad.wmnet 2322088 wikitext text/x-wiki 07r9r5reqcs99go563cbrhu631tr0ud 2322089 2322088 2025-07-09T13:25:35Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 2.8.15 upgrade (T398720) 2322089 wikitext text/x-wiki 06undfpiz6tqf53kybahyb89hazxu2l 2322090 2322089 2025-07-09T13:26:37Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 2.8.15 upgrade (T398720) 2322090 wikitext text/x-wiki fmco6ec36sqlob1knwege0f2uh20lsp 2322091 2322090 2025-07-09T13:30:27Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host flink-zk1002.eqiad.wmnet 2322091 wikitext text/x-wiki rt647b8hxtfuhe2c5nj2qsjkd9xtqck 2322092 2322091 2025-07-09T13:31:11Z Stashbot 7414 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host es1041.eqiad.wmnet 2322092 wikitext text/x-wiki bgx5d2gyuhl9brqvn6caj0eleddgz8k 2322093 2322092 2025-07-09T13:31:38Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on es1041.eqiad.wmnet with reason: Maintenance 2322093 wikitext text/x-wiki 1fq7o8prwpoysgelqa1eokia24cmj8g 2322094 2322093 2025-07-09T13:31:45Z Stashbot 7414 marostegui@cumin1002: START - Cookbook sre.hosts.reboot-single for host es1041.eqiad.wmnet 2322094 wikitext text/x-wiki htulituir5suruopw29tb9w81gx0u8j 2322095 2322094 2025-07-09T13:34:23Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1002.eqiad.wmnet 2322095 wikitext text/x-wiki j62plzuqc2lzs9m1grghtda8jc61zl5 2322098 2322095 2025-07-09T13:36:32Z Stashbot 7414 Daimona: mwscript-k8s --comment="T397270" -f --file /srv/mediawiki/php-1.45.0-wmf.9/extensions/CampaignEvents/maintenance/countryExceptionMappings.csv -- CampaignEvents:UpdateCountriesColumn --wiki metawiki --exceptions countryExceptionMappings.csv 2322098 wikitext text/x-wiki p7u3e5zzv01ebjjkym5y5aidho2kyy3 2322099 2322098 2025-07-09T13:36:36Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on es1044.eqiad.wmnet with reason: Maintenance 2322099 wikitext text/x-wiki 5ioo1dsvekixa2n3lu26uap9eppsjsu 2322100 2322099 2025-07-09T13:36:40Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1044 for upgrade', diff saved to https://phabricator.wikimedia.org/P78829 and previous config saved to /var/cache/conftool/dbconfig/20250709-133639-marostegui.json 2322100 wikitext text/x-wiki 2xwy5onst2t9lg9ozpwyf435522ofbj 2322101 2322100 2025-07-09T13:38:22Z Stashbot 7414 marostegui@cumin1002: START - Cookbook sre.hosts.reboot-single for host es1044.eqiad.wmnet 2322101 wikitext text/x-wiki ss0oot7jhvbp76qy9orkis7uzsnjmbu 2322102 2322101 2025-07-09T13:38:48Z Stashbot 7414 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host flink-zk1003.eqiad.wmnet 2322102 wikitext text/x-wiki 1rwhvn044smxpiq1hdxuon39i6hi8xe 2322103 2322102 2025-07-09T13:39:03Z Stashbot 7414 Daimona: mwscript-k8s --comment="T397270" -f --file /srv/mediawiki/php-1.45.0-wmf.9/extensions/CampaignEvents/maintenance/countryExceptionMappings.csv -- CampaignEvents:UpdateCountriesColumn --wiki officewiki --exceptions countryExceptionMappings.csv 2322103 wikitext text/x-wiki nyypd408ac2o8h8q4ixal0rtyt1f9ti 2322104 2322103 2025-07-09T13:40:09Z Stashbot 7414 Daimona: mwscript-k8s --comment="T397270" -f --file /srv/mediawiki/php-1.45.0-wmf.9/extensions/CampaignEvents/maintenance/countryExceptionMappings.csv -- CampaignEvents:UpdateCountriesColumn --wiki testwiki --exceptions countryExceptionMappings.csv 2322104 wikitext text/x-wiki nf311u6i454hn9cdu4bx1vw0c33ghu3 2322105 2322104 2025-07-09T13:41:33Z Stashbot 7414 Daimona: mwscript-k8s --comment="T397270" -f --file /srv/mediawiki/php-1.45.0-wmf.9/extensions/CampaignEvents/maintenance/countryExceptionMappings.csv -- CampaignEvents:UpdateCountriesColumn --wiki test2wiki --exceptions countryExceptionMappings.csv 2322105 wikitext text/x-wiki lp1gxuav66zo5t1cvql0b6zaamkjxre 2322107 2322105 2025-07-09T13:42:42Z Stashbot 7414 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1003.eqiad.wmnet 2322107 wikitext text/x-wiki 7q8ogt0mdec3cvtzxu08kgj2adxryqu 2322108 2322107 2025-07-09T13:43:05Z Stashbot 7414 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host es1041.eqiad.wmnet 2322108 wikitext text/x-wiki q1mbalb85fk5iov1robft2t5j07frms 2322109 2322108 2025-07-09T13:44:54Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply 2322109 wikitext text/x-wiki 3kgwwy5qummopw1vs2xyn9hdl2c8389 2322110 2322109 2025-07-09T13:44:56Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply 2322110 wikitext text/x-wiki 20j2hi0yolxt30q57nkb7r3tk7ek7ow 2322111 2322110 2025-07-09T13:45:20Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply 2322111 wikitext text/x-wiki 3gwecn07to4q8avjce49bwalu6yyumi 2322112 2322111 2025-07-09T13:45:22Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply 2322112 wikitext text/x-wiki bwfjtkvmbz1ux74mqmuz7uatsn40mn6 2322113 2322112 2025-07-09T13:45:54Z Stashbot 7414 claime: Depooling chartmuseum in codfw 2322113 wikitext text/x-wiki k11zuziiyyfgv73j3is6nrmm0wncdek 2322114 2322113 2025-07-09T13:46:04Z Stashbot 7414 cgoubert@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw 2322114 wikitext text/x-wiki 9kp7jyr8bl7blyu83rt9dz6y8cjn67w 2322115 2322114 2025-07-09T13:46:29Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet 2322115 wikitext text/x-wiki b11qggl7xjpgbqllqtju3tjgtls3jeq 2322116 2322115 2025-07-09T13:46:35Z Stashbot 7414 vgutierrez: deploy measure/measure-goog certs in the upload CDN cluster - T394484 2322116 wikitext text/x-wiki 7pv17x2lrg5gyhypfvi2x009rjhhubr 2322118 2322116 2025-07-09T13:49:55Z Stashbot 7414 marostegui@cumin1002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host es1044.eqiad.wmnet 2322118 wikitext text/x-wiki ewyefxe7dcuu3ot22fuvh6buswdktsc 2322119 2322118 2025-07-09T13:50:18Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet 2322119 wikitext text/x-wiki ffekyskaab428h487z46t2t2vhg0j7h 2322120 2322119 2025-07-09T13:50:27Z Stashbot 7414 cgoubert@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw 2322120 wikitext text/x-wiki g7k15pi1nfyw6rzlfgequ6zeve4h5la 2322121 2322120 2025-07-09T13:50:48Z Stashbot 7414 claime: Depooling chartmuseum in eqiad 2322121 wikitext text/x-wiki 3mjoiz9ekp8zh8qlq381drrske3tpyq 2322122 2322121 2025-07-09T13:50:49Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7002.wikimedia.org,service=authdns-update [reason: testing alert] 2322122 wikitext text/x-wiki 34hol6oqnou0slbhpbezk56ove8ydtj 2322123 2322122 2025-07-09T13:50:55Z Stashbot 7414 cgoubert@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad 2322123 wikitext text/x-wiki s0gcbrrbgjbkddb0ji6w23gmv52n8jd 2322124 2322123 2025-07-09T13:51:08Z Stashbot 7414 cgoubert@cumin1003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet 2322124 wikitext text/x-wiki am4eup4eh5o2byo1tpcj4n3e2h53cid 2322127 2322124 2025-07-09T13:52:45Z Stashbot 7414 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1167574|ApiQueryCategoryMembers: Try stop forcing index in read new code (T399037)]], [[gerrit:1167573|ApiQueryCategoryMembers: Try stop forcing index in read new code (T399037)]], [[gerrit:1167570|Fix categorylinks read new code for excluding categories (T398861 T398939)]], [[gerrit:1167569|Fix categorylinks read new code for excluding categories (T39886 2322127 wikitext text/x-wiki ejftujwolqzyyzzbm1qpxkvv29nri1h 2322128 2322127 2025-07-09T13:53:31Z Stashbot 7414 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet 2322128 wikitext text/x-wiki hk05qognqy0d0q57fl3gq89laxzckk4 2322129 2322128 2025-07-09T13:53:43Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephosd2004-dev.codfw.wmnet with OS bookworm 2322129 wikitext text/x-wiki n4kgig9v7qoz6asotp1cge6201qda70 2322130 2322129 2025-07-09T13:53:44Z Stashbot 7414 cgoubert@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=eqiad 2322130 wikitext text/x-wiki iog8ive8astbmdxlwy45k947aq6cjs7 2322131 2322130 2025-07-09T13:54:06Z Stashbot 7414 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1050.eqiad.wmnet with OS bullseye 2322131 wikitext text/x-wiki qhgtx7u4z6t7maby958coe0zkpcvdyy 2322132 2322131 2025-07-09T13:54:14Z Stashbot 7414 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1051.eqiad.wmnet with OS bullseye 2322132 wikitext text/x-wiki 172gf7ouewl95qc2ja1144w5l7q05va 2322133 2322132 2025-07-09T13:54:16Z Stashbot 7414 hnowlan: delete three wedged thumbor pods showing signs of T374350 2322133 wikitext text/x-wiki mbyrq7v4plpo0d6973ub22fmk372df5 2322134 2322133 2025-07-09T13:54:52Z Stashbot 7414 zabe@deploy1003: zabe: Backport for [[gerrit:1167574|ApiQueryCategoryMembers: Try stop forcing index in read new code (T399037)]], [[gerrit:1167573|ApiQueryCategoryMembers: Try stop forcing index in read new code (T399037)]], [[gerrit:1167570|Fix categorylinks read new code for excluding categories (T398861 T398939)]], [[gerrit:1167569|Fix categorylinks read new code for excluding categories (T398861 T398939)]] synced 2322134 wikitext text/x-wiki g65yw3unhrtupu3dhfqfnixfle1wg0r 2322135 2322134 2025-07-09T13:54:54Z Stashbot 7414 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7002.wikimedia.org,service=authdns-update [reason: testing alert] 2322135 wikitext text/x-wiki 4ny8ih9onm65tgpoev8i3u9x1ts23mh 2322136 2322135 2025-07-09T13:55:05Z Stashbot 7414 sukhe@dns1004: START - running authdns-update 2322136 wikitext text/x-wiki mhoxbtrkj3hkz0qlohr4isylb8bsmc8 2322137 2322136 2025-07-09T13:55:49Z Stashbot 7414 sukhe@dns1004: END - running authdns-update 2322137 wikitext text/x-wiki nmuq0oj05sp7iij8fn03clv4jaz2f77 2322138 2322137 2025-07-09T13:55:51Z Stashbot 7414 zabe@deploy1003: zabe: Continuing with sync 2322138 wikitext text/x-wiki ifzk5ea8f30kd2r8zr0b6qzk71c79to 2322139 2322138 2025-07-09T13:56:38Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1050.eqiad.wmnet with OS bullseye 2322139 wikitext text/x-wiki ojwil4b0hxqzrnmooxdztbe95zyiaw3 2322140 2322139 2025-07-09T13:56:57Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply 2322140 wikitext text/x-wiki r23yxr6pnkc0913ax87b1w1nghytifl 2322141 2322140 2025-07-09T13:57:07Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply 2322141 wikitext text/x-wiki h6e99mjcbnuphg5mwmf5dzr5uwwxx10 2322142 2322141 2025-07-09T13:57:14Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: apply 2322142 wikitext text/x-wiki jf6rwq1wpzrpb2luu91ohpsqcrfuf15 2322143 2322142 2025-07-09T13:57:28Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply 2322143 wikitext text/x-wiki lxpb5hx7p9cb1xabznrfwdrt2mm4v1v 2322144 2322143 2025-07-09T13:57:33Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'es1044 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P78835 and previous config saved to /var/cache/conftool/dbconfig/20250709-135732-root.json 2322144 wikitext text/x-wiki oeu3z1k4mzq5pqjge7sq2cqgpgif0ww 2322146 2322144 2025-07-09T13:58:07Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: apply 2322146 wikitext text/x-wiki 1l30e9u0b1o8q4jtktjiftvd87g4f43 2322147 2322146 2025-07-09T13:58:51Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply 2322147 wikitext text/x-wiki b1iv3fctox4y00vwhs7vr17c2ietnec 2322148 2322147 2025-07-09T13:58:58Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply 2322148 wikitext text/x-wiki l5k3a02kglnpm0wx3ygty8qy0ht889g 2322150 2322148 2025-07-09T13:59:24Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P78836 and previous config saved to /var/cache/conftool/dbconfig/20250709-135923-root.json 2322150 wikitext text/x-wiki 7gok1eg8prt67tw3jf39rsdovet35e5 2322151 2322150 2025-07-09T13:59:39Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply 2322151 wikitext text/x-wiki 4afu5a5eo8q3rm8ofoye681p3xryuqg 2322154 2322151 2025-07-09T14:01:26Z Stashbot 7414 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1167574|ApiQueryCategoryMembers: Try stop forcing index in read new code (T399037)]], [[gerrit:1167573|ApiQueryCategoryMembers: Try stop forcing index in read new code (T399037)]], [[gerrit:1167570|Fix categorylinks read new code for excluding categories (T398861 T398939)]], [[gerrit:1167569|Fix categorylinks read new code for excluding categories (T3988 2322154 wikitext text/x-wiki odhaul51uykemka1wf91hlym1hv8nzp 2322155 2322154 2025-07-09T14:07:27Z Stashbot 7414 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2322155 wikitext text/x-wiki 8ci2umrkr9to3s404tkcoabph0w25m0 2322156 2322155 2025-07-09T14:08:28Z Stashbot 7414 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2322156 wikitext text/x-wiki hvdk27zc8r10k7by9z5sjo05w59gurz 2322157 2322156 2025-07-09T14:09:33Z Stashbot 7414 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2322157 wikitext text/x-wiki 6raqr8mqxbtx1f19j9nca1x3z3acwrm 2322159 2322157 2025-07-09T14:10:30Z Stashbot 7414 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2322159 wikitext text/x-wiki hooa00cz025ar6vh8kzwb7ztljn8oim 2322160 2322159 2025-07-09T14:10:50Z Stashbot 7414 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2322160 wikitext text/x-wiki 3xs6rqkwlpcnqzsx43j1drzcz4rkazj 2322162 2322160 2025-07-09T14:11:38Z Stashbot 7414 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2322162 wikitext text/x-wiki os3ozvv72qc60w0gqgcwiu00rtxrcqi 2322163 2322162 2025-07-09T14:12:40Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'es1044 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P78837 and previous config saved to /var/cache/conftool/dbconfig/20250709-141238-root.json 2322163 wikitext text/x-wiki tfql9j93azku10bn8wqzc3zw3cg1xpm 2322164 2322163 2025-07-09T14:14:30Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P78838 and previous config saved to /var/cache/conftool/dbconfig/20250709-141428-root.json 2322164 wikitext text/x-wiki 2s3w1h3h5n3yceqr2wj6yot1be484d1 2322165 2322164 2025-07-09T14:14:51Z Stashbot 7414 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply 2322165 wikitext text/x-wiki 11pzvxcpzh9czvthhtisf2oiw3pai9z 2322166 2322165 2025-07-09T14:14:59Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd2004-dev.codfw.wmnet with reason: host reimage 2322166 wikitext text/x-wiki o219jiiy8e2a7lldwgible8hr8c7dpl 2322167 2322166 2025-07-09T14:15:32Z Stashbot 7414 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply 2322167 wikitext text/x-wiki 40bsd3u9oabp79uiumuubju1an2edf2 2322168 2322167 2025-07-09T14:16:27Z Stashbot 7414 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: cirrussearch-codfw-psi@codfw 2322168 wikitext text/x-wiki 5oo45e7vznoummr097px1rpr1f5e7np 2322169 2322168 2025-07-09T14:16:40Z Stashbot 7414 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply 2322169 wikitext text/x-wiki sc55057gukx07hmxbkjze5iusldz7o8 2322170 2322169 2025-07-09T14:17:10Z Stashbot 7414 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply 2322170 wikitext text/x-wiki gzp8kc3v2jws78cr6bvtt4qk1qgxjco 2322171 2322170 2025-07-09T14:17:20Z Stashbot 7414 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply 2322171 wikitext text/x-wiki 2kbxclkfvnvxwmky4587gfbqa22456u 2322172 2322171 2025-07-09T14:17:51Z Stashbot 7414 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply 2322172 wikitext text/x-wiki k7xngt8wkbcjfs89s4g2oycex1nxobq 2322173 2322172 2025-07-09T14:17:58Z Stashbot 7414 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd2004-dev.codfw.wmnet with reason: host reimage 2322173 wikitext text/x-wiki aizf5g5mwdnric8f38lh3h69c7wofo8 2322175 2322173 2025-07-09T14:23:52Z Stashbot 7414 moritzm: installing bash updates from Bookworm point release 2322175 wikitext text/x-wiki omj6619kprrasuq323kl5q2p03r35b7 2322176 2322175 2025-07-09T14:23:59Z Stashbot 7414 vgutierrez@cumin1002: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: cirrussearch-codfw-psi@codfw 2322176 wikitext text/x-wiki k4y69hbpcerw09hy21fo6irx5ptwqz0 2322177 2322176 2025-07-09T14:24:39Z Stashbot 7414 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: cirrussearch-codfw-omega@codfw 2322177 wikitext text/x-wiki 6fb2h4zuehkenkvf0g2rdkvbw3n86uu 2322178 2322177 2025-07-09T14:27:45Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'es1044 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P78839 and previous config saved to /var/cache/conftool/dbconfig/20250709-142744-root.json 2322178 wikitext text/x-wiki hfdyr7tvgb72mxposs9ng15ugvqtifa 2322179 2322178 2025-07-09T14:29:38Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P78840 and previous config saved to /var/cache/conftool/dbconfig/20250709-142934-root.json 2322179 wikitext text/x-wiki 6lpikf18rh29vgj1yjg0beflwjn3e56 2322180 2322179 2025-07-09T14:30:07Z Stashbot 7414 vgutierrez@cumin1002: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: cirrussearch-codfw-omega@codfw 2322180 wikitext text/x-wiki k8ei9hvmti4x5bzp5dvfab6m38fpb69 2322181 2322180 2025-07-09T14:30:38Z Stashbot 7414 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: cirrussearch@codfw 2322181 wikitext text/x-wiki 57jzwd8icxzw3iamx4s0c01slqen22z 2322182 2322181 2025-07-09T14:34:05Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1051.eqiad.wmnet with OS bullseye 2322182 wikitext text/x-wiki i17w84pgfsz82zan9kqoczeuv8yhntq 2322183 2322182 2025-07-09T14:34:49Z Stashbot 7414 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet 2322183 wikitext text/x-wiki f4vdos0pp1gx23u7rs0qcwhxlxun4ks 2322184 2322183 2025-07-09T14:35:51Z Stashbot 7414 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd2004-dev.codfw.wmnet with OS bookworm 2322184 wikitext text/x-wiki r6ec5jhi352utvp66vk7e1um5mfv1a0 2322189 2322184 2025-07-09T14:40:15Z Stashbot 7414 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2322189 wikitext text/x-wiki 9widnhesrkspelgtpgec5edyckfqcjb 2322190 2322189 2025-07-09T14:41:26Z Stashbot 7414 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs 2322190 wikitext text/x-wiki ktphlztdbkqp3sf8hfuzuv801x7r8jo 2322191 2322190 2025-07-09T14:41:28Z Stashbot 7414 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: cirrussearch@codfw 2322191 wikitext text/x-wiki ox63u9sr8x8gl8oeowr8j3cf1ripni9 2322192 2322191 2025-07-09T14:42:51Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'es1044 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P78841 and previous config saved to /var/cache/conftool/dbconfig/20250709-144250-root.json 2322192 wikitext text/x-wiki pxuwuzmshbt9vvbmtgom8q5av75ig23 2322193 2322192 2025-07-09T14:43:09Z Stashbot 7414 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1050.eqiad.wmnet with OS bullseye 2322193 wikitext text/x-wiki blwu3ka475u69q8fx5qf6s44i4r3esp 2322195 2322193 2025-07-09T14:44:41Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P78842 and previous config saved to /var/cache/conftool/dbconfig/20250709-144440-root.json 2322195 wikitext text/x-wiki 14eye1py3id3kvwc1ifa6agqed23wj4 2322196 2322195 2025-07-09T14:45:58Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2322196 wikitext text/x-wiki ffzo9io5joh8gy2aaplgo3lqwheguvy 2322197 2322196 2025-07-09T14:49:14Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003" 2322197 wikitext text/x-wiki 7b9gx5bfpodz7h2cgmcb6jjpopnbd1y 2322198 2322197 2025-07-09T14:49:18Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entry for rgw.codfw.dpe.anycast.wmnet - cmooney@cumin1003" 2322198 wikitext text/x-wiki 9ra0hes2mtqo7d42p20xhga8ex0wpiu 2322199 2322198 2025-07-09T14:49:19Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2322199 wikitext text/x-wiki j7grasuiewg187muavg9zezsnwsn43w 2322201 2322199 2025-07-09T14:57:02Z Stashbot 7414 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: cirrussearch-eqiad-psi@eqiad 2322201 wikitext text/x-wiki 3bs32pj1b1lslqi9tviipx7n4otm7yn 2322202 2322201 2025-07-09T14:57:10Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1050.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322202 wikitext text/x-wiki 704f4z4svvkxefkf003fae0v88p7u05 2322203 2322202 2025-07-09T14:57:32Z Stashbot 7414 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1167637|Revert^2 "Enable categorylinks read new on a few large wikis"]] 2322203 wikitext text/x-wiki s3cjs4oenorpclsiujf1hj33gd6jo2f 2322204 2322203 2025-07-09T14:59:36Z Stashbot 7414 zabe@deploy1003: zabe: Backport for [[gerrit:1167637|Revert^2 "Enable categorylinks read new on a few large wikis"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2322204 wikitext text/x-wiki sq6d5t0x85t5zbmdo8q1wunfh219oc7 2322205 2322204 2025-07-09T15:00:21Z Stashbot 7414 zabe@deploy1003: zabe: Continuing with sync 2322205 wikitext text/x-wiki 6dhcbt1p4e5sc0yn3a6msbpo497srf7 2322206 2322205 2025-07-09T15:00:22Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322206 wikitext text/x-wiki ex41t7lnopib8ec9mdlqovj2q39czl1 2322207 2322206 2025-07-09T15:03:21Z Stashbot 7414 vgutierrez@cumin1002: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: cirrussearch-eqiad-psi@eqiad 2322207 wikitext text/x-wiki ab00md11lprlqitiwzvlbihqgo4z4iu 2322209 2322207 2025-07-09T15:04:25Z Stashbot 7414 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: cirrussearch-eqiad-omega@eqiad 2322209 wikitext text/x-wiki h0pucpvktvyzl81w30n85faqw8srcih 2322210 2322209 2025-07-09T15:04:55Z Stashbot 7414 moritzm: installing abseil security updates 2322210 wikitext text/x-wiki pm2nrhorbiboa7yntyvr2v13cftzrxp 2322211 2322210 2025-07-09T15:05:43Z Stashbot 7414 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1167637|Revert^2 "Enable categorylinks read new on a few large wikis"]] (duration: 08m 11s) 2322211 wikitext text/x-wiki q02hm16j68i9u985yne3a6l25xg9mw9 2322212 2322211 2025-07-09T15:09:27Z Stashbot 7414 vgutierrez@cumin1002: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: cirrussearch-eqiad-omega@eqiad 2322212 wikitext text/x-wiki st2ict00mytnudqc6n1qzjhuoxpa0ex 2322213 2322212 2025-07-09T15:10:07Z Stashbot 7414 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: cirrussearch@eqiad 2322213 wikitext text/x-wiki otvhqwkw76yunpn55maqs5mlagq58mj 2322215 2322213 2025-07-09T15:12:38Z Stashbot 7414 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1051.eqiad.wmnet with OS bullseye 2322215 wikitext text/x-wiki 14o0tzerk6akj09m4csywgc5kb6or48 2322216 2322215 2025-07-09T15:19:55Z Stashbot 7414 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1050.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322216 wikitext text/x-wiki r9elqfx9b0nnqmhnern99ad0xz0xorr 2322217 2322216 2025-07-09T15:20:56Z Stashbot 7414 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs 2322217 wikitext text/x-wiki 38mu838cxhb4c510xjhnuv52w3ul6pe 2322218 2322217 2025-07-09T15:22:16Z Stashbot 7414 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322218 wikitext text/x-wiki iwuvukpcss937rjx03u6duak1o0jnpn 2322219 2322218 2025-07-09T15:22:44Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1050.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322219 wikitext text/x-wiki k7kzlyu6h8iwgjwua82s5gu9ilov73w 2322220 2322219 2025-07-09T15:23:04Z Stashbot 7414 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs 2322220 wikitext text/x-wiki bnfpw7526idddu13iajmb4as2dsgxmo 2322221 2322220 2025-07-09T15:23:06Z Stashbot 7414 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: cirrussearch@eqiad 2322221 wikitext text/x-wiki o2kp66z86v8u1aurviyysdtrssfunkx 2322222 2322221 2025-07-09T15:25:10Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322222 wikitext text/x-wiki 590dshyits5mrj4oi4wovlx9wdur0hh 2322224 2322222 2025-07-09T15:32:51Z Stashbot 7414 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322224 wikitext text/x-wiki ck4e94hsyjs46ekqv7pb10i04u8iioi 2322225 2322224 2025-07-09T15:33:02Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephmon1004.eqiad.wmnet with OS bookworm 2322225 wikitext text/x-wiki slqtj2tuyvc4rkjb30ujs0jspdqwv6a 2322226 2322225 2025-07-09T15:33:34Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322226 wikitext text/x-wiki 5sfz5mn17dhetu0ky4317sa1wrgydjw 2322227 2322226 2025-07-09T15:37:09Z Stashbot 7414 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322227 wikitext text/x-wiki gmllh9j2sbdarq2cvl8tekytf256pjc 2322228 2322227 2025-07-09T15:40:51Z Stashbot 7414 volans: uploaded spicerack_11.3.0 to apt.wikimedia.org bullseye-wikimedia,bookworm-wikimedia 2322228 wikitext text/x-wiki swpm378eypqhrr29or4le3oen2i03ew 2322229 2322228 2025-07-09T15:42:23Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322229 wikitext text/x-wiki j8u4kox4hgqdj0f15be09e9177qvjye 2322230 2322229 2025-07-09T15:43:18Z Stashbot 7414 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322230 wikitext text/x-wiki fagcf8vb5kxa6m98hvfzoob0gtx5u3t 2322231 2322230 2025-07-09T15:43:40Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322231 wikitext text/x-wiki ewzn5txn5btsjjfi6vo5qqsi6047sa4 2322232 2322231 2025-07-09T15:44:19Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1050.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322232 wikitext text/x-wiki 2s2rnnbrre9hshnrikcaqtnsnctbl8j 2322233 2322232 2025-07-09T15:44:31Z Stashbot 7414 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322233 wikitext text/x-wiki 6t95zho3gg3lo16wix5am6lrk32qugn 2322234 2322233 2025-07-09T15:54:08Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon1004.eqiad.wmnet with reason: host reimage 2322234 wikitext text/x-wiki 9348qgh4vhk24nrq7kf3hhoj1a6jmy1 2322235 2322234 2025-07-09T15:57:44Z Stashbot 7414 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon1004.eqiad.wmnet with reason: host reimage 2322235 wikitext text/x-wiki 9njtjdywopmoblzzm4rhf0t9kmh26un 2322236 2322235 2025-07-09T16:02:19Z Stashbot 7414 vgutierrez: switching esams, eqsin and drmrs to Let's Encrypt unified/upload certs - T398596 2322236 wikitext text/x-wiki 267eq64un89hyv3b1qxs169psqr4k8v 2322238 2322236 2025-07-09T16:06:25Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply 2322238 wikitext text/x-wiki pe1tj6nxp0sfba7ds9w6xbs9uztt031 2322239 2322238 2025-07-09T16:06:36Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply 2322239 wikitext text/x-wiki amsxhyvplvqj9ayurye6tncg6qjqinm 2322240 2322239 2025-07-09T16:07:03Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply 2322240 wikitext text/x-wiki e4lwtuscfayyod7nj4qi295j4if2asj 2322241 2322240 2025-07-09T16:07:14Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply 2322241 wikitext text/x-wiki mj1t12reqcbq2yz74h2lcvd7w1oq4fp 2322242 2322241 2025-07-09T16:07:57Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply 2322242 wikitext text/x-wiki coa14glc9qb1utlmgjxuv2ausu50fj9 2322243 2322242 2025-07-09T16:08:08Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply 2322243 wikitext text/x-wiki c5guq19ictuwbo7c71mhw3n4tsje0cg 2322244 2322243 2025-07-09T16:08:50Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply 2322244 wikitext text/x-wiki m1lkpclc4k2fvx5amlpregef8r8fsdj 2322245 2322244 2025-07-09T16:15:45Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host cp2044.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322245 wikitext text/x-wiki ipsmpi92ij0015l94ysqajx0550wgn2 2322246 2322245 2025-07-09T16:17:10Z Stashbot 7414 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon1004.eqiad.wmnet with OS bookworm 2322246 wikitext text/x-wiki 5v1das312gtt4e8tlwbk2s9jjmvwh5v 2322247 2322246 2025-07-09T16:17:25Z Stashbot 7414 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp2044.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322247 wikitext text/x-wiki 8r6czuf4pcgzmfeojh1nc5uyyp2r8jw 2322248 2322247 2025-07-09T16:50:10Z Stashbot 7414 bking@cumin1002: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for wdqs2023.codfw.wmnet: Renew puppet certificate - bking@cumin1002 2322248 wikitext text/x-wiki 8x24o2ebyoa13g37f205147cawy3rdb 2322258 2322248 2025-07-09T17:34:21Z Stashbot 7414 sukhe: re-enabling Puppet on P{ganeti7002* or ganeti7003*}: it was left disabled there during rollout of CR 1166222 by sukhe 2322258 wikitext text/x-wiki fuhku6z9mk29yoq5zrk0r36plqt846f 2322272 2322258 2025-07-09T18:23:06Z Stashbot 7414 brett@cumin2002: START - Cookbook sre.hosts.reimage for host lvs1017.eqiad.wmnet with OS bullseye 2322272 wikitext text/x-wiki 4hois5i6yn5pfb9nqvskougp9m3flda 2322273 2322272 2025-07-09T18:23:25Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Upgrade 2322273 wikitext text/x-wiki j1rzisuv82fxqx070l7t8xo4kagzcqv 2322274 2322273 2025-07-09T18:33:22Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Security Upgrade 2322274 wikitext text/x-wiki azrryq2e1vy4f71sblevsg8fwknk1my 2322275 2322274 2025-07-09T18:35:56Z Stashbot 7414 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Upgrade 2322275 wikitext text/x-wiki rh03lplygw7ct05e7yvhh1tucrknntb 2322276 2322275 2025-07-09T18:40:56Z Stashbot 7414 sukhe: removing ocsp from deployment-prep: commit 3307286c7d18827b87231b61652efbaf0e3ba4c8: T399114 2322276 wikitext text/x-wiki 4idw17t2mf11x36707zulyf5npcb5uy 2322277 2322276 2025-07-09T18:42:32Z Stashbot 7414 sukhe: re-adding ocsp from deployment-prep: commit 3307286c7d18827b87231b61652efbaf0e3ba4c8: T399114: will remove after Puppet removal 2322277 wikitext text/x-wiki 4h1c9us8qg5q2sh258q48wqkzwpzy5c 2322278 2322277 2025-07-09T18:45:58Z Stashbot 7414 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Upgrade 2322278 wikitext text/x-wiki ggjphnurh8xpreby2wbzf2vqk8byfcw 2322283 2322278 2025-07-09T19:10:41Z Stashbot 7414 vriley@cumin1002: START - Cookbook sre.hosts.provision for host es1047.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322283 wikitext text/x-wiki nbohbeym9vjg34m58rmb4vbg5jiphq2 2322284 2322283 2025-07-09T19:12:03Z Stashbot 7414 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host es1047.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322284 wikitext text/x-wiki m9dmjo0jhi9ivdw93b7pxkmbbq2c8gz 2322285 2322284 2025-07-09T19:16:02Z Stashbot 7414 vriley@cumin1002: START - Cookbook sre.hosts.provision for host es1047.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322285 wikitext text/x-wiki at8ug5mm772ivrx02033jekipi63xi9 2322286 2322285 2025-07-09T19:16:04Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply 2322286 wikitext text/x-wiki lfxag95mkld560psq7bf2fpq25brj87 2322287 2322286 2025-07-09T19:16:05Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply 2322287 wikitext text/x-wiki 35icnr1n4cdrgu33lsqp1ti7ipr8c57 2322288 2322287 2025-07-09T19:16:28Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply 2322288 wikitext text/x-wiki n9dm2vtwnhqgckl6d18a5ae9f9edbnp 2322289 2322288 2025-07-09T19:16:33Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply 2322289 wikitext text/x-wiki 93rvs3svdyxh301qxc6xqu5e544f4by 2322290 2322289 2025-07-09T19:16:47Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply 2322290 wikitext text/x-wiki sy3i6m9x0vkdky7cd5dexaw3l8h2t2s 2322291 2322290 2025-07-09T19:16:50Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply 2322291 wikitext text/x-wiki mcclh9z03vkaqfq6c3hkewwyiwkj23p 2322292 2322291 2025-07-09T19:21:15Z Stashbot 7414 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host es1047.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322292 wikitext text/x-wiki crgmytbh4gok3wp9zx8lexfuihjze3h 2322293 2322292 2025-07-09T19:24:24Z Stashbot 7414 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host es1047.eqiad.wmnet with OS bookworm 2322293 wikitext text/x-wiki bb16l8msjwnno5jg7ud4val4qii31vv 2322294 2322293 2025-07-09T19:29:09Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1050.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322294 wikitext text/x-wiki tf7a0d5pdwezy2ij88w1g4tuk2pn6nj 2322295 2322294 2025-07-09T19:29:11Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322295 wikitext text/x-wiki neux53hyo4dey94vwhdhpj16bj01gjr 2322296 2322295 2025-07-09T19:42:57Z Stashbot 7414 vriley@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on es1047.eqiad.wmnet with reason: host reimage 2322296 wikitext text/x-wiki nixspptke5yjg8hoim1jkakcxktzghj 2322297 2322296 2025-07-09T19:43:26Z Stashbot 7414 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs1017.eqiad.wmnet with OS bullseye 2322297 wikitext text/x-wiki o2cz39iw07westcfvmh15cto0xmuvdi 2322298 2322297 2025-07-09T19:47:18Z Stashbot 7414 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1047.eqiad.wmnet with reason: host reimage 2322298 wikitext text/x-wiki 2q9y6atpvkt2f1dr5xv5cwkj9ttbi15 2322299 2322298 2025-07-09T19:49:07Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1050.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322299 wikitext text/x-wiki glwgxy6ql7qriy370jxzc722ic64y5i 2322300 2322299 2025-07-09T19:49:13Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED 2322300 wikitext text/x-wiki r99sn7p42p28fxtihsadaw2g53z7c84 2322301 2322300 2025-07-09T19:51:10Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1050.eqiad.wmnet with OS bullseye 2322301 wikitext text/x-wiki 2c8fnr8eclutdvqxf967syi4onc3oox 2322302 2322301 2025-07-09T19:51:20Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1051.eqiad.wmnet with OS bullseye 2322302 wikitext text/x-wiki f0csa1m76628w1z8njmx680l789l353 2322303 2322302 2025-07-09T19:51:40Z Stashbot 7414 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1051.eqiad.wmnet with OS bullseye 2322303 wikitext text/x-wiki cxzhd4bxe2f1hw3148fqe4tskh3xj86 2322304 2322303 2025-07-09T19:52:57Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1051.eqiad.wmnet with OS bullseye 2322304 wikitext text/x-wiki nts2bj8df84juucwrs8y1tmn61g2sui 2322305 2322304 2025-07-09T20:04:48Z Stashbot 7414 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1167688|shwiki: Add bs, hr and sr as import sources (T399113)]], [[gerrit:1153557|Remove white outline from Wikifunctions favicon (T326094)]] 2322305 wikitext text/x-wiki sir0x7fv6elo1lxb6tpx1bsuklu9f0e 2322306 2322305 2025-07-09T20:06:56Z Stashbot 7414 jforrester@deploy1003: jforrester, jhsoby, aleksandar: Backport for [[gerrit:1167688|shwiki: Add bs, hr and sr as import sources (T399113)]], [[gerrit:1153557|Remove white outline from Wikifunctions favicon (T326094)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2322306 wikitext text/x-wiki b74loxyqhg346ory0c9nfonq0022mwq 2322307 2322306 2025-07-09T20:07:26Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1050.eqiad.wmnet with reason: host reimage 2322307 wikitext text/x-wiki lcnvw984vtn8qlvjbghfm6yham4t4su 2322308 2322307 2025-07-09T20:08:17Z Stashbot 7414 jforrester@deploy1003: jforrester, jhsoby, aleksandar: Continuing with sync 2322308 wikitext text/x-wiki g0uw16r4zunrnvaqybhgawthp8e0t8p 2322309 2322308 2025-07-09T20:10:06Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1051.eqiad.wmnet with reason: host reimage 2322309 wikitext text/x-wiki a6um3uco7loi5svrfinfz3en4d3dn4e 2322310 2322309 2025-07-09T20:10:50Z Stashbot 7414 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1002" 2322310 wikitext text/x-wiki gctbktfkppfmd9kpx6duvbrl3wg1sx3 2322311 2322310 2025-07-09T20:10:54Z Stashbot 7414 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1002" 2322311 wikitext text/x-wiki 7jvj6irk8g50v8ukgitqaxrvp4b6boo 2322312 2322311 2025-07-09T20:10:55Z Stashbot 7414 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1047.eqiad.wmnet with OS bookworm 2322312 wikitext text/x-wiki hoztkelc686mbhhpqqdos8246d826us 2322313 2322312 2025-07-09T20:13:40Z Stashbot 7414 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1167688|shwiki: Add bs, hr and sr as import sources (T399113)]], [[gerrit:1153557|Remove white outline from Wikifunctions favicon (T326094)]] (duration: 08m 52s) 2322313 wikitext text/x-wiki 0iwb7u4bl38grssooyp61fzfwncymvw 2322314 2322313 2025-07-09T20:13:51Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1050.eqiad.wmnet with reason: host reimage 2322314 wikitext text/x-wiki 4o5cn7rw3gh5okb8qaxbirdsk6fns36 2322315 2322314 2025-07-09T20:13:59Z Stashbot 7414 James_F: jforrester@deploy1003:~$ echo 'https://en.wikipedia.org/static/favicon/wikifunctions.ico' | mwscript-k8s --attach purgeList.php -- --wiki enwiki # T326094 2322315 wikitext text/x-wiki kzxv3ou6iy91mf0erm4f801cm6jruz4 2322317 2322315 2025-07-09T20:17:14Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1051.eqiad.wmnet with reason: host reimage 2322317 wikitext text/x-wiki l2ua0gup3lc8x1f3kygcoewx4p7aevj 2322318 2322317 2025-07-09T20:17:40Z Stashbot 7414 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1167622|Pre-deploy Readers Use Cases Survey on enwiki (T398870)]] 2322318 wikitext text/x-wiki ts0kwt06sgbpt90dgk57ibunodl0hk9 2322319 2322318 2025-07-09T20:19:50Z Stashbot 7414 jforrester@deploy1003: jforrester, dani: Backport for [[gerrit:1167622|Pre-deploy Readers Use Cases Survey on enwiki (T398870)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2322319 wikitext text/x-wiki 4sk5vtdn7dk6wj4srqszsfm3cu0vyk8 2322320 2322319 2025-07-09T20:23:25Z Stashbot 7414 jforrester@deploy1003: jforrester, dani: Continuing with sync 2322320 wikitext text/x-wiki 1s3e8783r1oqsz0wca9hzew9lfkilcr 2322321 2322320 2025-07-09T20:28:41Z Stashbot 7414 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1167622|Pre-deploy Readers Use Cases Survey on enwiki (T398870)]] (duration: 11m 00s) 2322321 wikitext text/x-wiki m1m4hs433svyb2kcabjo7udx14negpf 2322323 2322321 2025-07-09T20:30:42Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" 2322323 wikitext text/x-wiki p0zpugy3adid3ptd1npwvgkixp2x3ej 2322324 2322323 2025-07-09T20:31:03Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" 2322324 wikitext text/x-wiki 0np3ai32vbb2puadjt72hwp8bvuk57q 2322325 2322324 2025-07-09T20:31:04Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1050.eqiad.wmnet with OS bullseye 2322325 wikitext text/x-wiki e3sijqh3fisw7ninolnvwzy5rysr83e 2322326 2322325 2025-07-09T20:33:30Z Stashbot 7414 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" 2322326 wikitext text/x-wiki qjopbrro8giycxibk4bcfz4zvbucoij 2322327 2322326 2025-07-09T20:34:11Z Stashbot 7414 vriley@cumin1002: START - Cookbook sre.hosts.provision for host es1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2322327 wikitext text/x-wiki ig1f13acoctq95m7i7oofs9tvitw5ob 2322328 2322327 2025-07-09T20:35:25Z Stashbot 7414 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host es1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2322328 wikitext text/x-wiki pnl6cq5gf1v1souhpmvxfp46k37yk6r 2322329 2322328 2025-07-09T20:36:01Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" 2322329 wikitext text/x-wiki q9lkobb36i09y94at4pvl4cfp0bszid 2322330 2322329 2025-07-09T20:36:02Z Stashbot 7414 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1051.eqiad.wmnet with OS bullseye 2322330 wikitext text/x-wiki e10wtb58qleb6lx71zxxla9ntthjnl5 2322331 2322330 2025-07-09T20:50:03Z Stashbot 7414 vriley@cumin1002: START - Cookbook sre.hosts.provision for host es1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2322331 wikitext text/x-wiki k35sbpzj0pzsnv0p2sm5k0xfs1i61fh 2322333 2322331 2025-07-09T20:55:55Z Stashbot 7414 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host es1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2322333 wikitext text/x-wiki 969khqy9cw2jth5hefbardqevczofvf 2322336 2322333 2025-07-09T21:05:15Z Stashbot 7414 dancy@deploy1003: Started scap sync-world: Backport for [[gerrit:558052|Add DEPRECATED_LANGUAGE_CODE_MAPPING to wgInterlanguageLinkCodeMap (T248352)]] 2322336 wikitext text/x-wiki crus9r1clr36donngqfjtnzv24e1p15 2322337 2322336 2025-07-09T21:05:49Z Stashbot 7414 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host es1048.eqiad.wmnet with OS bookworm 2322337 wikitext text/x-wiki aovreyom6driod8ufoxd4iz4vdj0vn2 2322338 2322337 2025-07-09T21:07:21Z Stashbot 7414 dancy@deploy1003: dancy, fomafix: Backport for [[gerrit:558052|Add DEPRECATED_LANGUAGE_CODE_MAPPING to wgInterlanguageLinkCodeMap (T248352)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2322338 wikitext text/x-wiki rujvlr2ejgjl630hspv1vadgc7kobz2 2322339 2322338 2025-07-09T21:10:16Z Stashbot 7414 dancy@deploy1003: dancy, fomafix: Continuing with sync 2322339 wikitext text/x-wiki fzpkxmxwxvlw3vnyom63gahjw506k5r 2322342 2322339 2025-07-09T21:15:41Z Stashbot 7414 dancy@deploy1003: Finished scap sync-world: Backport for [[gerrit:558052|Add DEPRECATED_LANGUAGE_CODE_MAPPING to wgInterlanguageLinkCodeMap (T248352)]] (duration: 10m 25s) 2322342 wikitext text/x-wiki 8zgyjgofihw84u13ccakiubsuuwxq9r 2322343 2322342 2025-07-09T21:24:31Z Stashbot 7414 vriley@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on es1048.eqiad.wmnet with reason: host reimage 2322343 wikitext text/x-wiki rz2sxz3rubj4fu7qb5k7zf4qefhs72k 2322344 2322343 2025-07-09T21:28:15Z Stashbot 7414 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1048.eqiad.wmnet with reason: host reimage 2322344 wikitext text/x-wiki l2viti1x4ybrogtb2oot7exvr09trzm 2322348 2322344 2025-07-09T21:45:59Z Stashbot 7414 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1002" 2322348 wikitext text/x-wiki jrc2138k3uxgs689zlx0tpxepldfuwc 2322349 2322348 2025-07-09T21:46:03Z Stashbot 7414 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1002" 2322349 wikitext text/x-wiki 9xezdiuand74pkv1z2vasrqgxyqdjbl 2322350 2322349 2025-07-09T21:46:05Z Stashbot 7414 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1048.eqiad.wmnet with OS bookworm 2322350 wikitext text/x-wiki d6lvpf3lq3jklv6qzp80ygjlffsfdpi 2322353 2322350 2025-07-09T22:18:52Z Stashbot 7414 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1166499|ukwiki: allow bureaucrats to assign and remove temporary-account-viewer group (T398738)]] 2322353 wikitext text/x-wiki 3byamhek4rj674s1pe8n039qoiupivf 2322354 2322353 2025-07-09T22:21:02Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz, dreamrimmer: Backport for [[gerrit:1166499|ukwiki: allow bureaucrats to assign and remove temporary-account-viewer group (T398738)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2322354 wikitext text/x-wiki huu3dyjxlml8m3hjcrfpl621p2ix4q1 2322355 2322354 2025-07-09T22:23:43Z Stashbot 7414 dreamyjazz@deploy1003: dreamyjazz, dreamrimmer: Continuing with sync 2322355 wikitext text/x-wiki 5vdktujp0rf5dqbvqqhokfgl9ipi0k9 2322356 2322355 2025-07-09T22:29:11Z Stashbot 7414 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1166499|ukwiki: allow bureaucrats to assign and remove temporary-account-viewer group (T398738)]] (duration: 10m 18s) 2322356 wikitext text/x-wiki nk4ys1ixbbqo361f1awrvfo524zw3d9 2322359 2322356 2025-07-09T23:21:52Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephosd1006.eqiad.wmnet with OS bookworm 2322359 wikitext text/x-wiki a261qa0v145x21422grwa91hwyixio5 2322361 2322359 2025-07-10T00:42:41Z Stashbot 7414 andrew@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1006.eqiad.wmnet with OS bookworm 2322361 wikitext text/x-wiki 2j5jjbotpt5ecfk5fq20uc1p119iacz 2322362 2322361 2025-07-10T01:39:27Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephosd1006.eqiad.wmnet with OS bookworm 2322362 wikitext text/x-wiki 9nanhv4thyl0vud62x6jns0f5y5s5b2 2322363 2322362 2025-07-10T01:53:01Z Stashbot 7414 andrew@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1006.eqiad.wmnet with OS bookworm 2322363 wikitext text/x-wiki 0el6aa9xwy06wsb3c5e9v3rrmjaudzw 2322364 2322363 2025-07-10T01:53:31Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hosts.dhcp for host cloudcephosd1006.eqiad.wmnet 2322364 wikitext text/x-wiki cny6qbmvfkhgxxf09v4fi2yx2loyhjm 2322365 2322364 2025-07-10T01:55:04Z Stashbot 7414 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudcephosd1006.eqiad.wmnet 2322365 wikitext text/x-wiki c9yyosc6sedayigo0bhkedfodbi52g3 2322366 2322365 2025-07-10T02:03:51Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1006.eqiad.wmnet'] 2322366 wikitext text/x-wiki c8kgq8fwzebttk7lx9qniy89p88arsw 2322367 2322366 2025-07-10T02:03:57Z Stashbot 7414 root@cumin1003: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['cloudcephosd1006.eqiad.wmnet'] 2322367 wikitext text/x-wiki 1izurathp8k1twc8jqwwn9138zb1t3r 2322368 2322367 2025-07-10T02:03:59Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1006.eqiad.wmnet'] 2322368 wikitext text/x-wiki he92z47snqmcuvpdtw0qqmqec4yhrkx 2322369 2322368 2025-07-10T02:04:04Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1006.eqiad.wmnet'] 2322369 wikitext text/x-wiki 3k84bukl8o3am977l3e65m50e2n1psx 2322370 2322369 2025-07-10T02:04:28Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1006.eqiad.wmnet'] 2322370 wikitext text/x-wiki 23tl2d17bq1hycaxlq3kcdvwsxwxumj 2322371 2322370 2025-07-10T02:11:18Z Stashbot 7414 root@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudcephosd1006.eqiad.wmnet'] 2322371 wikitext text/x-wiki 3p8k0qiaummo8eh69bd1bjnurl271tp 2322372 2322371 2025-07-10T02:28:52Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1006.eqiad.wmnet'] 2322372 wikitext text/x-wiki cx00erfrhbsqkdwfh68kd8i78g598ou 2322373 2322372 2025-07-10T02:29:08Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1006.eqiad.wmnet'] 2322373 wikitext text/x-wiki hn7bcbbazpvu6hluetnc3gdycxrabrn 2322374 2322373 2025-07-10T02:29:44Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1006.eqiad.wmnet'] 2322374 wikitext text/x-wiki 8qrm1aq7pnromuqz8iaah4j1eigycsf 2322375 2322374 2025-07-10T02:37:28Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1006.eqiad.wmnet'] 2322375 wikitext text/x-wiki t8bjuuqpemyqotbojpwitvj7znnz3hx 2322376 2322375 2025-07-10T02:39:24Z Stashbot 7414 root@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1006.eqiad.wmnet'] 2322376 wikitext text/x-wiki nkl63unakwzwinuafcloewivgcnhrfx 2322377 2322376 2025-07-10T02:46:48Z Stashbot 7414 root@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1006.eqiad.wmnet'] 2322377 wikitext text/x-wiki lgek8l2frp4fpfd9vctc5jduzqu3saz 2322378 2322377 2025-07-10T02:53:23Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephosd1006.eqiad.wmnet with OS bookworm 2322378 wikitext text/x-wiki iiiwoqtzxoe5psdjdcjotpeiy2ynb6h 2322379 2322378 2025-07-10T03:01:36Z Stashbot 7414 andrew@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1006.eqiad.wmnet with OS bookworm 2322379 wikitext text/x-wiki owyx0pdy1jqcw0krzxo30tjrahj3wuh 2322380 2322379 2025-07-10T03:01:51Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephosd1006.eqiad.wmnet with OS bookworm 2322380 wikitext text/x-wiki 0qzmbfyis2r5dgha11iu345mq9axx4i 2322381 2322380 2025-07-10T03:36:50Z Stashbot 7414 andrew@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1006.eqiad.wmnet with OS bookworm 2322381 wikitext text/x-wiki 3b9lpd2o6tnqnnt1qpf3qc5y320a8r4 2322382 2322381 2025-07-10T03:37:13Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephosd1006.eqiad.wmnet with OS bookworm 2322382 wikitext text/x-wiki 6qw2q4idbhcxlzgewd793g17p5z5xlm 2322383 2322382 2025-07-10T03:55:26Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1006.eqiad.wmnet with reason: host reimage 2322383 wikitext text/x-wiki 8bj6gsmot0n78judl5eci3lbxyhvhun 2322384 2322383 2025-07-10T03:58:41Z Stashbot 7414 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1006.eqiad.wmnet with reason: host reimage 2322384 wikitext text/x-wiki qdolioxsltjoc23oveun37a8wj67b6z 2322385 2322384 2025-07-10T04:10:04Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams-internal: apply 2322385 wikitext text/x-wiki nodln1w7kehpv2lf82uni56g4co4xci 2322386 2322385 2025-07-10T04:12:52Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams-internal: apply 2322386 wikitext text/x-wiki 4w1i0hbjddwlghprw4sul5ksf5x976m 2322387 2322386 2025-07-10T04:13:10Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams-internal: apply 2322387 wikitext text/x-wiki 71v8eon76odzlb1vgr11dy8xl4fct8w 2322388 2322387 2025-07-10T04:14:17Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams-internal: apply 2322388 wikitext text/x-wiki qi07ca5igswgy5rx6rawvtn6w3rs4ia 2322389 2322388 2025-07-10T04:16:19Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams-internal: apply 2322389 wikitext text/x-wiki radeut70rnzf6kw0nqgy6crb4nbr7mq 2322390 2322389 2025-07-10T04:16:29Z Stashbot 7414 andrew@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1006.eqiad.wmnet with OS bookworm 2322390 wikitext text/x-wiki lj299x58cnxxgo5d3ynj9hfem4ro079 2322391 2322390 2025-07-10T04:17:21Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams-internal: apply 2322391 wikitext text/x-wiki opw3iyqbfuf8eiyb5hyub1c2yy3ema3 2322392 2322391 2025-07-10T04:18:01Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply 2322392 wikitext text/x-wiki 0s2oy5swlln4ifs0hvrbhzpyshfv3d3 2322393 2322392 2025-07-10T04:29:12Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply 2322393 wikitext text/x-wiki 0nmybxjcud8uyn3jvoljhz4yvh1t7q0 2322394 2322393 2025-07-10T04:32:46Z Stashbot 7414 tchin@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply 2322394 wikitext text/x-wiki ci23if41m5e2y93g6rrvd0e1p3lro98 2322395 2322394 2025-07-10T04:32:56Z Stashbot 7414 tchin@deploy1003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply 2322395 wikitext text/x-wiki gdfa46kdpnat8530dfu32id75jnfii9 2322398 2322395 2025-07-10T04:55:56Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply 2322398 wikitext text/x-wiki as4l2miacbbbjbduga6jw29g5a9an5u 2322399 2322398 2025-07-10T04:56:43Z Stashbot 7414 tchin@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply 2322399 wikitext text/x-wiki nilu2pkko2zxud2vh01nny7k29wsirz 2322400 2322399 2025-07-10T04:57:50Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply 2322400 wikitext text/x-wiki 9o1r3odr9z6rics5ha7vaxhm2s6bkll 2322401 2322400 2025-07-10T04:58:36Z Stashbot 7414 tchin@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply 2322401 wikitext text/x-wiki 6folbj3odmu7izqerzmm2uxcj7ffgui 2322402 2322401 2025-07-10T05:21:43Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Bugfixes - oblivian@cumin1003" 2322402 wikitext text/x-wiki j1cb9b58er1o2y82rw4tfk8hmk4hi5z 2322403 2322402 2025-07-10T05:21:45Z Stashbot 7414 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes - oblivian@cumin1003 2322403 wikitext text/x-wiki kfb6uxm6cehbgtrqdiw7l9biyqqjm3i 2322404 2322403 2025-07-10T05:22:18Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes - oblivian@cumin1003 2322404 wikitext text/x-wiki 9jj5wskwmj6s4o5zlfo4rdybqo0c9mp 2322405 2322404 2025-07-10T05:22:20Z Stashbot 7414 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Bugfixes - oblivian@cumin1003" 2322405 wikitext text/x-wiki pup28vtxy0ejtjc20sda6nosxhb4v0b 2322406 2322405 2025-07-10T05:38:23Z Stashbot 7414 kartik@deploy1003: helmfile [staging] START helmfile.d/services/machinetranslation: apply 2322406 wikitext text/x-wiki 06qhb4e1l58p0kw1muov8m80uvpranz 2322407 2322406 2025-07-10T05:54:20Z Stashbot 7414 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply 2322407 wikitext text/x-wiki p9eeb85662juu9rvmolbsp2hltov2jk 2322408 2322407 2025-07-10T06:25:57Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1036.eqiad.wmnet with reason: Maintenance 2322408 wikitext text/x-wiki l0z68kty2v7a3kil47x00hb9e8pqr3h 2322409 2322408 2025-07-10T06:35:32Z Stashbot 7414 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2228.codfw.wmnet with reason: Maintenance 2322409 wikitext text/x-wiki livuut387t64ddgyj3b5d4l07m6aezy 2322410 2322409 2025-07-10T06:35:37Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2228 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P78846 and previous config saved to /var/cache/conftool/dbconfig/20250710-063535-marostegui.json 2322410 wikitext text/x-wiki 5iitf3tau1jo6vuq54z9uy2iuojlyfr 2322411 2322410 2025-07-10T06:39:06Z Stashbot 7414 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2228.codfw.wmnet with reason: Maintenance 2322411 wikitext text/x-wiki 5nw2dosibbe8lx3xwfae3vcafa7hn9y 2322412 2322411 2025-07-10T06:44:39Z Stashbot 7414 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply 2322412 wikitext text/x-wiki h50ylb5nru39i729v1tsen4b61x92ek 2322413 2322412 2025-07-10T06:44:47Z Stashbot 7414 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply 2322413 wikitext text/x-wiki dpa1izrd6yf5mz5e7c0b3f5jx04720t 2322414 2322413 2025-07-10T06:45:55Z Stashbot 7414 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1210.eqiad.wmnet with reason: Maintenance 2322414 wikitext text/x-wiki 0s9jcac5dvg8oxvy767hudvtcyd1xwc 2322415 2322414 2025-07-10T06:45:59Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1210 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P78847 and previous config saved to /var/cache/conftool/dbconfig/20250710-064558-marostegui.json 2322415 wikitext text/x-wiki ah7qfafrp4kv1ej63bcjbrvv7bgcblc 2322416 2322415 2025-07-10T06:46:06Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P78848 and previous config saved to /var/cache/conftool/dbconfig/20250710-064605-root.json 2322416 wikitext text/x-wiki 32cv11blrf67xkr1k9dyyfydqn1lykb 2322417 2322416 2025-07-10T06:47:09Z Stashbot 7414 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade Replica to GitLab 18.0 2322417 wikitext text/x-wiki hw6fmznaz08nwgzl9jqqjy00xnsvruy 2322418 2322417 2025-07-10T06:49:11Z Stashbot 7414 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply 2322418 wikitext text/x-wiki qa996dyg8g2h047zb8xfzx37modkvkd 2322419 2322418 2025-07-10T06:52:10Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and A:cp - 2.8.15 upgrade (T398720) 2322419 wikitext text/x-wiki rjvnek59zyt6a5j81kfuws3duz92myn 2322420 2322419 2025-07-10T06:52:39Z Stashbot 7414 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply 2322420 wikitext text/x-wiki b2skzw4hc79oyjdvge8ewu60gqf6ln2 2322421 2322420 2025-07-10T06:53:51Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db1210 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P78849 and previous config saved to /var/cache/conftool/dbconfig/20250710-065350-root.json 2322421 wikitext text/x-wiki aktmu87fdtix9ug26gcx4g2a8vp4i3j 2322422 2322421 2025-07-10T06:55:38Z Stashbot 7414 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply 2322422 wikitext text/x-wiki 0puh77jaxg2m64zfafs3ob1igb36ptj 2322423 2322422 2025-07-10T06:58:30Z Stashbot 7414 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply 2322423 wikitext text/x-wiki p0t13je8zfeym2xh4jzsbczd2vbkqdu 2322424 2322423 2025-07-10T06:58:52Z Stashbot 7414 marostegui@cumin1002: START - Cookbook sre.mysql.parsercache 2322424 wikitext text/x-wiki dc4tqknv2msoexfp7ymte9ra65fzygg 2322425 2322424 2025-07-10T06:59:03Z Stashbot 7414 marostegui@cumin1002: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) 2322425 wikitext text/x-wiki b0wywulx903ej3yh7xa7opmu183mu4o 2322426 2322425 2025-07-10T06:59:28Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - 2.8.15 upgrade (T398720) 2322426 wikitext text/x-wiki fngk82mttdii8ow8ikyu8nqzusswpry 2322427 2322426 2025-07-10T07:00:40Z Stashbot 7414 moritzm: installing libbpf security updates 2322427 wikitext text/x-wiki 6ioj6c5mhopqg9tx0zg3znxw2vecoii 2322428 2322427 2025-07-10T07:01:12Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P78851 and previous config saved to /var/cache/conftool/dbconfig/20250710-070111-root.json 2322428 wikitext text/x-wiki bk0q3e3p1izb50llresszqwfk6gj6li 2322429 2322428 2025-07-10T07:08:57Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db1210 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P78852 and previous config saved to /var/cache/conftool/dbconfig/20250710-070855-root.json 2322429 wikitext text/x-wiki nxrttuqhj3awefl95peosws9oeo0g2c 2322430 2322429 2025-07-10T07:10:28Z Stashbot 7414 kartik@deploy1003: helmfile [staging] START helmfile.d/services/machinetranslation: apply 2322430 wikitext text/x-wiki rmlxkip4e65lc2akwfw4e6u8f1ujd6j 2322431 2322430 2025-07-10T07:16:18Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P78853 and previous config saved to /var/cache/conftool/dbconfig/20250710-071616-root.json 2322431 wikitext text/x-wiki th97cjse6oktb72ele9eiuwku3toytg 2322432 2322431 2025-07-10T07:17:55Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1213.eqiad.wmnet with reason: Maintenance 2322432 wikitext text/x-wiki f4oo0pfgaq1z8tdaq7ujqpcqy1acin2 2322433 2322432 2025-07-10T07:18:19Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1217.eqiad.wmnet with reason: Maintenance 2322433 wikitext text/x-wiki eu72wg3olk1d5438p3i83n8q7e9t2oa 2322434 2322433 2025-07-10T07:24:02Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db1210 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P78855 and previous config saved to /var/cache/conftool/dbconfig/20250710-072401-root.json 2322434 wikitext text/x-wiki 5c5r5g993o3sg6r3pc8u75a0u46le41 2322435 2322434 2025-07-10T07:25:49Z Stashbot 7414 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply 2322435 wikitext text/x-wiki lkmryvvay11xn8muua9juqk7ez039uj 2322436 2322435 2025-07-10T07:29:10Z Stashbot 7414 hashar: Restarting CI Jenkins 2322436 wikitext text/x-wiki fdw4slwvyyr2sv5jioqgltllawgu2aw 2322437 2322436 2025-07-10T07:31:25Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P78856 and previous config saved to /var/cache/conftool/dbconfig/20250710-073123-root.json 2322437 wikitext text/x-wiki sglzjfg10btme2mbdpxbw79iyup3ki9 2322438 2322437 2025-07-10T07:39:08Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db1210 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P78857 and previous config saved to /var/cache/conftool/dbconfig/20250710-073907-root.json 2322438 wikitext text/x-wiki qom2w1tqiigqd8fh2ys6kwr3un5j15r 2322439 2322438 2025-07-10T07:39:48Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and A:cp - 2.8.15 upgrade (T398720) 2322439 wikitext text/x-wiki 62e3y53cgcd3t8vhpdx4rqsl1j5p2a9 2322440 2322439 2025-07-10T07:44:27Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - 2.8.15 upgrade (T398720) 2322440 wikitext text/x-wiki 5yc6h9vnpj3t2thveucb3et1ja1qynz 2322441 2322440 2025-07-10T07:44:29Z Stashbot 7414 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2178.codfw.wmnet with reason: Maintenance 2322441 wikitext text/x-wiki pw820s2awxrcd3edi97wqsnastnzkkk 2322442 2322441 2025-07-10T07:44:33Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2178 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P78858 and previous config saved to /var/cache/conftool/dbconfig/20250710-074432-marostegui.json 2322442 wikitext text/x-wiki e6zrvibnow6ymyxpf1fvffk4j6x9azm 2322443 2322442 2025-07-10T07:44:55Z Stashbot 7414 elukey@deploy1003: helmfile [staging] START helmfile.d/services/machinetranslation: sync 2322443 wikitext text/x-wiki lhqodq8y4dekrjbp8llj7wkx6c4o6zf 2322444 2322443 2025-07-10T07:47:28Z Stashbot 7414 elukey@deploy1003: helmfile [staging] DONE helmfile.d/services/machinetranslation: sync 2322444 wikitext text/x-wiki rtbsn5060iskxvbseevpfadwcd7vp8e 2322445 2322444 2025-07-10T07:51:01Z Stashbot 7414 vgutierrez: switching to upload cert globally on upload CDN cluster - T394484 2322445 wikitext text/x-wiki 5oksjtifb371ep38cfwqx3cdr5apecu 2322446 2322445 2025-07-10T07:52:04Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db2178 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P78859 and previous config saved to /var/cache/conftool/dbconfig/20250710-075202-root.json 2322446 wikitext text/x-wiki coct5qqulgysl7ll1qu7d564c48yq3y 2322447 2322446 2025-07-10T07:54:53Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1047.eqiad.wmnet with reason: Maintenance 2322447 wikitext text/x-wiki fwcd6iswzc33e4u6wmexpbsy1o03wxj 2322448 2322447 2025-07-10T07:55:12Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1213.eqiad.wmnet with reason: Maintenance 2322448 wikitext text/x-wiki kfq6qox6afjbad6d3lhya01ee5wqgr2 2322449 2322448 2025-07-10T08:00:46Z Stashbot 7414 moritzm: installing python-urllib3 security updates 2322449 wikitext text/x-wiki 5kv273wdnkqwf81f8cpmnf8g2w68hpu 2322450 2322449 2025-07-10T08:05:54Z Stashbot 7414 klausman: Depooling Liftwing prod in codfw so we can roll out some changes that restart all services (cf. T398533) 2322450 wikitext text/x-wiki cvaads5wmz0qcgnpqwwp4akmixf4ew7 2322451 2322450 2025-07-10T08:07:01Z Stashbot 7414 klausman@cumin1002: conftool action : get/pooled; selector: dnsdisc=inference,name=codfw 2322451 wikitext text/x-wiki 04lu30ddfdw8rdt4qot7mq9jdf0wk5k 2322452 2322451 2025-07-10T08:07:09Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db2178 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P78860 and previous config saved to /var/cache/conftool/dbconfig/20250710-080708-root.json 2322452 wikitext text/x-wiki 0qlszz4n92f6mrne6k5ac4havx1llmq 2322453 2322452 2025-07-10T08:07:42Z Stashbot 7414 klausman@cumin1002: conftool action : set/pooled=false; selector: dnsdisc=inference,name=codfw 2322453 wikitext text/x-wiki 8h06ybcaw2fzcejk9qn6x8pq99au0qc 2322454 2322453 2025-07-10T08:10:15Z Stashbot 7414 moritzm: installing containerd security updates 2322454 wikitext text/x-wiki bsnkgfb1x8te22pe6zqadzmzt7ftvau 2322455 2322454 2025-07-10T08:11:30Z Stashbot 7414 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.9 refs T392179 2322455 wikitext text/x-wiki dosgbh3zirpjuevz78t50h3t942knug 2322456 2322455 2025-07-10T08:12:15Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_codfw and A:cp - 2.8.15 upgrade (T398720) 2322456 wikitext text/x-wiki 4ge78rygz03m9bvn738dcy902uzuipx 2322457 2322456 2025-07-10T08:15:10Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 2.8.15 upgrade (T398720) 2322457 wikitext text/x-wiki 20vz50pmqmjzogfvnk5c0ji6if1rxie 2322458 2322457 2025-07-10T08:22:15Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db2178 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P78861 and previous config saved to /var/cache/conftool/dbconfig/20250710-082213-root.json 2322458 wikitext text/x-wiki 4ku29uwdpydxf2v3ghe0b48heldufy1 2322459 2322458 2025-07-10T08:30:09Z Stashbot 7414 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. 2322459 wikitext text/x-wiki 8ywo4bqcpaq64gkziw6okqt0ppp3r5i 2322460 2322459 2025-07-10T08:31:19Z Stashbot 7414 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. 2322460 wikitext text/x-wiki 4dljzjy4e0gswv72832b5qycz32ylt9 2322461 2322460 2025-07-10T08:37:21Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db2178 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P78863 and previous config saved to /var/cache/conftool/dbconfig/20250710-083719-root.json 2322461 wikitext text/x-wiki 94hg8udzp09h6sxaudwwo1dh0sjrga6 2322462 2322461 2025-07-10T08:40:07Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: sync 2322462 wikitext text/x-wiki foqm9sfantoq7egrf1hs0mfwyticzbk 2322463 2322462 2025-07-10T08:40:32Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: sync 2322463 wikitext text/x-wiki s111loj9jwf3mkqy6gsisgy16364t90 2322464 2322463 2025-07-10T08:45:18Z Stashbot 7414 moritzm: installing setuptools security updates 2322464 wikitext text/x-wiki mgjp9bktjv2vwq8d5kfmojmzcals2vi 2322465 2322464 2025-07-10T08:51:19Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 2.8.15 upgrade (T398720) 2322465 wikitext text/x-wiki 7sn4ebq4m5ezn2gk2355grku8r6a3jd 2322466 2322465 2025-07-10T08:53:06Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_codfw and A:cp - 2.8.15 upgrade (T398720) 2322466 wikitext text/x-wiki lrrwh27qx0scu3gp1uc1hbe7d6e10yr 2322467 2322466 2025-07-10T09:02:25Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2322467 wikitext text/x-wiki ht8xookciayjbja3qjeqyni9nau3eh7 2322468 2322467 2025-07-10T09:02:40Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2322468 wikitext text/x-wiki 2m7u33psfub9j06jtg0qztc9d3vwhe4 2322469 2322468 2025-07-10T09:05:52Z Stashbot 7414 vgutierrez: restarting purged on cp5017 2322469 wikitext text/x-wiki 74ytrncnp746ff210q6yqbpmawsqtoc 2322470 2322469 2025-07-10T09:12:54Z Stashbot 7414 fceratto@cumin1002: dbctl commit (dc=all): 'Update db2240 T397163', diff saved to https://phabricator.wikimedia.org/P78865 and previous config saved to /var/cache/conftool/dbconfig/20250710-091250-fceratto.json 2322470 wikitext text/x-wiki elrfab0k2icgwbnbz6ule1yr3s2j7dh 2322471 2322470 2025-07-10T09:14:08Z Stashbot 7414 fceratto@cumin1002: START - Cookbook sre.mysql.pool db2161 gradually with 4 steps - Pooling in 2322471 wikitext text/x-wiki l1tiijtz89dki3g2u8iaxgtw0zv27ls 2322472 2322471 2025-07-10T09:14:12Z Stashbot 7414 fceratto@cumin1002: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2161 gradually with 4 steps - Pooling in 2322472 wikitext text/x-wiki 62yni45w7tlpgplogwfxf8nccawbtm5 2322474 2322472 2025-07-10T09:15:06Z Stashbot 7414 fceratto@cumin1002: START - Cookbook sre.mysql.pool db2240 gradually with 4 steps - Pooling in 2322474 wikitext text/x-wiki msf3k5t6is4oigso0ws3ur4u3n204jt 2322475 2322474 2025-07-10T09:15:10Z Stashbot 7414 fceratto@cumin1002: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2240 gradually with 4 steps - Pooling in 2322475 wikitext text/x-wiki avzylx240ym29eijjztqw8ihpxfrt0a 2322480 2322475 2025-07-10T09:43:22Z Stashbot 7414 moritzm: installing initramfs-tools bugfix updates from Bookworm point release 2322480 wikitext text/x-wiki o59dnea8omg8toliqoal6w617t9rlex 2322481 2322480 2025-07-10T09:44:04Z Stashbot 7414 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply 2322481 wikitext text/x-wiki o7awin11pv3zsw9nlnqeazm9sx33h7u 2322482 2322481 2025-07-10T09:44:32Z Stashbot 7414 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply 2322482 wikitext text/x-wiki 7nfn6vvvalvo7lrehhcpy8gqjo8u96r 2322483 2322482 2025-07-10T09:45:39Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: apply 2322483 wikitext text/x-wiki 0ngntjpj19uiait78ao19vbaun5r5gk 2322484 2322483 2025-07-10T09:45:49Z Stashbot 7414 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply 2322484 wikitext text/x-wiki mr36jzghaejyu7yib31xias7n1t3tmi 2322485 2322484 2025-07-10T10:04:53Z Stashbot 7414 vgutierrez: resetting eqiad.resource-topic offsets for cp5017 consumer group 2322485 wikitext text/x-wiki 0fl2pfgql2zbx5e3n1kz1ripy8acr25 2322486 2322485 2025-07-10T10:04:54Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/changeprop: apply 2322486 wikitext text/x-wiki 7zijvdbcf9pyd3vcwostux4zq7isx6w 2322487 2322486 2025-07-10T10:05:03Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/changeprop: apply 2322487 wikitext text/x-wiki sevlckdnw21ilaaxklhzwxypt15i1qs 2322488 2322487 2025-07-10T10:05:09Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: apply 2322488 wikitext text/x-wiki 5uwlmrkiokmudn1frdp8v5u77qrumoi 2322489 2322488 2025-07-10T10:05:19Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply 2322489 wikitext text/x-wiki s10dsd4m1cnhw9pctg9a5fxq748ezwf 2322490 2322489 2025-07-10T10:05:24Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply 2322490 wikitext text/x-wiki exn7vwuw4vutg94lu74rl8lyyba6jhw 2322491 2322490 2025-07-10T10:05:38Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply 2322491 wikitext text/x-wiki 9i9fv6hzjllcn74m23xg6ryajcrk7zx 2322492 2322491 2025-07-10T10:05:42Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: apply 2322492 wikitext text/x-wiki r2t0gytcrhwlgaqn0xrogckczs8jy2z 2322493 2322492 2025-07-10T10:05:49Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply 2322493 wikitext text/x-wiki 5cv52bl2ym3syswrwnrdl3c5d7jdk2r 2322494 2322493 2025-07-10T10:33:17Z Stashbot 7414 elukey: kafka preferred-replica-election on kafka-main2010 2322494 wikitext text/x-wiki eslk5b364j4nyikhw4i10tnh2oodgi5 2322502 2322494 2025-07-10T11:04:09Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1039', diff saved to https://phabricator.wikimedia.org/P78867 and previous config saved to /var/cache/conftool/dbconfig/20250710-110408-marostegui.json 2322502 wikitext text/x-wiki 5nxka8hl33o57viyjy74sifo0e0ebge 2322503 2322502 2025-07-10T11:04:16Z Stashbot 7414 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1039.eqiad.wmnet with reason: Maintenance 2322503 wikitext text/x-wiki toby9raxlvs9oov0frfq2vl087wchkg 2322504 2322503 2025-07-10T11:06:36Z Stashbot 7414 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade Replica to GitLab 18.0 2322504 wikitext text/x-wiki 6zghq6tg3vy4w4oclgc2duuq2reaos1 2322505 2322504 2025-07-10T11:09:29Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 2.8.15 upgrade (T398720) 2322505 wikitext text/x-wiki olb63413hiix5kywgpd2wz6exxrin1v 2322506 2322505 2025-07-10T11:14:15Z Stashbot 7414 slyngshede@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 2.8.15 upgrade (T398720) 2322506 wikitext text/x-wiki gycrzw0zptejjasygfbgyr3kpa6xkai 2322508 2322506 2025-07-10T11:21:51Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1007.eqiad.wmnet'] 2322508 wikitext text/x-wiki 7ztbza4kii251ydcn537084mc8mhxha 2322509 2322508 2025-07-10T11:24:59Z Stashbot 7414 vgutierrez: rolling restart of purged in eqsin 2322509 wikitext text/x-wiki 0ot2tmd7klmo4ld6jeqmi7ycvb3cy81 2322511 2322509 2025-07-10T11:29:41Z Stashbot 7414 andrew@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1007.eqiad.wmnet'] 2322511 wikitext text/x-wiki 6hslot52z2tekhyx97ze82pfxufy322 2322512 2322511 2025-07-10T11:30:36Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/changeprop: apply 2322512 wikitext text/x-wiki n9dsadfc01laumnkec79yauf3w0sjn8 2322513 2322512 2025-07-10T11:30:44Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephosd1007.eqiad.wmnet with OS bookworm 2322513 wikitext text/x-wiki b5bshtkprkjl638teyyhih19tcuskwp 2322514 2322513 2025-07-10T11:30:49Z Stashbot 7414 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/changeprop: apply 2322514 wikitext text/x-wiki gmnugr6cu8qaftwytnp371w8rojakpj 2322515 2322514 2025-07-10T11:33:18Z Stashbot 7414 fceratto@cumin1002: START - Cookbook sre.mysql.sanitarium_restart 2322515 wikitext text/x-wiki 9cmzn8hsurum2b8msv62jmridpqn8ns 2322517 2322515 2025-07-10T11:34:59Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply 2322517 wikitext text/x-wiki bn3rq1agya9d88f8y5xy08i104xgana 2322518 2322517 2025-07-10T11:35:12Z Stashbot 7414 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply 2322518 wikitext text/x-wiki bpuu8p6yo4aiac9q15x44yvgsjg7r3m 2322519 2322518 2025-07-10T11:35:17Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: apply 2322519 wikitext text/x-wiki m3casudxeu5xxulrptakfa0w5rhkbnt 2322520 2322519 2025-07-10T11:35:33Z Stashbot 7414 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply 2322520 wikitext text/x-wiki h28xl5qjn0y1k8hx5dhe9hk5d7y90tc 2322522 2322520 2025-07-10T11:39:20Z Stashbot 7414 fceratto@cumin1002: END (PASS) - Cookbook sre.mysql.sanitarium_restart (exit_code=0) 2322522 wikitext text/x-wiki 70lwvl480eetfel1fhhqn8nwb8890p2 2322524 2322522 2025-07-10T11:41:19Z Stashbot 7414 andrew@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1007.eqiad.wmnet with OS bookworm 2322524 wikitext text/x-wiki b4ylfen720fyaei3fkazkpnwrxjhwzm 2322525 2322524 2025-07-10T11:44:28Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephosd1007.eqiad.wmnet with OS bookworm 2322525 wikitext text/x-wiki dywtfs0ajukjhh7br9wx227vdirjhyv 2322526 2322525 2025-07-10T11:46:10Z Stashbot 7414 fceratto@cumin1002: START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis mediawikiwiki, testwiki in section s5 2322526 wikitext text/x-wiki m0itiweeu7bdzcivntui8jg2j59dk0y 2322527 2322526 2025-07-10T11:47:37Z Stashbot 7414 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1200.eqiad.wmnet with reason: Maintenance 2322527 wikitext text/x-wiki 8gkcwhf226cperwp3my2c0pfnm25577 2322528 2322527 2025-07-10T11:47:40Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1200 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P78869 and previous config saved to /var/cache/conftool/dbconfig/20250710-114739-marostegui.json 2322528 wikitext text/x-wiki fn9j74zuybjqgj4qoifeuw471j6wy3t 2322529 2322528 2025-07-10T11:49:44Z Stashbot 7414 andrew@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1007.eqiad.wmnet with OS bookworm 2322529 wikitext text/x-wiki geu181lxn42h9p1soaktoob8xiosop5 2322533 2322529 2025-07-10T11:51:31Z Stashbot 7414 fceratto@cumin1002: END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Checking sanitization for wikis mediawikiwiki, testwiki in section s5 2322533 wikitext text/x-wiki 41ie10pp8w489szzjsmbaoy5rpsjsb9 2322535 2322533 2025-07-10T11:52:03Z Stashbot 7414 fceratto@cumin1002: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis mediawikiwiki, testwiki in section s5 2322535 wikitext text/x-wiki gp38ddliavu2etg22m5byro6kz4zyf5 2322536 2322535 2025-07-10T11:53:17Z Stashbot 7414 andrew@cumin1003: START - Cookbook sre.hosts.reimage for host cloudcephosd1007.eqiad.wmnet with OS bookworm 2322536 wikitext text/x-wiki qticn8nx3l7he4nanmpu41zkng86jfq 2322537 2322536 2025-07-10T11:53:36Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 2.8.15 upgrade (T398720) 2322537 wikitext text/x-wiki aw55pvwwonypi2f6trduqofodjag8n5 2322538 2322537 2025-07-10T11:55:36Z Stashbot 7414 marostegui@cumin1002: dbctl commit (dc=all): 'db1200 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P78870 and previous config saved to /var/cache/conftool/dbconfig/20250710-115534-root.json 2322538 wikitext text/x-wiki 5bpkcf95gjesttgzim21umpso1flgqm 2322539 2322538 2025-07-10T11:57:01Z Stashbot 7414 slyngshede@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 2.8.15 upgrade (T398720) 2322539 wikitext text/x-wiki 3m2rbq7ts5xfxixh0m3umjpo75o86k7 0 11471 2322282 2284533 2025-07-09T18:56:24Z Krinkle 44 2322282 wikitext text/x-wiki ixnurdw5gd0gfisme7qe0tc6g556h1g 0 17290 2322082 2321658 2025-07-09T13:13:58Z Stashbot 7414 James_F: Zuul: [labs/toollabs] Archive repository, for T398202 2322082 wikitext text/x-wiki aa3oesqgrl3zp27nun54x1hllje369z 2322117 2322082 2025-07-09T13:48:20Z Stashbot 7414 Daimona: (beta) mwscript CampaignEvents:UpdateCountriesColumn --wiki metawiki --exceptions /srv/mediawiki/php-master/extensions/CampaignEvents/maintenance/countryExceptionMappings.csv # T397270 2322117 wikitext text/x-wiki ml88z7rbgp7ruhmqd7hnavlur039yt7 2322214 2322117 2025-07-09T15:11:28Z Stashbot 7414 dancy: Upgraded gitlab-runner to v18.0.3 in gitlab-cloud-runners 2322214 wikitext text/x-wiki djwkvhdtnivzmm0lv0yz8h0okwjxgec 2322237 2322214 2025-07-09T16:03:01Z Stashbot 7414 bd808: removed role::aptly::server from deployment-deploy04 (T399076) 2322237 wikitext text/x-wiki lt8sh6zgw13mr95up8a6b82fp6eudbj 498 30942 2322125 2321854 2025-07-09T13:51:55Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.reactivate 2322125 wikitext text/x-wiki dr58xqdhzh01z6cbodhz6zmf4vbswhu 2322126 2322125 2025-07-09T13:52:03Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) 2322126 wikitext text/x-wiki ggpdq9fbxtul35glw6g4v3mp21gqqy9 2322186 2322126 2025-07-09T14:37:33Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.reactivate 2322186 wikitext text/x-wiki eo85dylx5m2luff3sxa925s1jluxjum 2322187 2322186 2025-07-09T14:37:39Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) 2322187 wikitext text/x-wiki i8lo4kmum539coh7msugdj9gxmp3kst 2322188 2322187 2025-07-09T14:39:41Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.reactivate 2322188 wikitext text/x-wiki ofgodpearpgqfoaofw3j4qji6wbp59u 2322194 2322188 2025-07-09T14:43:59Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) 2322194 wikitext text/x-wiki ppgg19ofqd4fsi1xx730n0etrmo6c87 2322208 2322194 2025-07-09T15:04:05Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.upgrade_mons (T306820) 2322208 wikitext text/x-wiki nnlzur90voobo9robug5okssbihplue 2322223 2322208 2025-07-09T15:28:43Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.upgrade_mons (exit_code=0) 2322223 wikitext text/x-wiki ouf6vrja5mt5sysrybx1ivxgnm7y3l3 2322265 2322223 2025-07-09T18:01:05Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.upgrade_osds (T306820) 2322265 wikitext text/x-wiki 51vploncra003a4i4hovdtkfe5hllwn 2322358 2322265 2025-07-09T23:05:28Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.upgrade_osds (exit_code=99) 2322358 wikitext text/x-wiki mthoy4sh2t3gfvnvcb3qsqwel9arryq 2322396 2322358 2025-07-10T04:50:53Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.reactivate 2322396 wikitext text/x-wiki 3d325sttbw72so7yfpuudm3wpoxh972 2322397 2322396 2025-07-10T04:54:54Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) 2322397 wikitext text/x-wiki iz46o8acbmsf6wr2719e74na67fmm21 0 443085 2322507 1864901 2025-07-10T11:15:26Z CGoubert-WMF 32775 /* Main dashboards */ 2322507 wikitext text/x-wiki a4alb0c8tw12z0zqm2zzgv8o4mxdyt4 498 443261 2322530 2299700 2025-07-10T11:49:52Z Stashbot 7414 wmbot~dcaro@hephaestus: START - Cookbook wmcs.openstack.cloudvirt.vm_console 2322530 wikitext text/x-wiki nu4hvkrlrhydokrlhhilkvuj0399xrc 2322531 2322530 2025-07-10T11:51:00Z Stashbot 7414 wmbot~dcaro@hephaestus: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) 2322531 wikitext text/x-wiki hx64oe8yz3bouh0pqua2iypbub5q4tm 2322532 2322531 2025-07-10T11:51:05Z Stashbot 7414 wmbot~dcaro@hephaestus: START - Cookbook wmcs.openstack.cloudvirt.vm_console 2322532 wikitext text/x-wiki dryks2uemc9gtg61ih6koubooxujzte 2322534 2322532 2025-07-10T11:52:02Z Stashbot 7414 wmbot~dcaro@hephaestus: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) 2322534 wikitext text/x-wiki 3mfpx4t1q9lf1i79ezv9msw3luenofg 0 443291 2322185 2294556 2025-07-09T14:36:12Z XCollazo-WMF 31846 /* Things to do every Tuesday */ Fix Airflow Main instance link 2322185 wikitext text/x-wiki 0793dp6tun89bzjs4fu1808wsnvwq0t 0 444177 2322264 2227170 2025-07-09T17:58:09Z SSingh (WMF) 13928 add prometheus metrics text 2322264 wikitext text/x-wiki hywwlhks1wl71m29hddplirlgu0imna 0 447343 2322495 2302511 2025-07-10T10:35:40Z JWodstrcil (WMF) 24826 update gitlab test instance name 2322495 wikitext text/x-wiki 2zfqymbx7j31o7vwbt8bdzh3vphpulb 0 448236 2322497 2321925 2025-07-10T10:44:38Z JWodstrcil (WMF) 24826 2322497 wikitext text/x-wiki g7ixggvxljolnmcb7y8fohnieyu3y87 0 448239 2322501 2321976 2025-07-10T11:02:07Z JWodstrcil (WMF) 24826 2322501 wikitext text/x-wiki 77yhavgap1bvvxqlvbjdmvq8bpwbl8w 0 448262 2322523 2192589 2025-07-10T11:40:56Z JWodstrcil (WMF) 24826 2322523 wikitext text/x-wiki omdapcb1jhcv5todwo87ptv0nxt20fu 0 448359 2322044 2272736 2025-07-09T12:15:20Z ABran-WMF 37196 fix blubber links 2322044 wikitext text/x-wiki t1l4nkyvddiri2nvequbx2qdw4rfres 0 448411 2322510 2318238 2025-07-10T11:25:48Z JWodstrcil (WMF) 24826 2322510 wikitext text/x-wiki sxusz77mf64a81mi4glyl9igguz3hib 2 448673 2322269 2266095 2025-07-09T18:10:08Z Ottomata 88 /* Avoid using team names in code or systems. */ 2322269 wikitext text/x-wiki 0lcz1t8wjxjsd9fbqh2s8zlq7d0gyy2 0 449138 2322516 2318262 2025-07-10T11:34:55Z JWodstrcil (WMF) 24826 2322516 wikitext text/x-wiki 4hrc1fk8i5bpp41hyq3lx1nr54a6ck8 0 449160 2322360 2321852 2025-07-10T00:01:58Z Dexbot 30554 Bot: Updating the report 2322360 wikitext text/x-wiki 7a4vdm9trkwrjzmgabiq4r6w0fq9s3t 2322473 2322360 2025-07-10T09:15:02Z Dexbot 30554 Bot: Updating the report 2322473 wikitext text/x-wiki jyl9cdwri57lafdptgyqemk3e24nw0o 0 451218 2322500 2191892 2025-07-10T11:00:31Z JWodstrcil (WMF) 24826 2322500 wikitext text/x-wiki lml43dewzxpwkiu403t5kmrzpam1txt 0 453101 2322261 2310867 2025-07-09T17:45:31Z AStein-WMF 39485 added a new link to the FR tech analytics stack 2322261 wikitext text/x-wiki 5qy99ino8eypeye83lde66ni0wqbmp7 0 453107 2322255 2274465 2025-07-09T17:25:13Z Jgreen (wmf) 18 /* SSH Config file */ 2322255 wikitext text/x-wiki ch63kznf5vgzmjxuqchuu0hqn5fy4i0 2322256 2322255 2025-07-09T17:25:37Z Jgreen (wmf) 18 /* SSH Fingerprints */ 2322256 wikitext text/x-wiki cbbupdoyp47kwny4x61ta0bflj0ifhw 2322257 2322256 2025-07-09T17:28:35Z Jgreen (wmf) 18 /* SSH Fingerprints */ 2322257 wikitext text/x-wiki kahsjzcnb0mvpb97z93t92ikvguy3a3 0 453463 2322149 2320794 2025-07-09T13:59:23Z CGoubert-WMF 32775 Add Chartmuseum reboot doc 2322149 wikitext text/x-wiki 3vkvdyx08hsz8wispuks6bgcwsyuiha 2322152 2322149 2025-07-09T14:00:05Z CGoubert-WMF 32775 /* Chartmuseum */ 2322152 wikitext text/x-wiki 2btpusszch0wty3z512xw538iui4tlb 0 455709 2322496 2322004 2025-07-10T10:39:41Z JWodstrcil (WMF) 24826 2322496 wikitext text/x-wiki tbqqmnj72emcs35j1ow4qshh5adn7se 0 459013 2322200 2319595 2025-07-09T14:55:38Z MFossati (WMF) 29357 /* Service Level Objectives */ update pipeline freshness target 2322200 wikitext text/x-wiki m14ko3c5x27oo18qriuw90t0bu6ql8o 2322521 2322200 2025-07-10T11:36:19Z MFossati (WMF) 29357 /* Teams */ major teams and components update 2322521 wikitext text/x-wiki gi88aitqhqkyro9j6k20cajevbhj8ll 0 459037 2322096 2322001 2025-07-09T13:36:01Z Taavi-WMF 41365 Taavi-WMF moved page [[GitLab/Object/ Storage]] to [[GitLab/Object/Storage]]: rm space 2322001 wikitext text/x-wiki dzc6a6olpq1nza6ksimafztikz6ru3s 0 459038 2322097 2025-07-09T13:36:01Z Taavi-WMF 41365 Taavi-WMF moved page [[GitLab/Object/ Storage]] to [[GitLab/Object/Storage]]: rm space 2322097 wikitext text/x-wiki 8xuy38a6v5fuagjt334zm9a156e6zv7 0 459039 2322260 2025-07-09T17:45:04Z AStein-WMF 39485 Created blank page 2322260 wikitext text/x-wiki phoiac9h4m842xq45sp7s6u21eteeq1 2322262 2322260 2025-07-09T17:52:29Z AStein-WMF 39485 created new page for analytics stack and added trino entry 2322262 wikitext text/x-wiki cj50qtasnw96c8fb4hnu1srqqqu13tg 2322263 2322262 2025-07-09T17:55:54Z AStein-WMF 39485 update formatting 2322263 wikitext text/x-wiki 1w60k2qd8kr2o5a1yxmyv498ph23z8w 2322266 2322263 2025-07-09T18:04:51Z AStein-WMF 39485 added dbt and dagster to analytics docs 2322266 wikitext text/x-wiki 7u9zqxv1eiuf6k71zn9scpxvziissrr 2322267 2322266 2025-07-09T18:05:26Z AStein-WMF 39485 fix formatting 2322267 wikitext text/x-wiki 2yj8sse9eifib3w8ika3oagam108g2c 2322270 2322267 2025-07-09T18:16:00Z AStein-WMF 39485 add more to analytics stack docs 2322270 wikitext text/x-wiki sarhjg9gtn6abet3uywkss42hh94z50 2322271 2322270 2025-07-09T18:21:46Z AStein-WMF 39485 add mariadb to analytics docs 2322271 wikitext text/x-wiki 81engo84k7uq3r961yngq74mtdqc9b3 0 459040 2322345 2025-07-09T21:39:45Z CDanis (WMF) 13326 Created page with "{{irdoc|status=draft}} == Summary == {{Incident scorecard | task = | paged-num = | responders-num = | coordinators = taavi, cdanis | start = 2025-05-08 17:16 | end = 2025-05-08 17:30 | metrics = Traffic SLO? | impact = For a duration of about 10 minutes, some European users received error messages or very slow page load times. }} [[File:2025-05-08 all-time traffic record.png|thumb|https://grafana.wikimedia.org/goto/GRHGOusNR?orgId=1]] The announcement of the electio..." 2322345 wikitext text/x-wiki 6tb0uvy3o0yoylrio83ini3rlbcymje 2322346 2322345 2025-07-09T21:41:24Z CDanis (WMF) 13326 2322346 wikitext text/x-wiki so5ab02txntw7s968wp2hygfyb4ne41 2322347 2322346 2025-07-09T21:44:11Z CDanis (WMF) 13326 /* What went well? */ 2322347 wikitext text/x-wiki 583i2177qbygdxzfbsl80ms533ywvc2 2322351 2322347 2025-07-09T21:52:57Z CDanis (WMF) 13326 /* What went well? */ 2322351 wikitext text/x-wiki ejumys5d2vg4odeeqboshwcid8ssogv