Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.46.0-wmf.22 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2398471 2398456 2026-04-02T12:03:30Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2398471 wikitext text/x-wiki f62gd7hkag9ke2oj0ufbvzrrqhw79lo 2398481 2398471 2026-04-02T12:16:33Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2398481 wikitext text/x-wiki 90jstejeg0actn3ekukr40p4ojxg6up 2398534 2398481 2026-04-02T13:06:19Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2398534 wikitext text/x-wiki 8wvd35q66xg9v3i3p75zx1qncljdtk2 2398536 2398534 2026-04-02T13:11:18Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2398536 wikitext text/x-wiki lu9qmce10k7l9u6eni1rfesaz3cx5f2 2398575 2398536 2026-04-02T14:41:33Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor 2398575 wikitext text/x-wiki dbj3a8n3mdm3h490hvq62huolfs0pis 2398578 2398575 2026-04-02T14:45:41Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor 2398578 wikitext text/x-wiki lqi31m3oqkcyn455uxtk0tj1gq5ysef 2398607 2398578 2026-04-02T15:16:19Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission 2398607 wikitext text/x-wiki l68h4zce1szfhlfkudz4f98av5lvu7u 2398612 2398607 2026-04-02T15:25:16Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission 2398612 wikitext text/x-wiki nez3yfyzzqyq4otpi1d54hn5ppgqhob 2398624 2398612 2026-04-02T15:46:56Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2398624 wikitext text/x-wiki pcr0irgbyp7wa3temfbw6drksgsrc6b 2398632 2398624 2026-04-02T16:00:55Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2398632 wikitext text/x-wiki 90v5xkae4fqsglry7cc1paiooyb84i5 2398644 2398632 2026-04-02T16:15:19Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2398644 wikitext text/x-wiki sapy101nkxqe45uccmfcg3skjvcf8ar 2398645 2398644 2026-04-02T16:26:17Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api 2398645 wikitext text/x-wiki dq9smgk4g4robz52p59d0lylp8prnnh 2398649 2398645 2026-04-02T16:32:15Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway 2398649 wikitext text/x-wiki 6und1gjlm3wt8rvkbngaj5ta32eeay4 2398650 2398649 2026-04-02T16:36:18Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway 2398650 wikitext text/x-wiki sxuenvfozk916dk5c5egp0mvgrxbzc4 2398653 2398650 2026-04-02T16:47:59Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission 2398653 wikitext text/x-wiki a5so88ndagvmxaj0cftbz7qhza5fvdm 2398655 2398653 2026-04-02T16:50:55Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission 2398655 wikitext text/x-wiki 90f5s19x92tdgnvytg71le3g38oqind 2398670 2398655 2026-04-02T17:10:23Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers 2398670 wikitext text/x-wiki mf5pt6t38wyr6d0lgv7qr311106y3qs 2398683 2398670 2026-04-02T17:29:33Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers 2398683 wikitext text/x-wiki fmepzlqonu7h8qgpmy84h8wbbuqv7tm 0 4108 2398502 2398458 2026-04-02T12:34:51Z ScheduleDeploymentBot 37566 Add [[gerrit:1267008]] to Thursday, April 02 UTC afternoon backport window 2398502 wikitext text/x-wiki g0dwi9fgk75cuig3q2a15p1z0f87blq 2398516 2398502 2026-04-02T12:45:09Z JKilani-WMF 37651 2398516 wikitext text/x-wiki 4116virhovchea6iht6lhcttmx6gfxr 2398524 2398516 2026-04-02T12:49:01Z JKilani-WMF 37651 Day 8 - DC Switchover 2398524 wikitext text/x-wiki 3w3tjj9noza019of7o3j7du0ayfvv43 2398532 2398524 2026-04-02T13:01:49Z JKilani-WMF 37651 2398532 wikitext text/x-wiki p0pzzs4o7xcrqz55uaygrz1g5esuhxk 2398547 2398532 2026-04-02T13:50:20Z ScheduleDeploymentBot 37566 Add [[gerrit:1267056]] to Thursday, April 02 UTC afternoon backport window 2398547 wikitext text/x-wiki msuvm8pnah1nqv3py3w7xf0kqujhhue 2398629 2398547 2026-04-02T15:52:23Z ATomasevich (WMF) 14799 /* {{Deployment_day|date=2026-04-02}} */ Bernard is taking this over 2398629 wikitext text/x-wiki 1u4g5af1uyzk0njo6migeih01dxn1yj 2398648 2398629 2026-04-02T16:31:20Z ScheduleDeploymentBot 37566 Add [[gerrit:1267008]] to Thursday, April 02 UTC late backport window 2398648 wikitext text/x-wiki 0w9opg5starnwd98x66nt0ekilx07tk 2398718 2398648 2026-04-02T20:57:17Z ScheduleDeploymentBot 37566 Add [[gerrit:1267204]] to Thursday, April 02 UTC late backport window 2398718 wikitext text/x-wiki 9hrctuoe2s00fvnhgdx3abbxnjt30li 498 6726 2398470 2398465 2026-04-02T12:01:13Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2398470 wikitext text/x-wiki 1leoagbohivjzq9gelvocpdre7kmd0z 2398531 2398470 2026-04-02T13:00:21Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2398531 wikitext text/x-wiki bkarwry1tluwm4jlkcp5w3s50n5q1ab 2398533 2398531 2026-04-02T13:04:48Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2398533 wikitext text/x-wiki 91ll7rt1i3d8wcxhm2b2jjnxvcas2nm 2398573 2398533 2026-04-02T14:36:19Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor 2398573 wikitext text/x-wiki 2pq7rd2utytptljl06utl4zcamcf8zp 2398574 2398573 2026-04-02T14:38:13Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor 2398574 wikitext text/x-wiki qyu000ipixlbuuciapuqu31e88pebfc 2398593 2398574 2026-04-02T15:04:31Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission 2398593 wikitext text/x-wiki efh66vbc43wcqdy779bc0y9v2wxtc1y 2398602 2398593 2026-04-02T15:12:13Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission 2398602 wikitext text/x-wiki b8ag8rru31psu1abloflgr9l37hd5g3 2398619 2398602 2026-04-02T15:32:22Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api 2398619 wikitext text/x-wiki qj44huf3n43fofvfrq6t78vft8cu3gb 2398623 2398619 2026-04-02T15:43:13Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api 2398623 wikitext text/x-wiki lnggdy2qa1rir2higt0tdc5lzn3pdf4 2398635 2398623 2026-04-02T16:06:36Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api 2398635 wikitext text/x-wiki 6mzn8wtguq2uejhk2srr7ts0fed6mdj 2398643 2398635 2026-04-02T16:15:13Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api 2398643 wikitext text/x-wiki 26zhd092g2gl4hrw9rqd5vrhgu14vcc 2398646 2398643 2026-04-02T16:27:20Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway 2398646 wikitext text/x-wiki c74gcki2jgkch2fwkn4phpoqvwgst4p 2398647 2398646 2026-04-02T16:30:52Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway 2398647 wikitext text/x-wiki idckb6lu4mx3avkgvygm9f6zy18gxqw 2398651 2398647 2026-04-02T16:38:57Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission 2398651 wikitext text/x-wiki h1a3gkhs8h9u1hde0fz52bpdztf2p9c 2398652 2398651 2026-04-02T16:41:46Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission 2398652 wikitext text/x-wiki eq8yu59dpiyl7hfch3blsv4nymiki45 2398656 2398652 2026-04-02T16:52:41Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers 2398656 wikitext text/x-wiki 6a3dqeebcee33lrsbotd0mi7d4wf1k6 2398669 2398656 2026-04-02T17:08:40Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers 2398669 wikitext text/x-wiki 4iiut5au8z1bfz9se6x6yaewf3lk64w 0 7919 2398472 2398469 2026-04-02T12:09:36Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1373.eqiad.wmnet with OS trixie 2398472 wikitext text/x-wiki ocmqsxvbk2q99ork9q0o8g5u05152ro 2398473 2398472 2026-04-02T12:09:47Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1374.eqiad.wmnet with OS trixie 2398473 wikitext text/x-wiki jitbzsprp44ssbjtefiythcgbvjztrk 2398474 2398473 2026-04-02T12:10:28Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1373 2398474 wikitext text/x-wiki 5d5mozbpwxgkr0u01cidia489coetr3 2398475 2398474 2026-04-02T12:10:29Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1373 2398475 wikitext text/x-wiki g44vydhlb3mk45ztelpowd4c0d2huce 2398476 2398475 2026-04-02T12:10:30Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1374 2398476 wikitext text/x-wiki 9nxu8ir0d9zx5w10tsmdkc0hr5b1vth 2398477 2398476 2026-04-02T12:10:31Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1374 2398477 wikitext text/x-wiki nyt8kg5t7gk916gei6ouugpai9te382 2398478 2398477 2026-04-02T12:11:42Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2398478 wikitext text/x-wiki f9at9l9gglp0j452ekwbkv4uwuronb7 2398479 2398478 2026-04-02T12:11:57Z Stashbot 7414 volans@dns1004: START - running authdns-update 2398479 wikitext text/x-wiki fhp09atw60p5jpynqjbxgfur2v91c9r 2398480 2398479 2026-04-02T12:13:46Z Stashbot 7414 volans@dns1004: END - running authdns-update 2398480 wikitext text/x-wiki se7txblon13b81l4y4xokpltu0kx7cr 2398488 2398480 2026-04-02T12:22:32Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1373.eqiad.wmnet with reason: host reimage 2398488 wikitext text/x-wiki 8p4my225kqsmvh8562qa5o6h4a3opzl 2398489 2398488 2026-04-02T12:22:34Z Stashbot 7414 jayme@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1374.eqiad.wmnet with reason: host reimage 2398489 wikitext text/x-wiki pc22ruofv0j7kt8w88ft45r8uwv5uga 2398490 2398489 2026-04-02T12:26:44Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T419635)', diff saved to https://phabricator.wikimedia.org/P90222 and previous config saved to /var/cache/conftool/dbconfig/20260402-122642-fceratto.json 2398490 wikitext text/x-wiki pgsxkluwmsy1bezmear40uwwncwwmk9 2398491 2398490 2026-04-02T12:28:50Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for es1042.eqiad.wmnet 2398491 wikitext text/x-wiki tqdigr2q1vdojwsrgbap8e9ivzpprde 2398492 2398491 2026-04-02T12:28:51Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for es1042.eqiad.wmnet 2398492 wikitext text/x-wiki fkc9w8wyxilkd25vj60ar8yadzjcbts 2398493 2398492 2026-04-02T12:29:18Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1373.eqiad.wmnet with reason: host reimage 2398493 wikitext text/x-wiki d0292n74yc17hdzbsf3l3zd3vl5jpzr 2398494 2398493 2026-04-02T12:30:46Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es1042.eqiad.wmnet: Restoring section 2398494 wikitext text/x-wiki tuo5gj3d9flnbeuzw10vq05l90gq80p 2398495 2398494 2026-04-02T12:31:46Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2398495 wikitext text/x-wiki 89dwwprwjvipwosj12tli8ssfuequle 2398496 2398495 2026-04-02T12:32:46Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool es1042.eqiad.wmnet: Restoring section 2398496 wikitext text/x-wiki 2oxticu5lt04ib3vj78xool9cjt01zg 2398497 2398496 2026-04-02T12:32:50Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1374.eqiad.wmnet with reason: host reimage 2398497 wikitext text/x-wiki eko1oiwds4q0g9931bs88f7zuz0kn6v 2398498 2398497 2026-04-02T12:32:58Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es1042.eqiad.wmnet: Restoring section 2398498 wikitext text/x-wiki mb6zoeqv8cof2l7vjfviknoarmswmb0 2398499 2398498 2026-04-02T12:33:00Z Stashbot 7414 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool es1042.eqiad.wmnet: Restoring section 2398499 wikitext text/x-wiki avhxikqmvza2vvpdduygr8mt1bxyoo1 2398500 2398499 2026-04-02T12:33:11Z Stashbot 7414 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool es1042: Restoring section 2398500 wikitext text/x-wiki 1l4lzoqpdawdoc0289irdv101wdzvf1 2398508 2398500 2026-04-02T12:36:51Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P90224 and previous config saved to /var/cache/conftool/dbconfig/20260402-123650-fceratto.json 2398508 wikitext text/x-wiki maizhe30a2u5txlage1zq3k1if6qw8r 2398509 2398508 2026-04-02T12:41:31Z Stashbot 7414 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. 2398509 wikitext text/x-wiki tgquqpq1b6loejd862ixx6wwkt5l71r 2398513 2398509 2026-04-02T12:44:19Z Stashbot 7414 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. 2398513 wikitext text/x-wiki 8jx7siwkvynnyjhnxf99qd46jayx9h5 2398515 2398513 2026-04-02T12:45:05Z Stashbot 7414 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. 2398515 wikitext text/x-wiki 4tt9bdxemy61tfc9i0u5zhdqtljpb88 2398518 2398515 2026-04-02T12:45:29Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1373.eqiad.wmnet with OS trixie 2398518 wikitext text/x-wiki degulcba75ue5om30wr8swumi5bk7ek 2398520 2398518 2026-04-02T12:45:52Z Stashbot 7414 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. 2398520 wikitext text/x-wiki nirofdlfkc9g8ot2zkkpp81cfx81n3h 2398521 2398520 2026-04-02T12:47:00Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P90225 and previous config saved to /var/cache/conftool/dbconfig/20260402-124659-fceratto.json 2398521 wikitext text/x-wiki bnsccast7qnvweuynegqcy2ivvylybv 2398522 2398521 2026-04-02T12:48:33Z Stashbot 7414 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1042: Restoring section 2398522 wikitext text/x-wiki jrgaeqgwd934mf98a74v49t83kvq937 2398525 2398522 2026-04-02T12:49:21Z Stashbot 7414 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1374.eqiad.wmnet with OS trixie 2398525 wikitext text/x-wiki 5un0u6liewp0wky8n4rig9wpn9ih71n 2398526 2398525 2026-04-02T12:50:19Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2398526 wikitext text/x-wiki nhq224jxgi4ro8hbamd6k19l25me9zu 2398527 2398526 2026-04-02T12:56:38Z Stashbot 7414 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2398527 wikitext text/x-wiki qmttvwwibzifrexm9hru0ysrtnvwd06 2398528 2398527 2026-04-02T12:57:09Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T419635)', diff saved to https://phabricator.wikimedia.org/P90227 and previous config saved to /var/cache/conftool/dbconfig/20260402-125707-fceratto.json 2398528 wikitext text/x-wiki 7cb2jnzj1hxpbqtbprgqc5ytpom7794 2398529 2398528 2026-04-02T12:57:25Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2194.codfw.wmnet with reason: Maintenance 2398529 wikitext text/x-wiki e3payq2xgvvu11wfbgu8pbxcx20158y 2398530 2398529 2026-04-02T12:57:35Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2194 (T419635)', diff saved to https://phabricator.wikimedia.org/P90228 and previous config saved to /var/cache/conftool/dbconfig/20260402-125732-fceratto.json 2398530 wikitext text/x-wiki gzncf1j961vqtsqwjgdassjnh6r20si 2398537 2398530 2026-04-02T13:17:36Z Stashbot 7414 jasmine@cumin1003: START - Cookbook sre.dns.admin DNS admin: pool codfw [reason: no reason specified, T414486] 2398537 wikitext text/x-wiki ckqavupz4ek4codrolaxdffpem0xmu2 2398538 2398537 2026-04-02T13:17:48Z Stashbot 7414 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool codfw [reason: no reason specified, T414486] 2398538 wikitext text/x-wiki gdu2bfhpnif0ni1wx0wjt83tyb62cuw 2398539 2398538 2026-04-02T13:18:35Z Stashbot 7414 jasmine@cumin1003: START - Cookbook sre.discovery.datacenter pool all active/active services in codfw: maintenance - T414486 2398539 wikitext text/x-wiki bv0t7flmaoumgm5ep9n3hcyokcoo3g4 2398540 2398539 2026-04-02T13:29:17Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T419635)', diff saved to https://phabricator.wikimedia.org/P90229 and previous config saved to /var/cache/conftool/dbconfig/20260402-132914-fceratto.json 2398540 wikitext text/x-wiki 3hnagvc68ni9i4lm2u5z4bvreuz5a2e 2398541 2398540 2026-04-02T13:37:45Z Stashbot 7414 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply 2398541 wikitext text/x-wiki gtz6kh8d89haj51vz30nsl0eqxrca9i 2398542 2398541 2026-04-02T13:39:24Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P90230 and previous config saved to /var/cache/conftool/dbconfig/20260402-133923-fceratto.json 2398542 wikitext text/x-wiki 3kt4208iy0x3pcfuv4z07cw67vum21o 2398543 2398542 2026-04-02T13:41:49Z Stashbot 7414 jasmine@cumin1003: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in codfw: maintenance - T414486 2398543 wikitext text/x-wiki 18kimsb7jtnscaq6ig71tzmp1pzpj0z 2398544 2398543 2026-04-02T13:42:59Z Stashbot 7414 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply 2398544 wikitext text/x-wiki 17pinhizccjk8b631k9u136dnsrdp41 2398546 2398544 2026-04-02T13:49:32Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P90231 and previous config saved to /var/cache/conftool/dbconfig/20260402-134931-fceratto.json 2398546 wikitext text/x-wiki hfetfuynqjbpklt5po2y764bw10ahpg 2398548 2398546 2026-04-02T13:51:27Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1266985|Fix suggestion mode availability check (T422143)]] 2398548 wikitext text/x-wiki iybpywyrfxo7pxm0e0cf78pdzus7lip 2398549 2398548 2026-04-02T13:53:10Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: sync-world failed: <CalledProcessError> Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py --http-proxy http://webproxy:8080 --https-proxy http://webproxy:8080 /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.46.0-wmf.21,1.46.0-wmf.22,next --multiversion-image-basename docker-registry.discovery.wmne 2398549 wikitext text/x-wiki 9f5pnlq7azfekot8pfvr9pd0v6xduv2 2398550 2398549 2026-04-02T13:54:49Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1266985|Fix suggestion mode availability check (T422143)]] 2398550 wikitext text/x-wiki t7qbgdxlwcorgmq3lpuipc2he2mnatz 2398551 2398550 2026-04-02T13:55:46Z Stashbot 7414 lucaswerkmeister-wmde@deploy1003: sync-world failed: <CalledProcessError> Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py --http-proxy http://webproxy:8080 --https-proxy http://webproxy:8080 /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.46.0-wmf.21,1.46.0-wmf.22,next --multiversion-image-basename docker-registry.discovery.wmne 2398551 wikitext text/x-wiki 0aceyqptitpnangae4mc25vhyvlg35h 2398552 2398551 2026-04-02T13:58:55Z Stashbot 7414 esanders@deploy1003: Started scap sync-world: Backport for [[gerrit:1266985|Fix suggestion mode availability check (T422143)]] 2398552 wikitext text/x-wiki m032wwso7w2ep7z0dmgygy5ibw6h9ve 2398553 2398552 2026-04-02T13:59:42Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T419635)', diff saved to https://phabricator.wikimedia.org/P90232 and previous config saved to /var/cache/conftool/dbconfig/20260402-135939-fceratto.json 2398553 wikitext text/x-wiki 6qpbyqm5byfyjzsmdr0n61hyxaoq9w3 2398554 2398553 2026-04-02T13:59:57Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2205.codfw.wmnet with reason: Maintenance 2398554 wikitext text/x-wiki f7zom7oobedkl5oi3nen68yir9hjytf 2398556 2398554 2026-04-02T14:00:07Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2205 (T419635)', diff saved to https://phabricator.wikimedia.org/P90233 and previous config saved to /var/cache/conftool/dbconfig/20260402-140004-fceratto.json 2398556 wikitext text/x-wiki 1v4ydf2e10iim5xzsyzyjb1hspmc7dr 2398557 2398556 2026-04-02T14:00:08Z Stashbot 7414 esanders@deploy1003: sync-world failed: <CalledProcessError> Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py --http-proxy http://webproxy:8080 --https-proxy http://webproxy:8080 /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.46.0-wmf.21,1.46.0-wmf.22,next --multiversion-image-basename docker-registry.discovery.wmnet/restricted/ 2398557 wikitext text/x-wiki 49gxwtwwu1h33c5ry1zxysxd2tpvxx9 2398558 2398557 2026-04-02T14:03:10Z Stashbot 7414 hashar: Jenkins CI: reloading configuration from disk to poll new nodes # T421114 2398558 wikitext text/x-wiki g9b3ljqytv7v56hbjxf8ow7r1chjmbh 2398560 2398558 2026-04-02T14:09:18Z Stashbot 7414 esanders@deploy1003: Started scap sync-world: Backport for [[gerrit:1266985|Fix suggestion mode availability check (T422143)]] 2398560 wikitext text/x-wiki mi4xu5rhwqb929vgp8c907edvbpqswb 2398561 2398560 2026-04-02T14:10:32Z Stashbot 7414 esanders@deploy1003: sync-world failed: <CalledProcessError> Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py --http-proxy http://webproxy:8080 --https-proxy http://webproxy:8080 /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.46.0-wmf.21,1.46.0-wmf.22,next --multiversion-image-basename docker-registry.discovery.wmnet/restricted/ 2398561 wikitext text/x-wiki eo1t7nn454k6vhbmikjlm7eglg9ox8k 2398564 2398561 2026-04-02T14:17:13Z Stashbot 7414 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1267062|Bump maxConnCount]] 2398564 wikitext text/x-wiki 4onafy3ahsj0hwsifg2qac4y1dsnrdz 2398565 2398564 2026-04-02T14:18:25Z Stashbot 7414 ladsgroup@deploy1003: sync-world failed: <CalledProcessError> Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py --http-proxy http://webproxy:8080 --https-proxy http://webproxy:8080 /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.46.0-wmf.21,1.46.0-wmf.22,next --multiversion-image-basename docker-registry.discovery.wmnet/restricted 2398565 wikitext text/x-wiki 8k3saa5tpsktomrmkipig3nmg6l57u2 2398570 2398565 2026-04-02T14:28:30Z Stashbot 7414 moritzm: installing pyasn1 security updates 2398570 wikitext text/x-wiki lyvkqdset9hyb00z3icd7a6mjt41ikv 2398571 2398570 2026-04-02T14:34:55Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205 (T419635)', diff saved to https://phabricator.wikimedia.org/P90236 and previous config saved to /var/cache/conftool/dbconfig/20260402-143452-fceratto.json 2398571 wikitext text/x-wiki 6kj0k97tw21yu3k942mvlruo710ucy0 2398576 2398571 2026-04-02T14:42:17Z Stashbot 7414 moritzm: installing libxml-parser-perl security updates 2398576 wikitext text/x-wiki atso0jqotc5ul52pnnnrfe630ov0byv 2398577 2398576 2026-04-02T14:45:07Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P90237 and previous config saved to /var/cache/conftool/dbconfig/20260402-144500-fceratto.json 2398577 wikitext text/x-wiki dphigls0my9trlw9uti18okl869cgnb 2398579 2398577 2026-04-02T14:51:13Z Stashbot 7414 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply 2398579 wikitext text/x-wiki fkeuvgsxrw36x9s1vsrotfs4gnqp4zc 2398580 2398579 2026-04-02T14:51:41Z Stashbot 7414 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply 2398580 wikitext text/x-wiki t22prrzxqheykjbl9t8ahs230zywqoj 2398581 2398580 2026-04-02T14:52:25Z Stashbot 7414 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply 2398581 wikitext text/x-wiki 6zzdqxjg2gb7iej6jxgvx2mti3c4h6x 2398582 2398581 2026-04-02T14:52:41Z Stashbot 7414 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply 2398582 wikitext text/x-wiki eg1vgs58kaeqwgkn5qpj9gwi6g8i5tl 2398583 2398582 2026-04-02T14:53:40Z Stashbot 7414 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply 2398583 wikitext text/x-wiki 840m4fj5rb465x330fraom1905zfo6d 2398584 2398583 2026-04-02T14:53:55Z Stashbot 7414 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply 2398584 wikitext text/x-wiki 3l7xlyvgr1uk54469e2mtmtdd1cnz8s 2398585 2398584 2026-04-02T14:55:10Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P90239 and previous config saved to /var/cache/conftool/dbconfig/20260402-145508-fceratto.json 2398585 wikitext text/x-wiki h6mf7oqlig7x4hu1c57mr3l2lxxbzry 2398586 2398585 2026-04-02T14:56:39Z Stashbot 7414 swfrench@deploy1003: Started scap sync-world: Manual sync-world to pick up 1267062, 1266985 - T422143 2398586 wikitext text/x-wiki 9w0twdp59ls996hizw54j78qsno9vbz 2398587 2398586 2026-04-02T14:56:44Z Stashbot 7414 pt1979@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mr1-eqiad,mr1-eqiad IPv6 with reason: switching from OSFP to BGP 2398587 wikitext text/x-wiki fo3xh9f53s32xgibw82fqomw7go84e7 2398588 2398587 2026-04-02T14:57:45Z Stashbot 7414 swfrench@deploy1003: sync-world failed: <CalledProcessError> Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py --http-proxy http://webproxy:8080 --https-proxy http://webproxy:8080 /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.46.0-wmf.21,1.46.0-wmf.22,next --multiversion-image-basename docker-registry.discovery.wmnet/restricted/ 2398588 wikitext text/x-wiki fu6l6kgzwvfgdilitsa0d6rfbxr3m5v 2398589 2398588 2026-04-02T14:59:32Z Stashbot 7414 papaul: ongoing maintenance on mr1-eqiad 2398589 wikitext text/x-wiki qa09x00ikpoxqiwbsxze8fps85h8qw1 2398590 2398589 2026-04-02T14:59:42Z Stashbot 7414 swfrench@deploy1003: Started scap sync-world: Manual full-rebuild sync-world to pick up 1267062, 1266985 - T422143 2398590 wikitext text/x-wiki k8k1qua624qddidi1di32gqx1oaqw0q 2398591 2398590 2026-04-02T15:03:04Z Stashbot 7414 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply 2398591 wikitext text/x-wiki igmt8o1yujhfaj7kpe30suxam6l0mfi 2398592 2398591 2026-04-02T15:03:46Z Stashbot 7414 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply 2398592 wikitext text/x-wiki ccg82hzsbzjh9ozalkf69rrhzqg1ly9 2398594 2398592 2026-04-02T15:05:19Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205 (T419635)', diff saved to https://phabricator.wikimedia.org/P90241 and previous config saved to /var/cache/conftool/dbconfig/20260402-150517-fceratto.json 2398594 wikitext text/x-wiki qw0g2fbvoym66p0prz7k833uwhr55d5 2398595 2398594 2026-04-02T15:05:35Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2227.codfw.wmnet with reason: Maintenance 2398595 wikitext text/x-wiki 2rhkmwuiszkirypt8311f4jeo7s5hec 2398596 2398595 2026-04-02T15:05:48Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2227 (T419635)', diff saved to https://phabricator.wikimedia.org/P90242 and previous config saved to /var/cache/conftool/dbconfig/20260402-150542-fceratto.json 2398596 wikitext text/x-wiki eq8ta79slwjui4t637iwb6x4qms6i86 2398597 2398596 2026-04-02T15:06:35Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. 2398597 wikitext text/x-wiki ftg213vlf6wbxcmdgw28nkbioop34uy 2398599 2398597 2026-04-02T15:07:05Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. 2398599 wikitext text/x-wiki 1wlcrxicjegdjo78e3lbd2ik8sse7ay 2398601 2398599 2026-04-02T15:11:40Z Stashbot 7414 moritzm: installing apache2 security updates 2398601 wikitext text/x-wiki r064g9an98lllg83290qvtfhqhp52aa 2398604 2398601 2026-04-02T15:12:45Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2398604 wikitext text/x-wiki bqdae2org1xxfemdgj9hi0ekxih53h1 2398605 2398604 2026-04-02T15:12:59Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2398605 wikitext text/x-wiki kz9m55ibivpzp1bfn1vief3ju1rc2yf 2398609 2398605 2026-04-02T15:22:32Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2398609 wikitext text/x-wiki j75yf9unxgslmyutawirko3jbaakx6p 2398610 2398609 2026-04-02T15:23:08Z Stashbot 7414 papaul: maintenance complete on mr1-eqiad 2398610 wikitext text/x-wiki rmuvc9jb6t2e8duik26gi4mb0g8pwsr 2398611 2398610 2026-04-02T15:23:22Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2398611 wikitext text/x-wiki aetlt8c3kmzinebhj06mlaynacsx17t 2398613 2398611 2026-04-02T15:26:13Z Stashbot 7414 swfrench-wmf: restarted docker-registry-restricted.service on registry200[45] - T422166 2398613 wikitext text/x-wiki 4mj874mesgpvii5cm9ea973gme9hd74 2398614 2398613 2026-04-02T15:26:30Z Stashbot 7414 swfrench@deploy1003: sync-world aborted: Manual full-rebuild sync-world to pick up 1267062, 1266985 - T422143 (duration: 26m 48s) 2398614 wikitext text/x-wiki 46m1bqkr5cdrawu0emgbvcwuj1n38p0 2398616 2398614 2026-04-02T15:27:38Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2398616 wikitext text/x-wiki r6z9y438e5lmlxa7df0xq287rgl1zu0 2398617 2398616 2026-04-02T15:27:46Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2398617 wikitext text/x-wiki 1z0y6dk9sdgk14cswahdywqvsy3cgao 2398618 2398617 2026-04-02T15:31:18Z Stashbot 7414 swfrench-wmf: restarted docker-registry-ml.service on registry200[45] - T422166 2398618 wikitext text/x-wiki 2z576ixnkewvly6ewbndkgbciiscg8v 2398620 2398618 2026-04-02T15:32:35Z Stashbot 7414 moritzm: installing freetype security updates 2398620 wikitext text/x-wiki 8jc3bfdb4fyx85bqy6x7j2prnn13ct1 2398621 2398620 2026-04-02T15:33:01Z Stashbot 7414 swfrench@deploy1003: Started scap sync-world: Manual full-rebuild sync-world to pick up 1267062, 1266985 (attempt 2) - T422143 2398621 wikitext text/x-wiki 8no6xbggsjeejb9ohza6idoqim93k0x 2398622 2398621 2026-04-02T15:39:20Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227 (T419635)', diff saved to https://phabricator.wikimedia.org/P90244 and previous config saved to /var/cache/conftool/dbconfig/20260402-153918-fceratto.json 2398622 wikitext text/x-wiki 8vhzdphxif439p0b0lquhiqdmkwd3bg 2398626 2398622 2026-04-02T15:49:27Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P90245 and previous config saved to /var/cache/conftool/dbconfig/20260402-154925-fceratto.json 2398626 wikitext text/x-wiki pin451ntatrbcajiqy792a0t0ygiv7l 2398627 2398626 2026-04-02T15:50:07Z Stashbot 7414 swfrench@deploy1003: swfrench: Manual full-rebuild sync-world to pick up 1267062, 1266985 (attempt 2) - T422143 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2398627 wikitext text/x-wiki 46ospqvbm3u220wrtguvkmfoz48ufp1 2398628 2398627 2026-04-02T15:51:13Z Stashbot 7414 swfrench@deploy1003: swfrench: Continuing with sync 2398628 wikitext text/x-wiki b6eqygladiy7vc4zkjx2yk93t5y9ff8 2398631 2398628 2026-04-02T15:59:35Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P90246 and previous config saved to /var/cache/conftool/dbconfig/20260402-155934-fceratto.json 2398631 wikitext text/x-wiki pkiday32ku5hvmmu3afh2ksxzexjp5n 2398633 2398631 2026-04-02T16:02:58Z Stashbot 7414 swfrench@deploy1003: Finished scap sync-world: Manual full-rebuild sync-world to pick up 1267062, 1266985 (attempt 2) - T422143 (duration: 29m 56s) 2398633 wikitext text/x-wiki 78yjubbvw6qn84ltffvdkxhp184kait 2398634 2398633 2026-04-02T16:03:41Z Stashbot 7414 Lucas_WMDE: UTC afternoon backport+config window (very belatedly) done ^^ 2398634 wikitext text/x-wiki oykfteyjctompq69wkrk9o7ov9o0raj 2398636 2398634 2026-04-02T16:09:45Z Stashbot 7414 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227 (T419635)', diff saved to https://phabricator.wikimedia.org/P90247 and previous config saved to /var/cache/conftool/dbconfig/20260402-160942-fceratto.json 2398636 wikitext text/x-wiki r6xyb537nvag6z8aeb0eidw0utgu8ti 2398637 2398636 2026-04-02T16:10:00Z Stashbot 7414 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2239.codfw.wmnet with reason: Maintenance 2398637 wikitext text/x-wiki 6uepumx3hfrohbzmdwjs3yyy8omuzkw 2398639 2398637 2026-04-02T16:12:32Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2398639 wikitext text/x-wiki 8obfg7ed3vo79phmtr4hvzp2rl3pyj2 2398640 2398639 2026-04-02T16:12:47Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2398640 wikitext text/x-wiki 3ughl84mq6tdc0b70ycmin20xxgkniw 2398641 2398640 2026-04-02T16:12:55Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2398641 wikitext text/x-wiki pzjmaxj84yxj8iiu0u11causedsdh4v 2398642 2398641 2026-04-02T16:13:01Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2398642 wikitext text/x-wiki reqpoxhw787h5ivtj5mluk3fk9tad7w 2398657 2398642 2026-04-02T16:53:24Z Stashbot 7414 cdobbins@cumin2002: START - Cookbook sre.loadbalancer.admin rebooting P{lvs3009.esams.wmnet} and A:liberica 2398657 wikitext text/x-wiki gpysog0ohfal67l1z4p99n5ejnk717s 2398658 2398657 2026-04-02T16:57:02Z Stashbot 7414 cdobbins@cumin2002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs3009.esams.wmnet} and A:liberica 2398658 wikitext text/x-wiki stp4hk4swrkdrnmqfdhdxf5v9giaql0 2398659 2398658 2026-04-02T16:59:31Z Stashbot 7414 cdobbins@cumin2002: START - Cookbook sre.loadbalancer.admin rebooting P{lvs3008.esams.wmnet} and A:liberica 2398659 wikitext text/x-wiki 4sjd3t39mxc53h04z2sj82e7x884p8f 2398662 2398659 2026-04-02T17:02:25Z Stashbot 7414 swfrench@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. 2398662 wikitext text/x-wiki 7p4ppnjc32xnbtnu4bv821etnxsqoxj 2398663 2398662 2026-04-02T17:03:03Z Stashbot 7414 cdobbins@cumin2002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs3008.esams.wmnet} and A:liberica 2398663 wikitext text/x-wiki ppa5xprsxb2m4o6eorngunr09yrxyv6 2398665 2398663 2026-04-02T17:05:14Z Stashbot 7414 swfrench@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. 2398665 wikitext text/x-wiki 390jxsjx1do89hmf996yc8fxskd7o69 2398666 2398665 2026-04-02T17:05:34Z Stashbot 7414 swfrench@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. 2398666 wikitext text/x-wiki 4oni6xefsqbinqpn8w9i4w81aj5q8xk 2398667 2398666 2026-04-02T17:07:05Z Stashbot 7414 swfrench@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. 2398667 wikitext text/x-wiki p5v5r9acs9bwxx5txruxccf17g9itwx 2398668 2398667 2026-04-02T17:08:15Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. 2398668 wikitext text/x-wiki 29ndadc3x1wptsiy0ws52pp9pqhiq4g 2398671 2398668 2026-04-02T17:10:36Z Stashbot 7414 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. 2398671 wikitext text/x-wiki 3tsddb4taveb3munca9m8onm36lp2ba 2398672 2398671 2026-04-02T17:10:48Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2398672 wikitext text/x-wiki axw667g6ouea4nmo0ol6v92z7f9r8in 2398673 2398672 2026-04-02T17:11:08Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2398673 wikitext text/x-wiki sisr4fgcwb9fphu9g6kumo8qj7drl7j 2398674 2398673 2026-04-02T17:11:31Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. 2398674 wikitext text/x-wiki qq3sltitfdutfwdu603djtdzh49lcxq 2398675 2398674 2026-04-02T17:11:50Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2398675 wikitext text/x-wiki c0uee11220uqc68oyqu19qabxufwn93 2398676 2398675 2026-04-02T17:12:02Z Stashbot 7414 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2398676 wikitext text/x-wiki habwvdkacbu4itji8xmtc9tlnf7gjmm 2398677 2398676 2026-04-02T17:12:13Z Stashbot 7414 daniel@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply 2398677 wikitext text/x-wiki qozqq9ugn7pjy90neu3eqy2d7z0nhun 2398678 2398677 2026-04-02T17:12:40Z Stashbot 7414 daniel@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply 2398678 wikitext text/x-wiki 0ynmfoj3zt4f37rluau93hxztwrqt6r 2398679 2398678 2026-04-02T17:13:57Z Stashbot 7414 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. 2398679 wikitext text/x-wiki 3juu400ps8rr3in326agqmgan01hwt3 2398680 2398679 2026-04-02T17:16:11Z Stashbot 7414 swfrench@dns1004: START - running authdns-update 2398680 wikitext text/x-wiki ekoss868ecqihroumomomin0cwdsnv2 2398681 2398680 2026-04-02T17:18:08Z Stashbot 7414 swfrench@dns1004: END - running authdns-update 2398681 wikitext text/x-wiki mrhatlyud6r3n04x8fz26yk0wzqi38r 2398684 2398681 2026-04-02T17:59:52Z Stashbot 7414 ebysans@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply 2398684 wikitext text/x-wiki 5r0e1e459ywxbnfyvklqpuj5klwat1a 2398685 2398684 2026-04-02T18:00:10Z Stashbot 7414 ebysans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply 2398685 wikitext text/x-wiki 66b37d042sxig5bj0hl0fpshce287ps 2398686 2398685 2026-04-02T18:00:30Z Stashbot 7414 ebysans@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply 2398686 wikitext text/x-wiki 0dn8r02cfp3j3etp8tq79v0lt38eth3 2398687 2398686 2026-04-02T18:00:48Z Stashbot 7414 ebysans@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply 2398687 wikitext text/x-wiki 9v43wdwkze35rhv7o05wmu2opgaykx8 2398689 2398687 2026-04-02T18:24:47Z Stashbot 7414 cdobbins@cumin2002: START - Cookbook sre.loadbalancer.admin rebooting P{lvs5006.eqsin.wmnet} and A:liberica 2398689 wikitext text/x-wiki t7gdvdh95fcmrpq03zsmn4sbmcoxevg 2398690 2398689 2026-04-02T18:28:03Z Stashbot 7414 cdobbins@cumin2002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs5006.eqsin.wmnet} and A:liberica 2398690 wikitext text/x-wiki 31g8fcdi6rjwsq30h146etjykuw7y8x 2398691 2398690 2026-04-02T18:45:52Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.dns.netbox 2398691 wikitext text/x-wiki 4aji92kyghq5oc4asvcja2gdt9turoc 2398692 2398691 2026-04-02T18:51:57Z Stashbot 7414 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns names for new lumen 100g transport - cmooney@cumin1003" 2398692 wikitext text/x-wiki gmlye57bizrwo8eldmleoudn2snxuav 2398693 2398692 2026-04-02T18:52:24Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns names for new lumen 100g transport - cmooney@cumin1003" 2398693 wikitext text/x-wiki i2fzvc7q8g456yovo4f6teiokvcrjls 2398694 2398693 2026-04-02T18:52:25Z Stashbot 7414 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2398694 wikitext text/x-wiki i1hpb67izk372wup8qndyvsjgiw466q 2398695 2398694 2026-04-02T18:55:10Z Stashbot 7414 cmooney@dns2005: START - running authdns-update 2398695 wikitext text/x-wiki 4o1o9oovnpuvcaidwtaooe3vefd7ozt 2398696 2398695 2026-04-02T18:56:34Z Stashbot 7414 cmooney@dns2005: END - running authdns-update 2398696 wikitext text/x-wiki j9kh8upj7we0vry8kznrljb71s9idxt 2398697 2398696 2026-04-02T19:01:19Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2398697 wikitext text/x-wiki o9g35h055c5b3ddoorz7zsrtcifn7g0 2398698 2398697 2026-04-02T19:01:24Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2398698 wikitext text/x-wiki 7dyuhtcbb4i9tdx1fn3szgves062ctu 2398699 2398698 2026-04-02T19:09:13Z Stashbot 7414 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on releases2003.codfw.wmnet with reason: T418109 2398699 wikitext text/x-wiki 15n7rks8ic4gwjewpuz20u2wv33ip49 2398700 2398699 2026-04-02T19:24:12Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2398700 wikitext text/x-wiki qg1t1y5c9lt4bykyataqnt79m0q8964 2398701 2398700 2026-04-02T19:24:17Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply 2398701 wikitext text/x-wiki 06uf5l0xkk0owqugjkybyw7i85p1znm 2398702 2398701 2026-04-02T19:56:29Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2398702 wikitext text/x-wiki sc39uys6liszwzbe6kifdyr121fkir1 2398703 2398702 2026-04-02T19:56:33Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2398703 wikitext text/x-wiki f7tb6z76n60dtpjuqklr9cjmxhlv09k 2398704 2398703 2026-04-02T19:56:48Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2398704 wikitext text/x-wiki gfabwj2dohjp0lbsrw1h72yvkpidqdm 2398705 2398704 2026-04-02T19:56:50Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2398705 wikitext text/x-wiki 4v3br78qvukj5z4h6lfrsf5xc2tesso 2398706 2398705 2026-04-02T19:57:47Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2398706 wikitext text/x-wiki bhkp6tspxi3aapdb6dqk41pp8o96vk4 2398707 2398706 2026-04-02T19:57:49Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2398707 wikitext text/x-wiki cg6vo0aex1o0p503d6avd6y1lnuq24e 2398708 2398707 2026-04-02T20:02:57Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2398708 wikitext text/x-wiki 22u3p3z7eiagh3ok2vatuy865hyvdvp 2398709 2398708 2026-04-02T20:03:03Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2398709 wikitext text/x-wiki lxhshyj0z9yb76lsu5kxvbblimorv6k 2398710 2398709 2026-04-02T20:09:36Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2398710 wikitext text/x-wiki rozvtgeo0fhcn2zl2ff8t08bmggi0jd 2398711 2398710 2026-04-02T20:09:44Z Stashbot 7414 otto@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply 2398711 wikitext text/x-wiki cis5nf5gefvvfmbljphl6txk0bnz13y 2398712 2398711 2026-04-02T20:25:38Z Stashbot 7414 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1264569|zhwikinews: 20th anniversary logo change (T420165)]] 2398712 wikitext text/x-wiki gysbaaf9eukokfadwu8u80xkpus6ogg 2398713 2398712 2026-04-02T20:29:24Z Stashbot 7414 kemayo@deploy1003: 1f616emo, kemayo: Backport for [[gerrit:1264569|zhwikinews: 20th anniversary logo change (T420165)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2398713 wikitext text/x-wiki rqopbe6oq85k7zlthlbzf1flwjjmbwu 2398714 2398713 2026-04-02T20:31:11Z Stashbot 7414 kemayo@deploy1003: 1f616emo, kemayo: Continuing with sync 2398714 wikitext text/x-wiki i4k2eyjd55hobyfi4ukrc44oq0dtcgr 2398715 2398714 2026-04-02T20:37:25Z Stashbot 7414 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1264569|zhwikinews: 20th anniversary logo change (T420165)]] (duration: 11m 46s) 2398715 wikitext text/x-wiki 8on393vve7woerarhgeawayd49gyffg 2398716 2398715 2026-04-02T20:52:11Z Stashbot 7414 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1267008|Add logged-in reader retention instrument (T420490)]] 2398716 wikitext text/x-wiki k4ou3nhrci3pxiwvgwxb2j7g0tivcd2 2398717 2398716 2026-04-02T20:53:52Z Stashbot 7414 kemayo@deploy1003: annet, kemayo: Backport for [[gerrit:1267008|Add logged-in reader retention instrument (T420490)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2398717 wikitext text/x-wiki pugb6vy94vjk84o71m3ccaz366s4jw7 2398719 2398717 2026-04-02T20:59:13Z Stashbot 7414 kemayo@deploy1003: annet, kemayo: Continuing with sync 2398719 wikitext text/x-wiki j0gqumwdbgvwqx8jgpun3rar5fdxspu 2398720 2398719 2026-04-02T21:03:52Z Stashbot 7414 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1267008|Add logged-in reader retention instrument (T420490)]] (duration: 11m 40s) 2398720 wikitext text/x-wiki neugax92lac3ans2ah7ebvfpfgq3ggh 2398722 2398720 2026-04-02T21:15:47Z Stashbot 7414 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1267204|SuggestedLinkEditCheck: fetchSuggestions return a jQuery.Promise]] 2398722 wikitext text/x-wiki 3794hb8wdfjjxlrjc8c74gvwgbsfvbv 2398723 2398722 2026-04-02T21:17:26Z Stashbot 7414 kemayo@deploy1003: kemayo: Backport for [[gerrit:1267204|SuggestedLinkEditCheck: fetchSuggestions return a jQuery.Promise]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2398723 wikitext text/x-wiki 87ehmsdidc5vo87l6nlhszdpziabxh5 2398724 2398723 2026-04-02T21:18:36Z Stashbot 7414 kemayo@deploy1003: kemayo: Continuing with sync 2398724 wikitext text/x-wiki 073o6kyqrjbu09dxruiq5yhomplmhdq 2398725 2398724 2026-04-02T21:26:26Z Stashbot 7414 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1267204|SuggestedLinkEditCheck: fetchSuggestions return a jQuery.Promise]] 2398725 wikitext text/x-wiki rjx13obe07x0xshqo97rfwcqwpuculq 2398726 2398725 2026-04-02T21:28:06Z Stashbot 7414 kemayo@deploy1003: kemayo: Backport for [[gerrit:1267204|SuggestedLinkEditCheck: fetchSuggestions return a jQuery.Promise]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2398726 wikitext text/x-wiki 428vdsdcl9i69mjpmp7danmycrtzfbh 2398727 2398726 2026-04-02T21:28:34Z Stashbot 7414 kemayo@deploy1003: kemayo: Continuing with sync 2398727 wikitext text/x-wiki bt6216ez96srswtkquevgpg9sf8m05n 2398728 2398727 2026-04-02T21:32:44Z Stashbot 7414 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1267204|SuggestedLinkEditCheck: fetchSuggestions return a jQuery.Promise]] (duration: 06m 18s) 2398728 wikitext text/x-wiki 4m61blm767sd3skf2243hvlaajnwjha 2398730 2398728 2026-04-02T21:58:23Z Stashbot 7414 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1267214|Fix section heading spacing on mobile (T414882)]] 2398730 wikitext text/x-wiki jvc8jeo8vbaeoipmcrsgupj48chq20v 2398731 2398730 2026-04-02T22:00:04Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1267214|Fix section heading spacing on mobile (T414882)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2398731 wikitext text/x-wiki ndv0rylv255tfw2dig9aubelvkpf31i 2398732 2398731 2026-04-02T22:01:43Z Stashbot 7414 jdlrobson@deploy1003: jdlrobson: Continuing with sync 2398732 wikitext text/x-wiki py0x9ifisne7lvtorwdsmf43afnpqsn 2398733 2398732 2026-04-02T22:05:56Z Stashbot 7414 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1267214|Fix section heading spacing on mobile (T414882)]] (duration: 07m 33s) 2398733 wikitext text/x-wiki ftyfaipn8rhlie65arpdnh7o1dw0yr7 2398735 2398733 2026-04-02T22:20:23Z Stashbot 7414 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply 2398735 wikitext text/x-wiki mp7s4fk1hpfwsyjfk4s1z8v2l916apw 2398736 2398735 2026-04-02T22:20:36Z Stashbot 7414 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply 2398736 wikitext text/x-wiki ezgc39fi6noe5uk4v47q2iack2jvnrb 2398737 2398736 2026-04-02T23:06:01Z Stashbot 7414 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply 2398737 wikitext text/x-wiki sgu65bzx3xb7hr9ak2pj9eg8apcmyns 2398738 2398737 2026-04-02T23:06:10Z Stashbot 7414 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply 2398738 wikitext text/x-wiki 6ro3urs6af6uomuhuk5lqxewsktwswh 2398739 2398738 2026-04-02T23:35:44Z Stashbot 7414 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1264110|Start reading from new file table in dewiki and fawiki (T416548)]] 2398739 wikitext text/x-wiki 32fn13lm94murinj78s3r62b9eibcm2 2398740 2398739 2026-04-02T23:37:21Z Stashbot 7414 zabe@deploy1003: zabe: Backport for [[gerrit:1264110|Start reading from new file table in dewiki and fawiki (T416548)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2398740 wikitext text/x-wiki 0jsyl7jgpim5oetpz564vmp4du7u15f 2398741 2398740 2026-04-02T23:37:40Z Stashbot 7414 zabe@deploy1003: zabe: Continuing with sync 2398741 wikitext text/x-wiki 0szlu1arqnptk9031mso7b6bhlkff2t 2398742 2398741 2026-04-02T23:41:54Z Stashbot 7414 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1264110|Start reading from new file table in dewiki and fawiki (T416548)]] (duration: 06m 10s) 2398742 wikitext text/x-wiki 51gwd13wg8xgg66sludxxltm23nr0d2 2398743 2398742 2026-04-02T23:51:35Z Stashbot 7414 zabe@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply 2398743 wikitext text/x-wiki 1sha5ab2ovnesjlftx11y159hjw8hsd 2398744 2398743 2026-04-02T23:52:58Z Stashbot 7414 zabe@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply 2398744 wikitext text/x-wiki pmuivveipet550qacllq8mq0bxefnqk 2398748 2398744 2026-04-03T00:51:20Z Stashbot 7414 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1267286|Disable QueryPage updates for Special:Unusedtemplates on testcommonswiki (T422062)]] 2398748 wikitext text/x-wiki czhkyqkhuuu83agcymshbb18km92ubj 2398749 2398748 2026-04-03T00:53:31Z Stashbot 7414 zabe@deploy1003: zabe: Backport for [[gerrit:1267286|Disable QueryPage updates for Special:Unusedtemplates on testcommonswiki (T422062)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2398749 wikitext text/x-wiki icj65kznxautn5hgj21oelomuhn7bvv 2398750 2398749 2026-04-03T00:53:55Z Stashbot 7414 zabe@deploy1003: zabe: Continuing with sync 2398750 wikitext text/x-wiki 35mahr0ie4okfgye7aavngtazavlg7b 2398751 2398750 2026-04-03T00:58:10Z Stashbot 7414 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1267286|Disable QueryPage updates for Special:Unusedtemplates on testcommonswiki (T422062)]] (duration: 06m 50s) 2398751 wikitext text/x-wiki j2x42qo5bl52yry2w7vb03juco5dnf8 2398752 2398751 2026-04-03T00:59:13Z Stashbot 7414 zabe: zabe@deploy1003:~$ mwscript updateSpecialPages.php testcommonswiki # T422062 2398752 wikitext text/x-wiki 1qqnzoo7062m15l93q1rikpcy778nk9 2398753 2398752 2026-04-03T02:00:54Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2398753 wikitext text/x-wiki 75jex607wn6953h3md5z9ihbo2yyqge 2398754 2398753 2026-04-03T02:07:10Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 21s) 2398754 wikitext text/x-wiki e8hpbd42t2es5apf1y3e61iruyw53hk 2398758 2398754 2026-04-03T08:06:26Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-test1007.eqiad.wmnet with OS trixie 2398758 wikitext text/x-wiki 85c72sniw033nnac5vlnko892o5wz14 2398759 2398758 2026-04-03T08:22:43Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-test1007.eqiad.wmnet with reason: host reimage 2398759 wikitext text/x-wiki si59q285xsef6hvc5u9j8sof1ffbjcj 2398760 2398759 2026-04-03T08:29:49Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-test1007.eqiad.wmnet with reason: host reimage 2398760 wikitext text/x-wiki o2bos8krgybt071zpxy4vfrcc8tvhsc 2398761 2398760 2026-04-03T08:48:44Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-test1007.eqiad.wmnet with OS trixie 2398761 wikitext text/x-wiki 5so9d6y9s7w3bedjs6qpi3un7usaypf 2398762 2398761 2026-04-03T08:51:42Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-test1008.eqiad.wmnet with OS trixie 2398762 wikitext text/x-wiki 6woslctkwgkd8whoryqr0b9fq6i5k8l 2398763 2398762 2026-04-03T09:07:41Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-test1008.eqiad.wmnet with reason: host reimage 2398763 wikitext text/x-wiki q52mu8bq9nqhvjtaw2ndcd7at2all65 2398764 2398763 2026-04-03T09:14:45Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-test1008.eqiad.wmnet with reason: host reimage 2398764 wikitext text/x-wiki lf7omucxn9cd5wq8w52hlwdkjmifynq 2398767 2398764 2026-04-03T09:33:35Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-test1008.eqiad.wmnet with OS trixie 2398767 wikitext text/x-wiki 6z7m5crq5ge9nsc8a38nu6rn005xxkd 2398768 2398767 2026-04-03T09:36:11Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-test1009.eqiad.wmnet with OS trixie 2398768 wikitext text/x-wiki 9v6dj4gyrs3cwii7nk2bj1u2x393kod 2398769 2398768 2026-04-03T09:52:45Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-test1009.eqiad.wmnet with reason: host reimage 2398769 wikitext text/x-wiki hzm8q3h3selt6vntvditoxrhx136lqu 2398770 2398769 2026-04-03T09:52:46Z Stashbot 7414 brouberol@dns1004: START - running authdns-update 2398770 wikitext text/x-wiki 61ivayb0bfuqwsq50o87bn1n8lgqgef 2398771 2398770 2026-04-03T09:54:34Z Stashbot 7414 brouberol@dns1004: END - running authdns-update 2398771 wikitext text/x-wiki ixmfz2rh7kwcnaf9sfpmaq19qpyg6u9 2398772 2398771 2026-04-03T09:59:04Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-test1009.eqiad.wmnet with reason: host reimage 2398772 wikitext text/x-wiki rex6cinfnf1sxlik2ssq9va6qwy6yj7 2398773 2398772 2026-04-03T10:17:52Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-test1009.eqiad.wmnet with OS trixie 2398773 wikitext text/x-wiki gc1dhmmtd27ydd6csoluzbzujtgv6gn 2398775 2398773 2026-04-03T10:19:17Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host kafka-test1010.eqiad.wmnet with OS trixie 2398775 wikitext text/x-wiki 17t2oimhl2ke90h94nof34ejhfv5hgq 2398776 2398775 2026-04-03T10:29:32Z Stashbot 7414 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply 2398776 wikitext text/x-wiki cvkti24dcbhwk6rd9tnat6uyc60pdtg 2398777 2398776 2026-04-03T10:30:18Z Stashbot 7414 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply 2398777 wikitext text/x-wiki q8da4alw7eix6r5exjfygex6qd2fpt1 2398778 2398777 2026-04-03T10:32:55Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-test1010.eqiad.wmnet with reason: host reimage 2398778 wikitext text/x-wiki 33ejaeh9vbvi4j8f4mtm3ybjyvtqnl4 2398779 2398778 2026-04-03T10:39:07Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-test1010.eqiad.wmnet with reason: host reimage 2398779 wikitext text/x-wiki l7nl73tw5rsyi8phrab642xav47lgr1 2398780 2398779 2026-04-03T10:56:23Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-test1010.eqiad.wmnet with OS trixie 2398780 wikitext text/x-wiki tghekkqx5bmsx8mz6060ms638klr0em 0 11689 2398535 2396486 2026-04-02T13:11:00Z BTullis (WMF) 25295 typo 2398535 wikitext text/x-wiki ghqx0m3exx7r9eg9vas165m04stnyf9 0 17290 2398598 2398461 2026-04-02T15:07:04Z Stashbot 7414 bd808: Unblock 95.90.0.0/15 (T421485) 2398598 wikitext text/x-wiki a1sgk5ui8uzivomav9h9dmaiitz7a7z 2398606 2398598 2026-04-02T15:15:51Z Stashbot 7414 bd808: Unblock 82.216.0.0/16 (T421508) 2398606 wikitext text/x-wiki 2ukl1asxu8i9kecyjjwdfsw3pazi3h6 2398654 2398606 2026-04-02T16:50:46Z Stashbot 7414 thcipriani: restart jenkins 2398654 wikitext text/x-wiki 7uqapmqavskk3sblh0qn0jmbaf7giqg 2398755 2398654 2026-04-03T02:47:53Z Stashbot 7414 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1267398 2398755 wikitext text/x-wiki 0gf5869zsni9fxoj1i87zhvl2ys6mui 2398765 2398755 2026-04-03T09:15:11Z Stashbot 7414 hashar: Added Bookworm based Jenkins agents to the pool with label `Docker`. Hostnames are `integration-agent-docker-107*` # T421114 2398765 wikitext text/x-wiki 2ot5fs2o27af7cvrdb5nh43zmohovyt 2398766 2398765 2026-04-03T09:27:00Z Stashbot 7414 hashar: integration: nuked pywikibot/core pre-commit cache # T422242 2398766 wikitext text/x-wiki ggcziy555ftegloqgsemoy8gcihdmwp 498 30942 2398501 2398455 2026-04-02T12:34:31Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch 2398501 wikitext text/x-wiki ikkp81k66vwgpge5gddhcpkq88u7afj 2398503 2398501 2026-04-02T12:34:59Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch 2398503 wikitext text/x-wiki 70kyyoz5tkynhqm8epq0tpz9nttcwdq 2398504 2398503 2026-04-02T12:35:12Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch 2398504 wikitext text/x-wiki k8l89uksq8sqsa1yvz8bhvugsvnz2dl 2398505 2398504 2026-04-02T12:35:42Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch 2398505 wikitext text/x-wiki rruwdpzkuob21qs6cdbcu81evvw9zc4 2398506 2398505 2026-04-02T12:35:50Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2398506 wikitext text/x-wiki dl25sasc7lkw4dasfkmkzxd55mfyx2d 2398507 2398506 2026-04-02T12:36:18Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2398507 wikitext text/x-wiki adfwc26remnayf6y8e3qo7d2yz5f9re 2398510 2398507 2026-04-02T12:43:40Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch 2398510 wikitext text/x-wiki pzv1oznmzu45ddjvqjycd64yks7xb5a 2398511 2398510 2026-04-02T12:44:09Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch 2398511 wikitext text/x-wiki 4fpkrza8hy8yo0egvjz2rk6qne2on8a 2398512 2398511 2026-04-02T12:44:16Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2398512 wikitext text/x-wiki 77bn88sp4evwunfn1nx94g7dr9i8vrs 2398514 2398512 2026-04-02T12:44:38Z Stashbot 7414 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch 2398514 wikitext text/x-wiki h8aylllwvzvmzjjkt31goq76z3qjswf 2398517 2398514 2026-04-02T12:45:13Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch 2398517 wikitext text/x-wiki ggoafiln1ucsr1910a0vlseyhgvn0c7 2398519 2398517 2026-04-02T12:45:49Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch 2398519 wikitext text/x-wiki pr1d75zadh55iym4brsrvklydii57q7 2398774 2398519 2026-04-03T10:19:00Z Stashbot 7414 godog: move codfw neutron l3-agent queues to quorum - T421054 2398774 wikitext text/x-wiki o4gsujhk2kws73o04q6onjujkcrk1p6 498 443800 2398782 2390819 2026-04-03T11:34:37Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed eb7875d76b (temporarily disable notifications feature to unbreak tool: T420154, T421991) 2398782 wikitext text/x-wiki 6kjqi2kkszhmezj0a6lmm9kfc8rrdie 2398783 2398782 2026-04-03T11:54:59Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 1079b511fd (refactor authentication_area()) 2398783 wikitext text/x-wiki 6q8jjjpgxnjbakq6ugqrp36e9j48vyg 0 444143 2398555 2364943 2026-04-02T14:00:05Z MLechvien-WMF 49197 Removing Phabricator which has moved to Infra Foundation team 2398555 wikitext text/x-wiki 9miq47g7sim59mc7ifizaibt58sk9ky 498 444707 2398682 2362491 2026-04-02T17:28:54Z Stashbot 7414 mutante: - puppetmaster-1003 was stuck on puppet data from around December 2025. - reset git repo status 2398682 wikitext text/x-wiki r3awtx7ropxxiu8cl71m7d2l6zhys0y 3 446778 2398781 2398143 2026-04-03T11:15:58Z MBH 3865 /* Webservice on Toolforge */ reply to DCaro (WMF) ([[mw:c:Special:MyLanguage/User:JWBTH/CD|CD]]) 2398781 wikitext text/x-wiki 66wpqf6q71tzrrfo4yjp2w2lpm4ojnv 0 449031 2398747 2398301 2026-04-03T00:12:24Z CStone (WMF) 38439 2398747 wikitext text/x-wiki rojt1x2a0aa08coqmv7d5v14fvzvp49 0 449160 2398745 2398337 2026-04-03T00:02:43Z Dexbot 30554 Bot: Updating the report 2398745 wikitext text/x-wiki 03o3pwtcdbcap5yd297erpq1sce1gk7 0 450294 2398545 2388649 2026-04-02T13:49:08Z BRouberol-WMF 39141 /* List of instances */ 2398545 wikitext text/x-wiki rfsniv57h65gc680wew8leto2njghop 0 450684 2398572 2393234 2026-04-02T14:35:52Z BRouberol-WMF 39141 /* Development instance */ 2398572 wikitext text/x-wiki ivrl9safmh68b1z00oxnrk3i8synp8e 498 452890 2398734 2397141 2026-04-02T22:11:43Z Stashbot 7414 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23924337776 (https://github.com/cluebotng/component-configs/commits/266152f9f673810b0c9460b5828cb86e7aee31d9) 2398734 wikitext text/x-wiki mkmddj3chagz5ki3gm26crg6ys2mgs5 0 453090 2398562 2372991 2026-04-02T14:16:40Z Jgreen (wmf) 18 2398562 wikitext text/x-wiki 58klvidb8b27tzgr6wxpj77o5281ahe 2398563 2398562 2026-04-02T14:17:09Z Jgreen (wmf) 18 2398563 wikitext text/x-wiki nl8q5d33c4uvucuw7r8v3w1if800daf 2398566 2398563 2026-04-02T14:18:42Z Jgreen (wmf) 18 /* Supermicro - Connect by IPMI */ 2398566 wikitext text/x-wiki 8xra4hihs0qsq0rzkevmgi9r0q4x8r2 2398567 2398566 2026-04-02T14:20:54Z Jgreen (wmf) 18 /* Supermicro - Connect by IPMI */ 2398567 wikitext text/x-wiki qftku561pjllxu1njguhu2d9tsv7thh 2398568 2398567 2026-04-02T14:25:29Z Jgreen (wmf) 18 2398568 wikitext text/x-wiki 8uzviw286kthcuvoxh91rslprajv44c 2398569 2398568 2026-04-02T14:26:47Z Jgreen (wmf) 18 2398569 wikitext text/x-wiki 16xzqutca28g03v6b8iu0l71vpl8yu2 2398600 2398569 2026-04-02T15:09:51Z Jgreen (wmf) 18 /* Useful Commands/Operations */ 2398600 wikitext text/x-wiki e91qngum3nsbgtad9pflrza54tw3abm 2398603 2398600 2026-04-02T15:12:18Z Jgreen (wmf) 18 /* Useful Commands/Operations */ 2398603 wikitext text/x-wiki 0la4to3gm2fp12nkh7o0qlojwtu5kw4 2398615 2398603 2026-04-02T15:27:32Z Jgreen (wmf) 18 /* Supermicro */ 2398615 wikitext text/x-wiki l8sq13hxxdx8kfkvwgtqo7u67pjgqwv 2398625 2398615 2026-04-02T15:47:28Z Jgreen (wmf) 18 /* Connect by IPMI */ 2398625 wikitext text/x-wiki ma2xak2v1ncqfjl9u8ubuc1ftiwzkxf 2398638 2398625 2026-04-02T16:10:52Z Jgreen (wmf) 18 /* BIOS Settings */ 2398638 wikitext text/x-wiki 06eopyo3rytgi2zkf11f7v1o5io64in 2398721 2398638 2026-04-02T21:12:50Z Jgreen (wmf) 18 /* Useful Commands/Operations */ 2398721 wikitext text/x-wiki e2xx5z2e9awdjb18ux8tmlb07woict4 3 457373 2398608 2398259 2026-04-02T15:19:46Z BryanDavis 1604 /* Wikitech account renamed and attached to SUL */ Reply 2398608 wikitext text/x-wiki cdvnoum0k7bqj2tfum32gumxenmobf0 498 459005 2398757 2380062 2026-04-03T04:08:32Z Stashbot 7414 wmbot~soda@tools-bastion-15: soda built and uploaded a new version 2398757 wikitext text/x-wiki 3bag8uthdkjuhde6n0jynsnj5mt75ff 0 459014 2398484 2320797 2026-04-02T12:17:09Z JMeybohm (WMF) 16709 /* Required patches */ Add istio config to prere 2398484 wikitext text/x-wiki qiaqb29d0wwael139m1o49cdeyr2kn4 0 459115 2398756 2397135 2026-04-03T03:33:24Z AndrewBogott (WMF) 51047 /* helm chart repo */ 2398756 wikitext text/x-wiki 1zw7fmp75fdwmhyqy37snm5qkyp6aun 0 459699 2398729 2398302 2026-04-02T21:38:44Z Stashbot 7414 cstone: payments-wiki upgraded from 86bec442 to 80cda166 2398729 wikitext text/x-wiki pebj9hgr1jz5kf0etlyzs48c7j9dn7s 2398746 2398729 2026-04-03T00:04:17Z Stashbot 7414 wfan: civicrm upgraded from 49f541cd to 4ee11209 2398746 wikitext text/x-wiki 10opk2qjz3iontx79elxsveqlmiiomz 0 459743 2398664 2387239 2026-04-02T17:03:57Z LErickson-WMF 51715 /* Components */ 2398664 wikitext text/x-wiki 1hs7hv8jk36rqi69cozf48p4f26pl1l 0 459783 2398661 2395281 2026-04-02T17:01:20Z DKinzler (WMF) 156 /* Ratelimit Classes */ Update rate limit classes 2398661 wikitext text/x-wiki rcd54891wz4uzsq04ot6itnto8itg1b 498 459970 2398559 2396019 2026-04-02T14:03:39Z Stashbot 7414 wmbot~difronzo@tools-bastion-15: deployed v0.7.3 (by DiFronzo on GitHub) 2398559 wikitext text/x-wiki hnxgvf7qz8oz8ohgan1isluutqg88kz 0 460007 2398630 2396523 2026-04-02T15:52:56Z BryanDavis 1604 No need for an H1, just make a lede 2398630 wikitext text/x-wiki dtck0q7bbyus1qi7wp3oh4o9xn44vpa 0 460023 2398485 2026-04-02T12:19:51Z AYounsi (WMF) 10040 Initial doc, copied over from google doc 2398485 wikitext text/x-wiki ir9kvxe58vx8uwylmfnhpffxa9l176w 2398486 2398485 2026-04-02T12:22:14Z AYounsi (WMF) 10040 2398486 wikitext text/x-wiki fisx54lncg1ohqstdh14irsnmamrg63 2398487 2398486 2026-04-02T12:22:32Z AYounsi (WMF) 10040 2398487 wikitext text/x-wiki j30ela2jatjpbbcj1cn17lah84fmvmq 0 460024 2398660 2026-04-02T17:00:30Z LErickson-WMF 51715 Created page with "== Scope & Intent == This runbook is a '''troubleshooting guide''' for [[Wikidata Platform]] engineers to correct a situation where Blazegraph has become out of sync with the Wikidata source of truth because of events never making it to the streaming-updater pipeline in the first place. == Verify the problem == We currently find out about this situation via a bug report like [https://phabricator.wikimedia.org/T407702 T407702], in which a user reports that an item delet..." 2398660 wikitext text/x-wiki 98clq7fn0ayrzavsfdci4medga6evi3 0 460025 2398688 2026-04-02T18:20:26Z BKing (WMF) 29017 Created page with "Placeholder" 2398688 wikitext text/x-wiki rp6ekfc3smhpzk3bym7bz0jugu65jzc